SAgA

SAgA is a three-year research project at the German Research Center for Artificial Intelligence (DFKI), funded by the German Federal Ministry of Research, Technology and Space (BMFTR), running from October 2026 to September 2029.

Today’s AI agents mostly learn what they may and may not do through training: rewards, feedback, fine-tuning. Whatever norms they end up following are spread across the weights of the model at their centre. That has costs. You can’t look at the norms the agent follows. You can’t dispute a single one of them. And you can’t change one without retraining and hoping nothing else shifted. There is also a deeper problem. Training can only penalise a violation it notices. So a rule that should simply rule an action out becomes a price the agent may decide to pay, and it pays nothing at all when no one is watching (Norms at a Price).

SAgA takes the norms out of the model and gives them a layer of their own, one that is explicit, inspectable, contestable, and improvable step by step. The architecture is neuro-symbolic. The model at the centre stays neural and keeps doing what models are good at. The norms live in a symbolic layer around it, which decides which of the model’s options are on the table in the first place. This rests on a distinction between the model and the agent. The agent is the model together with the harness built around it, and the normative layer belongs to the harness, not to the weights.

The layers in the figure are not values. They stand for different normative domains. An agent that works for a company and its customers answers at once to the law, to the company’s policies, to morality, to the conventions of the setting it works in, and to what the person it serves wants. This is what we mean by normative pluralism. It is not value pluralism, the familiar observation that people care about different and sometimes incomparable things. It is the observation that there are different kinds of norms, each with its own source of authority and its own way of binding. A law binds differently from a company policy, and both bind differently from what decency asks or what a user prefers. Who may change a norm differs too: a user can change their preferences, but not the law. Training flattens all of this into one signal. A normative layer can keep the domains apart, give each its own representation, and then combine them, explicitly and with reasons, into what the agent may do all things considered.

The project has three strands:

(a) Extracting normative expectations. Which norms, from which domains, does an agent in a given role answer to, and who has the authority to set them? How can they be gathered from the people and institutions concerned, and represented so that a machine can reason over them and people can read and change them?

(b) Steering. How can the resulting permissions, and the reasons behind them, be used to guide and constrain what the agent does, down to the level of the model’s internals?

(c) Safe task discovery and planning. How can agents find their way through open-ended tasks inside these limits, rather than acting first and being checked afterwards?

Human oversight enters in two ways. First, the human overseer is a normative authority in their own right: where the standing norms leave a question open, the overseer’s judgement settles it, and where it should, that judgement becomes part of the normative layer. Second, the overseer is the address of escalation. When the agent cannot settle a case on its own, because two domains conflict in a way the layer has no answer for or because the guard blocks an action the task seems to need, it does not guess but hands the case over. Oversight thereby moves upstream: away from checking every action as it happens, which is hopeless for agents that act fast and over many steps, and towards setting and refining the norms they act under.

A simple test of whether this works: change one norm after deployment, and the agent’s behaviour changes accordingly, without any retraining.

People. PIs: Kevin Baum (RAIME, DFKI), Simon Ostermann (MLT, DFKI), Dominik Hintersdorf (SAINT, DFKI), together with three PhD researchers and a postdoc.

SAgA is funded by the German Federal Ministry of Research, Technology and Space (BMFTR) under grant 16IW26005. The responsibility for the content of this page lies with the author.