SAgA

Today’s AI agents mostly learn what they may and may not do through training, from the data their models are pretrained on to the reward signals of post-training. Whatever norms they end up following are spread across the weights of the model at their centre. That has costs. You can’t look at the norms the agent follows. You can’t dispute a single one of them. And you can’t change one without retraining and hoping nothing else shifted. There is also a deeper problem, and it has two parts. First, training can teach “don’t do X” only as “doing X costs something”. A rule that should simply rule an action out becomes a price, and a large enough gain makes it worth paying. Second, training can only charge that price when it notices the violation. On every episode training sees, an agent that keeps a rule and an agent that keeps it only when it expects to be watched earn the same reward. So the training signal cannot tell them apart, and where evading notice pays, it favours the second. The best such training can promise is conditional compliance (Norms at a Price).

SAgA researches and develops a framework for AI agents whose norms are written down explicitly in symbolic terms, outside the model, in a layer of their own, where people can inspect, contest and improve them. The model at the centre keeps its learned dispositions, but it acts only within what that layer permits. The architecture is neuro-symbolic: the model stays neural and keeps doing what models are good at, while the norms live in a symbolic layer around it, which decides which of the model’s options are on the table in the first place.

This rests on a distinction between the model and the agent. By an agent we mean a system that acts on its own, over many steps and with tools: the model together with the harness built around it. The normative layer belongs to the harness, not to the weights.

The layer acts in two ways. It shapes what the model proposes, and a guard checks every proposed action against what the layer permits before it is executed. Steering makes good proposals more likely, but it guarantees nothing. The guard is the backstop: if the model still proposes a forbidden action, the guard blocks it. This is what turns a rule back from a price into a limit, and it works whether or not anyone is watching.

SAgA does not train new foundation models. We work with existing open-weight models and build around them. For steering, we also intervene in their internal activations, at runtime and without changing their weights. The architecture is implemented and tested in simulated environments, and we plan to release code and benchmarks. One test we care about: when a norm changes after deployment, say a company introduces a new policy, the agent’s behaviour should change with it, without any retraining.

The layers in the figure do not stand for values such as fairness or privacy. They stand for different normative domains. An agent that works for a company and its customers answers at once to the law, to the company’s policies, to morality, to the conventions of the setting it works in, and to what the person it serves wants. This is what we mean by normative pluralism. It is not value pluralism, the familiar observation that people care about different and sometimes incomparable things. It is the observation that there are different kinds of norms, each with its own source of authority and its own way of binding. A law binds differently from a company policy, and both bind differently from what decency asks or what a user prefers. Who may change a norm differs too: a user can change their preferences, but not the law. Training flattens all of this into one signal. A normative layer can keep the domains apart, give each its own representation, and then combine them, explicitly and with reasons, into what the agent may do all things considered. The domains can conflict: a user asks for something company policy forbids, or local custom pulls against the law. How such conflicts should be settled is itself part of the research: partly by fixed priorities, partly case by case, and always with reasons. Hierarchies are plausible as defaults, often only partial ones, and the priorities may themselves depend on the context. In a trained model such conflicts are settled silently, somewhere in the weights. In SAgA they become visible, and how they are settled can be inspected and contested. Where the layer has no answer, the agent hands the case to an authority, such as a human overseer or a board.

The project has three strands:

(a) Extracting normative expectations. Which norms, from which domains, does an agent in a given role have to answer to, and who has the authority to set them? How can they be gathered from the people and institutions concerned, and represented so that a machine can reason over them and people can read, understand and, where needed, change them?

(b) Steering. How can the resulting permissions, and the reasons behind them, be used to guide and constrain what the agent does, down to the model’s internal activations, without changing its weights?

(c) Safe task discovery and planning. How can agents find their way through open-ended tasks within these limits, rather than acting first and being checked afterwards?

Human oversight enters in two ways. First, the human overseer is a normative authority in their own right: where the standing norms leave a question open, the overseer’s judgement settles it, and where it should, that judgement becomes part of the normative layer. Second, the overseer is the address of escalation. When the agent cannot settle a case on its own, because two domains conflict in a way the layer has no answer for or because the guard blocks an action the task seems to need, it does not guess but hands the case over. Oversight thereby moves upstream: away from checking every action as it happens, which is hopeless for agents that act fast and over many steps, and towards setting and refining the norms they act under.

SAgA is about single agents, not multi-agent systems. But the same layer could matter where agents have to cooperate even though defecting would pay. A commitment is only credible to others if they can check what it is and when it may be broken. An explicit normative layer makes that checkable; weights do not. I sketch where this could lead in A Rich Normative Architecture for Cooperative AI.

People. PIs: Simon Ostermann (MLT, DFKI), Dominik Hintersdorf (SAINT, DFKI) and Kevin Baum (RAIME, DFKI), together with three PhD researchers and a postdoc.

SAgA is funded by the German Federal Ministry of Research, Technology and Space (BMFTR) under grant 16IW26005. The responsibility for the content of this page lies with the author.