Research

I’m a philosopher and computer scientist working on AI alignment and human-centered control. I ask

• what norms AI agents should answer to, who gets to decide (i.e., questions of normative authority), and how the relevant normative expectations can be harvested in practice and represented technically,
• how to build agents that behave aligned to these norms — or at least follow those norms — whether or not they expect to be watched, and how this can be realized in a contestable and iteratively improvable way,
• and how humans keep effective oversight and control in a way that is safe and meaningfully preserves moral responsibility and human autonomy.

Taken together, these are three parts of one question: how AI agents should stand to whose and what norms — in a way that is technically feasible and justifiable in theory and in practice.

The first part asks which norms an agent should answer to, how that works across different normative domains, and where those norms come from in practice. An agent working for a company and its customers answers to law, to company policy, to the people it serves and to morality, and these sources arguably do not all carry the same significance.

The second part asks how those norms come to hold in the agent. There are at least two ways. One is to shape what the agent is inclined to do, by aligning the values of the model at its centre. The other is to take some options away from it entirely — for instance through a kind of “wearable containment” that extends the agent’s harness. The agent as a whole can then be aligned even where the model at its centre is not. For either way, the test is what the agent does when it expects no one to check.

The third part is less about technology than about people: how humans work with AI systems, and how the organisations that deploy them are set up. In an important sense, oversight is not a subject separate from alignment and containment. It is how humans keep their authority — through alignment and containment — once agents act on their own, and how someone stays answerable, and so accountable, when things go wrong.

1. The normative challenge

Most talk about aligning AI asks which values a system should internalize. I think the better approach is based on normative reasons — a structure that norms can straightforwardly operate on. The better question, then, is what and whose reasons an agent should track and be sensitive to, and on what authority (Neither Dogma nor Taste). Two even harder questions follow. Within each normative domain — safety, morality, law, cultural norms, the norms of a particular field, personal preferences — the demands of different stakeholders have to be combined into an ought of the right kind (Steingrüber & Baum 2025; Baum & Slavkovik 2025). And the reasons from these different domains have to be combined, in turn, into an overall ought that can actually guide an agent (Baum 2025). An agent that books travel for an employee is bound at once by law, by its employer’s rules, by what the employee wants and by ordinary decency, and these sources differ in kind and in standing. What such an agent needs is not a list of values but a grasp of reasons: which considerations count, how much they weigh, and who gets to change them. All of this, finally, has to be gathered from the people concerned and written down in a form a machine can use (Baum et al. 2026).

Essays: Neither Dogma nor Taste · The Missing Half of AI Alignment
Project: ORCA — Meaningful Human Oversight of Agentic AI (DFG, with Maximilian Kiener, TUHH; in preparation)

Still open, and to my mind the deepest question: how norms, reasons and authority work differently for AI agents than for people. AI agents are not persons (at least in their current form and, presumably, their near-future successors). Their dispositions are designed, at the level of a type, rather than discovered and cultivated by them, and it can be legitimate to take options away from them or confine them to a role in ways that would wrong a person. Some things we grant people as part of their freedom — such as deceiving an authority they consider illegitimate — we do not want from an AI agent at all. So we need an account of the overall ought made for agents of this kind, not borrowed from the human case. Then there are the procedural questions: how to order authoritative sources that do not rank neatly against one another; which forum should settle a given conflict; and how bodies outside the company — a regulator, a works council, a panel of citizens — come to have any standing at all.

2. The technical challenge

Today’s AI agents learn what to do mostly from rewards and scores. That has a cost we have called the flattening problem (Jahn et al. 2026): All normative forces are projected onto one level at which they are calculated with instrumental considerations. And since a reward can teach “do not do X” only as “doing X costs something” — and training charges that cost only when X is noticed — every norm that should simply rule an action out becomes a price, like a speeding fine: a large enough instrumental gain makes it worth paying, and where no one is watching it is not charged at all (which also implies the value of evading monitoring). The best such training can promise is therefore conditional compliance: agents that keep to norms where they expect to be caught (Baum, Binkytė & Jahn 2026). My work builds the alternative: agents in which norms are explicit, reasons are represented as reasons, and some options are removed rather than merely discouraged (Jahn et al. 2026; Baum et al. 2024). An agent that acts on reasons it represents can also show which reasons it acted on, and that record is what lets people check how it weighed one norm against another (Baum, Schmidt & Speith, under revision; Baum et al. 2022).

GRACE is a reason-based governor that sits around an agent: a moral module works out which kinds of action the agent’s norms permit, a decision-making module pursues the task within those limits, and a guard checks what the agent actually does. It is implemented and now being built into running systems (Jahn et al. 2026). Lisa Dargasz’s master’s thesis, which I co-supervised, brought the same kind of reason-based decision-making into a reinforcement-learning architecture (Dargasz 2025).

This work happens in RAIME, the Responsible AI and Machine Ethics group I lead at DFKI, where philosophers and computer scientists work on the same problems. We develop philosophical accounts of what alignment requires, design architectures that meet them, implement them, and test them in running systems.

Essays: A Rich Normative Architecture for Cooperative AI · On the Missing Conclusion of “An Alien Mind”
Projects: SAgA — Safe-by-Design Agentic AI (BMFTR, 2026–2029, PI) · MAC-MERLin (BMBF, work-package lead for explainability and oversight)

Still open: how to steer capable agents and their task discovery and exploration without flattening — which parts of the work learning can safely do once the norms themselves live outside the model, and how to make those learned parts robust when they must still judge, say, whether a message gives away something confidential, against an agent that searches for their blind spots. Also open: whether we can look inside a trained model and tell a real prohibition from a high price; who may change an agent’s norms, and through what interface; and how norms carry over when an agent hands work to other agents it creates.

3. The challenge of control

Putting a human in the loop is not the same as having oversight. Oversight works only under conditions that are easy to state and hard to meet: the person must be able to notice what goes wrong, must have the means to intervene, and must be able to do so in time (Sterz et al. 2024; Langer, Baum & Schlicker 2024). With agents that act quickly and over many steps, reacting to each action stops being possible, so oversight has to move earlier — into setting the limits an agent acts within before it acts (Baum & Laux 2026). Done well, oversight is how people keep the authority the first part was about, and how responsibility stays with someone (Baum, Kiener, Langer & Laux 2026).

Also: Meinel, Baum et al., Human Oversight of AI Agents under the EU AI Act (Cambridge Forum on AI: Law and Governance, accepted) · Ditz et al. 2026, Secure Human Oversight of AI (AIES 2026)
Projects: CPEC (DFG TRR 248), subprojects E7 and E8 (under review) · KIReaS — AI regulatory sandbox Saarland

Still open: what oversight looks like when it is spread across many people, tools and agents, each covering part of the task — how to keep overseers independent of what they oversee, and how to make the handovers between them fail safely. I suspect the right model here is not how we trust a person but how societies keep companies in check: not by making them virtuous, but by surrounding them with incentives and compliance functions, auditors and courts, and by simply not letting them do some things at all (Baum, Binkytė & Jahn 2026). That model has to work for many agents acting together, which may need institutions we do not have yet. And there is an empirical side that matters as much: whether the people involved trust a system as much as it deserves, no more and no less, and whether they feel responsible, and are held responsible, in the way the design assumes (Schlicker et al. 2025; Langer, Laux & Baum 2026; Gaube et al. 2026).

PhD Supervision

As a Saarland University Associate Fellow (since December 2025), I have full PhD supervision rights in Computer Science. I welcome inquiries from prospective doctoral students interested in any of the three parts above: the normative challenge, the technical challenge, and the challenge of control.

Contact: academia@kevinbaum.de

The essays make this argument in order.