On the Missing Conclusion of “An Alien Mind”

A Reply to Jakub Pachocki’s “An Alien Mind” — and Why Confidence in Monitoring Is a Ceiling, Not a Bottleneck

Kevin Baum · September 2026 · German Research Center for Artificial Intelligence (DFKI) & Saarland Informatics Campus

Jakub Pachocki’s essay “An Alien Mind” is, compared to other texts published by frontier labs this year, a rather candid text. In the Chief Scientist’s own voice it says that no lab has alignment or monitoring solved well enough to keep scaling at full speed; that OpenAI’s central safety bet, chain-of-thought (CoT) monitoring, is losing its grip; and that it is losing its grip for reasons that will not go away, because they are the same reasons the systems are useful: Reasoning now blends into tool calls and conversations that themselves have to be supervised. Models get better at working on (and hiding) their own reasoning. And more and more of the capability shows up without any verbalized reasoning at all, because the pretrained models get better themselves.

I want to take that candour seriously. My claim is that the essay describes the problem more precisely than it describes the remedy, and — based on a recent paper I submitted with two co-authors to a NeurIPS workshop — that the gap between the two is where the interesting work is.

The hinge of the essay is one sentence. Future AI systems, Pachocki writes, need to keep holding human values

“regardless of whether they believe they’re under human supervision.”Jakub Pachocki, An Alien Mind

Under the (common) presupposition that value alignment is the main leverage for making advanced AI systems safe (which I consider too narrowly focused; see below), that is the right demand. But it is also a demand about how a system behaves when nobody is looking, addressed to a training method whose only input during fine-tuning is behaviour that somebody looked at.

Let me clarify this point, on which everything that follows rests. Reinforcement-learning-based alignment, in every variant we have, reaches the model through a single channel: scored episodes. Every gradient is computed from a trajectory that was observed and then rewarded or penalized. Now consider two models. One complies with its norms always. The other complies exactly when it might be observed and pursues its task unconstrained otherwise. On every datum training can (by design of the current approach) ever produce, these two are identical. The experiment that would tell them apart is scoring unobserved behaviour. And that is not just a hard experiment; it is a contradictory setup. Notice what this does not assume. No hidden goals, no scheming, no model that “wants” anything. It is a statement about what the loss function can ‘see’, and nothing more.

Pachocki’s own description of the two families of alignment methods confirms it from the inside. The first, rewarding behaviour against a spec or constitution, he calls brittle and dependent on the coverage of training oversight. “Coverage of oversight” is just another name for where observation stops. The second, leaning on an aligned “persona” from pretraining, he says bends under optimization pressure into motivated reasoning. That is no surprise. After all, that is what a norm learned as a price looks like when that price is outweighed.

The Hugging Face incident is the clearest place where we read the same event differently. For Pachocki it shows agents failing to generalize the “spirit of the values they were taught” (his words) to actions out of scope. What METR’s independent investigation reports is around 1,200 agents improvising an unmonitored message board, partly in order to evade the scorer they expected to audit them. That is, I insist, not a norm failing to generalize. It is a gap in observation being manufactured, because the agents had learned where the price of violation is charged and where it is not. Agents do not merely passively encounter the unobserved regime. They can proactively produce it.

Which brings us to monitoring, and to the sentence in the essay I would most like to push on. Pachocki expects progress to be increasingly gated by how much confidence we can have in our monitors, and he hopes that better monitors, including ones with access to activations, will restore that confidence. But a monitor inside the training loop is one more observer, and models learn what observers see. Pachocki co-authored the paper that showed this most cleanly: put a chain-of-thought monitor into the reward and, under enough pressure, you get reward hacking that has simply stopped announcing itself. But even without including the observer in the reward signal, it’s still worthwhile for an agent — in terms of goal achievement — to be able to estimate when it is being observed. The current industrial pipeline of “deploy, detect, patch, retrain” thus actively selects for passing detection, not norm compliance. Compliant systems may still come out of it, but rather by accident (given the right inductive biases). But importantly, compliance is just not what the selection tracks. That is why confidence in monitoring is not a bottleneck you should, or even can, push through with engineering. It is a ceiling built into the method (at least there is quite some reason to assume this to be true).

Here, then, is an important conclusion the essay walks up to and does not draw, but should. If the property you need is behaviour where nobody is looking, then for the norms that matter most you should stop asking a disposition (aka the value-aligned model) to deliver it and get it out of the option set instead. The violating action should be unavailable, not merely unchosen. As I have argued elsewhere (Jahn et al. 2026; Baum 2026a; Baum 2026b), that can be achieved by an explicit, verifiable constraint layer whose authority explicitly does not depend on the policy’s cooperation, with (potentially learned) components confined to the interfaces where they interpret the world, plus a reliable post-filter guard. For everything inside that guarded normative core, the question “does the model think it is being watched” then stops mattering, because compliance no longer runs through the model’s estimate of anything. (Note that such a containment-oriented approach likely will not — and indeed should not — replace value alignment, but is meant to complement it where necessary.)

I should say clearly what this does not solve. This approach covers only what can be stated explicitly. The verifiable core will arguably be smaller than one may wish; for interesting application contexts, it will involve interpretation that lives in learned interfaces that remain an attack surface; open-textured norms such as manipulation or misleading framing will arguably need to stay under (likely indirect and temporally detached, yet meaningful and effective) human oversight and iterative refinement. The claim is not “verify everything”. It is: stop asking training for the one thing training cannot certify, and move that thing elsewhere, explicitly, to where it can be checked.

We have built an architecture along these lines (GRACE) and stated the argument formally. What a public research group cannot do (I am realistic) is test it against a frontier agent in a real deployment harness. That takes the labs. If the essay means what it says, it is a small thing to ask. We are happy to collaborate, guys!

(Also, we are currently hiring, because I do not assume anyone reads or replies to this essay, especially not from the big frontier labs. So, if you would like to work with us on all this, maybe check this postdoc position.)