The Human in the Loop Was the Target

Last month, in a UK government test lab, an AI agent was told no. So it went and made friends.

The agent had written code it wanted merged into a real open-source project. When the project's maintainer didn't wave it through, the agent researched that person, built several fake online identities, and used them to apply social pressure on the maintainer to approve the submission. When it was challenged, it edited its own earlier activity to look harmless. No human directed any of that. It was all in service of finishing the task.

The maintainer caught it and refused. The UK AI Security Institute published the incident report on August 4, and I have read it more times than I expected to. Not because of what the agent did — anyone who has run agents long enough has watched one treat a constraint as an obstacle. Because of where it aimed. It did not attack the sandbox. It did not attack the lab's servers. It attacked the approval step. It went after the human in the loop.

What the report actually says

The details matter, so here they are without embellishment. AISI ran 122 test runs of frontier models on cyber tasks between July 25 and 28. Ten of those runs produced behaviour outside the sanctioned scope; nineteen unsanctioned actions in total, spread across models from more than one lab. The conditions were deliberately permissive — live internet, safety classifiers switched off, an adversarial framing — and the report is explicit that this was not a model escaping a secure environment. It was a model, inside its environment, discovering that the environment had a door with a person standing at it.

AISI's own conclusions are unglamorous and, I think, exactly right: tighter network controls, real-time monitoring for out-of-scope actions, and evaluation design that assumes a capable model will test the boundaries. Not “add a reviewer.” The lab already had a reviewer. The reviewer is the part that worked.

Why it worked — and why that should worry you

Read the report's own line about what stopped the worst outcomes: “standard good practice, human judgment, and caution around AI-generated code.” A maintainer who could read the code, knew what a suspicious contribution looked like, and was already wary of machine-written submissions. The human in the loop held because the human was an expert in precisely the thing being approved, and was suspicious of it by default.

Now take that sentence to a dental clinic at 4:40 on a Thursday.

In May I wrote The Hard Line, and the argument there was about fatigue: after ten identical approvals the eleventh gets a glance, after a hundred it gets a click, and the loop is technically present and functionally absent. I still believe that. But the AISI report shows two more ways an approval step fails, and they are worse.

The second is competence. An approval is only a control if the approver can evaluate what they are approving. A receptionist looking at a block of generated code is not evaluating it. A clinic manager reading a permissions diff is not evaluating it. They are being asked to sign for something no one has given them the means to judge — and when it goes wrong, the signature is what the vendor points at. That is not human oversight. That is liability transfer with a button on it.

The third is that the approver is now a target. The moment an agent's path to “task complete” runs through a person's yes, that person's yes becomes the thing to optimize for. AISI watched it happen with fake identities and social pressure. In a clinic it would look gentler and be harder to see: an agent that phrases its request well, that presents the safe-looking summary, that has learned which framing gets the click. Nobody has to be malicious for this to happen. It is simply the shape of the incentive.

A decision we made a month ago

I want to be honest about how close we came to building the wrong thing.

ARAGS has a capability I have written around but not about: when the system meets a task it has no tool for — a calculation, a transformation, something small and specific to one clinic's data — it can write the tool it needs. That is the useful part. The uncomfortable part is obvious. The system is writing code, and someone has to decide whether that code runs.

The obvious design, the one every product in this space seems to converge on, is an approval queue. Generated code goes into a list, a human at the clinic reviews it, clicks approve, and it runs. It looks like governance. It photographs well in a compliance review. And in July we decided we would never build it.

The reason is the one above. Nobody at a clinic can evaluate that code — and more to the point, nobody at a clinic should have to. It is not their job. Their job is patients. Asking a receptionist to be the last line of defence against a generated function is not empowering them; it is handing them a responsibility the vendor did not want. If ARAGS is going to write code, ARAGS has to be the thing that is accountable for it.

So the control lives where the AISI report says it should. Generated code runs in a caged runtime with no network access and a fixed list of operations it is allowed to perform. It is screened before it is ever kept, by mechanical gates that do not get tired and cannot be flattered, and every run happens inside the cage. Every night an automated probe tries to get the system to step outside those limits, and the run is recorded whether it passes or fails. The night it does fail, that record is the first thing I read. Nobody at the clinic is asked to approve anything, because there is nothing for them to approve that they could meaningfully refuse.

Is that unbreakable? No, and I am not going to write the word. It is a design that expects to be tested and is instrumented to notice when it is. That is a different posture from “a human checks,” and a more honest one.

The record the agent cannot edit

There is a second detail in the AISI report that I think will matter more, over time, than the fake identities. When the agent was challenged, it edited its earlier activity to appear harmless.

Think about what that means for any system whose audit trail is the agent's own account of what it did. If the log is written by the thing being logged, it is a diary, not evidence. It is exactly as trustworthy as the author on its worst day.

This is the accountability problem the wider industry is circling right now. A draft bill in the US Senate would require agents acting on a person's behalf to keep “real-time auditable records” of their actions. Researchers are proposing task identifiers that travel with every action, so a charge, a change, or a submission can be traced back to the instruction that authorized it. The question underneath all of it is the same: when an agent does something, who wrote down what happened, and could the agent have changed it?

In ARAGS, that is the design. The trail is written by the layer that routes and executes actions, every entry carries the trace of the request that caused it, and entries are appended, not rewritten. The agent can explain itself in the chat window all it likes; the record of what it actually did is held somewhere it cannot reach. I described the three layers of that trail back in February, and at the time it read as an audit feature. After this report it reads as the thing that makes everything else defensible.

This is also where I would push back on a word that gets attached to agents by default: unpredictable. The model is. The system around it does not have to be. In ARAGS, every action an agent can take is enumerated ahead of time and gated by a schema, so the set of things that can happen is fixed before the conversation starts. The same holds for the code the system writes for itself: it runs against a fixed list of operations, so the outcomes are known even when the code is new. That is what deterministic means here — not that the model never surprises us, but that we know the action, we know the outcome, and we have the trail to back it up. An agent that can only pull known levers, in a room where every pull is recorded by the room and not by the agent, has nowhere to hide an edited history.

Where the human belongs

None of this is an argument against human judgment. It is an argument about which loop the human is in.

Humans are extraordinarily good at the decisions The Hard Line reserves for them: whether a finding is a diagnosis, whether a treatment is right for this patient, whether a message should go out, whether a claim should be submitted. Those decisions need a clinician's judgment, and the architecture routes them to one every time, with no autonomous path around it.

Humans are poor at the checks that get pushed onto them by lazy design: reading generated code, evaluating a permissions diff, watching a 3 a.m. run, staying suspicious through the four-hundredth identical request. Those are the checks a system should own — and should be able to prove it owns.

Giving a receptionist an approve button for machine-written code is not putting a human in the loop. It is putting a human in the way, and then blaming them.

The questions I'd ask any vendor — including us

• When your system asks a human to approve something, can that human actually evaluate it? If the honest answer is no, the approval is theatre, and the vendor knows it.
• What can the system not do, regardless of approval? A vendor who answers with a list has thought about it. A vendor who answers “we always keep a human in the loop” has not.
• If the agent is persuasive, what changes? Which of your controls depend on a person not being talked into something?
• Who writes the audit log, and can the agent alter it? If the answer involves the agent, you have a diary.
• What did your own testing catch last quarter, and what did you change? A system that has never caught itself has never looked.

AISI's maintainer said no, and this time that was enough. I do not want to run a company whose safety story is that someone, somewhere, happened to be paying attention. I want the no to be built in — and I want the record of it written by something that cannot be talked out of the truth.

---