What Green Actually Proves: Two Weeks Inside a Self-Checking System

Two weeks ago I wrote about the system that checks itself — the nightly rehearsal our platform runs on its own production stage, graded against its own audit trail. Since then it has been quietly doing its job, and this post is about what it caught. I want to write it down honestly, because what a self-checking system catches says more about how clinical AI matures than any feature announcement could.

Full disclosure up front: every one of the findings below was in our own machinery — one in an instrument, one in an alarm, one in a two-month-old assumption. None of them surfaced in anyone's workday. That is not luck. It is the entire purpose of the instrument layer: this class of defect is supposed to surface at 1 a.m., inside a synthetic tenant, in front of a machine that never sleeps — not at 9 a.m. in front of someone who has patients waiting.

Six green nights that proved less than they looked

One of our newer subsystems keeps a small cache of verified artifacts, and every night an audit inspects that cache at rest: integrity, provenance, re-validation against current rules. For six consecutive nights it came back green.

Then, during a hardening review, we asked the uncomfortable question: what exactly do these six green nights prove? The honest answer was: that two files sat undisturbed in storage. Nothing more. The machinery that creates those artifacts — the full path from request to verified result — had been exercised exactly once, by hand, on the day it shipped. The audit was real, the green was real, and the green was quietly overclaiming. Storage was being proven nightly; the engine was being taken on faith.

So we built a second nightly instrument that fires the actual machine end to end: a hostile request that must be refused, a legitimate request that must succeed, the resulting artifacts verified, executed fresh and executed from cache, and everything cleaned up afterward — in an order deliberately chosen so the cleanup can never manufacture the first instrument's tamper signal. The at-rest audit was not wrong. It was partial, and partial checks are most dangerous precisely when they are green.

The new probe's first catch was the alarm itself

The new instrument's first live night produced a page at 1 a.m.: critical error, plus an automated investigation that burned itself out on a non-event. Here is what had actually happened.

The probe deliberately feeds one of the cached tools an input it should reject — because proving a tool is not inert requires trying inputs it may refuse. The tool refused it, exactly as designed. But the serving path logged that refusal at the severity our alerting forwards to a human. A successful defense had been classified as a critical incident, and it would have paged every night, forever.

At that point there are two possible fixes, and the choice between them is where I think engineering culture actually lives. You can make the probe polite — stop sending the rude input, and the pages stop. Or you can fix the classification. We fixed the classification: infrastructure faults still page; a wrong answer inside a sandbox that did its job logs below the paging line, with full detail retained for review. The rule was not even new — a sibling instrument already encoded it verbatim. The serving path had simply never been forced to agree with its own audit until an instrument made it.

The night after the fix: silence. Alarm fatigue is a first-class defect, and the remedy is calibration — never blindness. An alarm that cries wolf trains its operator to ignore it, which is a slower and more dangerous failure than the one it was built to catch.

The ghost we had been blaming on someone else

The third catch happened this week, in daylight, and it is my favorite because it corrected us.

For two months, automated dependency updates kept arriving with the same suspicious signature: a build manifest rewritten, dozens of entries deleted that the update had no stated business touching. We treated those updates as corrupt — procedurally the right call, since a change that rewrites files outside its declared scope should never be trusted — and screened everything by hand.

This week we turned that manual screen into a standing instrument: a structural check that compares every proposed manifest against the current one and refuses any deletion outright. On its first day it caught two real defects — the familiar deletion pattern, live, and a separate update that could never have installed, refused before it could break a deployment.

But the bigger catch came from investigating the flagged deletions with execution instead of reasoning. The deleted entries turned out to be vestige of our own upgrade two months earlier — leftovers nothing in the project referenced anymore. The build runs green without them; we proved it by building, twice. Every "corrupt" update had been correctly deleting entries we should have deleted ourselves, and two careful human reviews had concluded otherwise. The reviews reasoned. The build knew. We regenerated the manifest, and the ghost is gone for good.

The doctrine, four lines

Each of these findings sharpened a rule we now treat as standing engineering doctrine:

• Unexecuted code is unknown code. A subsystem can be deployed, healthy, and monitored for months without ever having completed one real request. Green infrastructure says nothing about function.
• An instrument that has never fired is itself unknown — and so is an alerting policy that has never been triggered. Test the testers.
• Fix observability first. Every hour a system cannot see itself is an hour its defects are unfindable by anyone.
• When an instrument raises a false alarm, recalibrate the alarm — never blind the instrument. Coverage is sacred; classification is negotiable.

The fifth question

In July I suggested four questions any clinic should ask an AI vendor — ours included. One of them was "can you show me what the system actually did?" After these two weeks I would add a fifth: "What did your own tests catch last month, and what did you change because of it?"

A vendor who answers "nothing — everything has been green" is describing either perfect software or unexamined green, and only one of those exists. The honest answer looks like a changelog of the checker itself: here is what our instruments caught, here is what we recalibrated, here is why last night's green means more than the green from a month ago. Robustness is not the absence of findings. It is findings landing where they are supposed to land — in the instrument layer, at night, in synthetic tenants — while the workday stays boring.

That is the standard we hold ourselves to, in public, with receipts. The system checks itself; this is what it found; this is what we did about it. Next month there will be more, and I intend to keep writing them down.