In July I wrote about the system that checks itself before you wake up, and in August about what its green actually proves. This is the third post in that series, and it exists because the check learned a new trick this week: it makes a phone call.
What the call is
Every night, after it has ingested documents, run the assistant, filed a dictation and read the audit trail, the harness now places a call on each of our two synthetic clinics. Not a simulation of a call — a call, as far as the phone service can tell. It asks the service for a one-time call token, opens the same stream the carrier opens, speaks the carrier's handshake, and then plays our synthetic clinician's dictation clip as telephone audio: eight thousand samples a second, twenty milliseconds at a time, at the pace a person speaks. It waits for the greeting to finish before it starts, the way a polite caller would. Then it goes quiet for three seconds, the way a person does when they have finished a sentence, and hangs up.
The grading is mechanical, as it is everywhere in this system. Hard: audio came back, the call's session ended as a caller hang-up, and exactly one call record landed with a real duration in it. Soft, recorded but never the verdict: how long the greeting took to start, how long the model took to answer after the end of speech, and whether the interaction card appeared under the call's trace. The session and the record are deleted afterwards, so the synthetic clinic's history stays clean.
What it proves, and what it cannot
It proves the path. The carrier protocol is spoken correctly, the token admits the stream, the audio arithmetic runs both ways, the live model session opens and answers, the screening is in place, the record is written, and all of it happens on the deployed service, not on a developer's laptop. That is the thing a phone line can fail at in a dozen quiet ways, and now a dozen quiet ways are checked at one in the morning.
It cannot prove the conversation is good. The clip is a dictation about a tooth, not a patient asking for an appointment; the model's reply to it is whatever the model says to a clinician dictating at it, and the check does not judge that. A quality grade needs a judge, and I have written about why the judge in this system is never a model. The synthetic call is the right place to add one later — the clinics are synthetic, the clip is ours, no patient is in it — and it will be a second opinion, never the hinge.
What the first night caught
The run passed. It also surfaced three things, which is what passing with a full record looks like.
The first was a permission. The phone service writes the call's record to the clinic's silo and then archives a copy to the clinic's storage; the archive was refused, because the phone service had never been granted the right to write there. The old managed product never wrote one either, so nothing had ever noticed. The second followed from the first: the code counted the call only after the archive, so a failed archive meant an uncounted call, and the log said the whole call had failed to record when the record itself was fine. The count now follows the record, and an archive failure is its own line.
The third is the one worth being honest about. On the clinic that got the first call after a fresh deployment, the greeting took eight and a half seconds to start. On the second clinic, two minutes later on the same instance, it took one and a half. Seven seconds of cold start on the first call, which a patient would hear as a dead line. I read the logs, concluded the time was being spent building the live runtime's pieces — its modules, its model client, its screening client — inside the first call, and shipped a warm-up that builds them once at startup instead.
The fix was the wrong size
The warm-up works. It also takes four tenths of a second. The startup log said so the moment it deployed, which means the seven seconds were mostly somewhere else, and my reading of the logs was wrong. So rather than guess a second time, the service now stamps a timeline on every call's record: when the session was created, when the agent was built, when the live connection delivered its first message, when the first audio went out. The nightly record carries that timeline on every call from here on, so the next cold first call will say where the time went instead of me saying where I think it went.
That is the whole discipline, really. The check does not exist to make me look right. It exists to make the record right, and to make my next guess smaller than my last.
Want a phone line that is tested every night and tells you what it found? Apply for Beta Access.