Deployed Is Not Operational: The Session That Made 24 Agents Actually Talk

The broker had 24 agents registered. Five were reporting heartbeats. The fleet dashboard said "connected." And every single request from Mission Control to the broker returned 401 Unauthorized.

This is the moment most engineers celebrate a launch and move on to the next feature. The architecture is sound. The containers are running. The tests pass. Ship it. But the system doesn't work — not because of a design flaw, but because of one missing auth variable in a docker-compose file. One line, never injected. That's the entire difference between a distributed agent fleet and an expensive collection of isolated processes.

I spent an entire session closing these gaps. No new features. No architecture changes. Just making the thing I'd already built actually function as a system. It was the most important session of the entire Principal build.

The Gap Nobody Talks About

There's a phase in every system build that has no name. It sits between "deployed" and "operational" — a dead zone where every component works in isolation but fails in composition. The broker passes its test suite. Mission Control passes its test suite. Put them together and you get 401s, because the auth token that exists in the broker's config was never injected into MC's runtime environment.

This pattern repeated across every subsystem. The heartbeat bridge was monitoring five services but using the wrong service labels for two of them — close enough to look right at a glance, different enough to never match. The label existed. The service existed. They just didn't match. Two services were invisible to monitoring for no reason other than a string mismatch.

The kill switch could halt trading at four escalation levels. But there was no POST /halt/resume endpoint. You could stop everything. You couldn't start it again without redeploying. A kill switch without a resume is a self-destruct button, and nobody noticed until the first drill.

Operational integrity — the property of a system where every component not only works but communicates, authenticates, and recovers correctly in the actual deployment topology — is distinct from correctness. You can have 98% test coverage and zero operational integrity. I know because I had exactly that: 430 tests passing, and a fleet that couldn't execute a single cross-service request.

Two Failure Modes, Two Mechanisms

The most useful finding came from a controlled tunnel failure test. I stopped the infrastructure tunnel process on the Mac Mini and watched what happened.

launchd restarted it before the monitoring system even noticed. Process crashed, PID changed, traffic resumed. Sentinel's next health check at 60 seconds found everything healthy. Problem solved, right?

No. Because that's not the failure mode that actually hurts.

The 14 historical tunnel incidents averaged 32 hours of downtime each. Not because the process crashed — because the process was alive but broken. Running, consuming resources, reporting as healthy to process-level monitors, but not routing traffic. The URL file goes stale. Connections time out. The dashboard says green. The system is dead.

This maps directly to a principle from fault-tolerant systems design: failure mode separation. You don't build one resilience mechanism and assume it covers everything. You enumerate the distinct ways a component can fail, then build a specific detection and recovery path for each. launchd handles crash recovery. Sentinel's file_freshness remediation handles zombie processes. Neither is sufficient alone. Together, they cover the space.

The 36.8% current autonomous resolution rate is a lagging indicator — it includes 30 days of history before the new remediation paths existed. The mechanisms are correctly targeted. The metric will catch up. This is an important distinction for anyone building observability: your trailing metrics will make new fixes look ineffective for exactly one measurement window. Don't rip them out because the dashboard hasn't caught up to reality.

The Drill That Proved the System

A kill switch that has never been tested is not a kill switch. It's a comment in your architecture diagram.

I ran two drills back-to-back. Level 1: halt a single trading pair (DOGE-USD). Incident activated, pair stopped, resume endpoint called, clean recovery. Level 2: full trading halt across all pairs. Four daemons reported stopped. Incident activated. Resumed. Clean.

One caveat surfaced immediately: Level 2 calls launchctl stop on the local Mac Mini for trading daemons that actually run on a remote machine. The command succeeds as a no-op — it doesn't fail, it just doesn't do anything. The real stop-via-SSH path only exists at Level 4. This is the kind of gap that only appears during a drill, never during a code review. The function signature is correct. The behavior is wrong for the deployment topology.

This is why you drill. Not to verify the happy path — your tests do that. You drill to discover the gaps between your mental model of the system and its actual behavior in production topology. Every drill produces findings. If it doesn't, your drill isn't realistic enough.

The Unsexy Work That Makes Systems Real

Here's what the session actually produced: one environment variable added, three container labels corrected, one REST endpoint created, one CVE patched (Starlette GHSA-7f5h via FastAPI bump), two kill switch drills executed, and 15 Asana stories moved to DONE. The security scan went from FAIL to PASS on dependency CVEs.

None of this is architecturally interesting. None of it would make a good conference talk. All of it was blocking the system from being operational.

The engineering industry has a bias toward the creative phase — designing architectures, choosing technologies, writing the core logic. The phase after that, where you methodically verify that every seam between components actually conducts, gets treated as grunt work. It isn't. It's where operational integrity is manufactured, one corrected label and one missing token at a time.

After this session, the broker serves 24 agents with 5 active heartbeats. Mission Control connects cleanly. The kill switch halts and resumes. The security scan passes. 430 tests, 98% coverage, and — finally — a system that works as a system, not just as a collection of passing test suites.

The distance between "deployed" and "operational" is not a technical gap. It's a discipline gap. And the only way to close it is to sit down, run every path, and fix what's broken — with no new features to show for it at the end.