01
Signal
Side services reported failed builds
Repeated failures appeared around probe/bootstrap services. That was treated as evidence about those services—not proof that the accepted production path had failed.
REAL INCIDENT-RECOVERY PROOF
Diagnose the smallest broken component—without rebuilding the healthy system around it.
A sanitized engineering case study from a real cloud-to-local automation incident. The objective was not to “make everything green” by force. It was to separate noisy failures from the actual production path, preserve accepted controls, and narrow the repair target.
RECOVERY SNAPSHOTCORE CHANNEL ALIVE
Production brokerSUCCESS
Agent heartbeatHTTP 200
Agent pollingHTTP 200
Auxiliary GitHub ingressHTTP 401
Interpretation: one auxiliary path failed authentication while the accepted broker + agent pull channel remained live.
Observe→Confirm→Isolate→Repair smallest component→Smoke→Regression
02
Observe
Core broker state checked independently
The production broker deployment was verified SUCCESS while the Windows agent continued sending heartbeat requests and polling for work with HTTP 200 responses.
03
Isolate
Auxiliary ingress tested separately
A read-only health request reached the GitHub auxiliary ingress but returned HTTP 401. The mismatch was isolated to that auxiliary authentication path instead of being misdiagnosed as a full-channel outage.
04
Protect
Healthy invariants were left untouched
No secret rotation, Guard weakening, browser-owner duplication, production rebuild, or broad redeploy was used as a shortcut.
WHAT STAYED FROZEN
Known-good controls were treated as assets, not obstacles.
✓ One Browser Owner policy preserved
✓ Native Guard preserved
✓ Accepted FIR-5 / FIR-10 baseline preserved
✓ Production broker left in place
✓ No primary-browser CDP shortcut
WHAT WAS DELIBERATELY AVOIDED
No broad changes just to make the symptom disappear.
× No secret export or casual rotation
× No Guard weakening
× No second browser owner
× No full architecture rebuild
× No repeated redeploy loop
THE ENGINEERING VALUE
A failed component is not permission to erase system knowledge.
The useful proof here is not that an incident happened. It is that the incident was reduced from “automation is down” to a narrower, evidence-backed statement: the broker and agent pull channel were alive; the auxiliary GitHub ingress had an authentication mismatch; the remaining automation gap was safe command invocation / browser-owner inspection.
This kind of narrowing protects uptime, security controls and already-accepted work while reducing recovery risk.
Internal URLs, secrets, ports, IDs and proprietary implementation details are intentionally omitted. This page exposes the diagnostic method and verified outcomes, not the private control stack.
WHEN RECOVERY NEEDS DISCIPLINE