Part 4 of 5 in the series “How We Built a Distributed AI Team.”
By now the team is a mesh of autonomous agents that talk to each other, not just to me. Earlier posts covered why we moved off a single orchestrator, how the agents became first-class members with their own credentials and memory, and how we gave them a shared brain so a distributed team wouldn’t become a room full of amnesiac silos.
This one is about the day one of those agents could, in principle, have taken the whole thing down — except it happened to a sister team first, and we got to watch.
The gift of someone else’s outage
A sister team runs an agent mesh of the same shape as ours: independent agents that trigger each other by mentioning each other in a shared chat thread. Agent-to-agent triggering is the whole point of a mesh. It’s also, it turns out, the whole danger.
One afternoon their mesh went from perfectly healthy to killall in about sixteen minutes. A single conversation thread spiralled into a runaway fan-out — agents waking agents waking agents — until the only thing that stopped it was killing every process on the box by hand. Not a polite ping-pong between two bots, but an exponential branching thing: each spawned session emitted several messages that each mentioned someone else, so every generation multiplied.
Nobody’s data leaked, nothing was destroyed. It was, in the most useful possible sense, a free fire drill — someone else paid for the lesson; we just had to be humble enough to take it. So before touching anything, we audited our own mesh against the exact failure that hit theirs. And the most important finding was not a bug in their system. It was a guard in ours that we’d been quietly trusting for months, that had never once actually worked.
Six things had to line up — and they did
The first discipline of a good post-incident read is to resist the single-cause story. Runaways are never one mistake. Theirs needed six independent failures stacked together, and every one is a thing a reasonable engineer might ship:
- The loop-breaker had been deliberately removed. The original rule — “don’t react to your own bots’ messages” — was deleted on purpose, to enable agent-to-agent triggering. The feature and the failure were the same line of code with and without a brake.
- A message could spawn its own author. No sender-exclusion, so a bot mentioning itself woke itself.
- No idempotency. Nothing was keyed on message ID, so a single at-least-once redelivery from the platform re-spawned a whole session.
- No throttle of any kind. No rate limit, no cooldown, no concurrency cap, no depth counter. Every routed mention became an unconditional new process.
- The prompt mandated output. Each agent was instructed to post an immediate acknowledgement and an @-mention when it finished — at least two trigger-bearing messages per run. The prompt itself was a recursion amplifier.
- A shared, fixed-name handoff file and stale context. Sessions read the oldest slice of a fast-growing thread, so they were blind to recent work and kept redoing finished tasks — generating still more messages.
And one lifecycle detail turned a bug into an outage: the sessions were spawned detached, as independent processes that outlived the thing that launched them. That’s why nothing short of killall worked. The listener could have been stopped cleanly; the hundreds of orphaned children it had already flung off could not.
Read that list again and notice something uncomfortable: not one of those six is stupid. Each is a defensible local decision. The disaster lives in the stack — which is why you can’t reason your way to safety one commit at a time. You have to look at the whole spawn path at once.
The one defense that makes the whole class impossible
Here is the single structural choice that meant the sister team’s failure mode could not physically happen in our mesh, and it’s almost embarrassingly simple.
We spawn agent sessions serially and blocking, with a hard timeout, as tracked child processes — never detached.
Our router runs one session at a time, waits for it to finish (or times out and kills it at three minutes), and only then considers the next. That’s it. That one property means the absolute worst case of a runaway loop is a slow loop — roughly one spawn every few minutes, fully visible and fully killable — instead of thousands of concurrent orphaned processes eating the machine.
It’s the difference between “a bug wastes some money” and “a bug takes down the box.” A detached, fire-and-forget spawn buys throughput and hands you an unkillable failure mode. For an autonomous mesh, I’ll take slow-and-killable over fast-and-detached every time.
(When we parallelise for speed later, this implicit rate cap vanishes and has to be rebuilt deliberately — but that’s the next post’s problem.)
The guard that had never fired
Now the part that actually humbled us.
Our mesh advertised a loop-breaker. On paper it was elegant: the router stamped every outgoing message with a little marker — a conversation ID, a hop count, a fan-width — and read it back off the next message to refuse extending a chain past a set number of hops. A self-propagating circuit breaker. We’d had it for months. We’d have told you, confidently, that our mesh could not loop.
Then, as part of the audit, we did something we should have done on day one: we ran a live two-bot test and actually watched the marker travel.
It didn’t.
When a bot posts through the chat platform’s connector and the message comes back to us through the subscription API, the platform did not preserve our embedded marker. It was stripped somewhere in the round-trip — so the hop count reset to zero on every single hop, and the conversation ID regenerated every time. Our celebrated circuit breaker had been reading a value that was freshly zero on arrival, forever. It had never once done anything.
No amount of code review would have caught this. The code was correct, the logic was sound, the unit tests passed. The marker was written exactly as designed and read exactly as designed — it just quietly ceased to exist in the few inches of network between “post” and “receive.” Only real traffic through the real transport revealed it.
The beat I most want you to take away: a guard that has never fired is a guard you cannot trust. Not “probably works.” Unknown. Until you have watched a defense actually trip under production conditions, you don’t have a defense — you have a comment in the code that describes one.
Bind loops on something the transport can’t take away
The fix followed directly from the failure. If the transport strips anything I inject and hope to read back, then I must never bind a safety limit to something I inject. I have to bind it to something the transport cannot strip because it’s load-bearing for the transport itself.
For us that turned out to be the thread root ID — the identity of the conversation. The platform can drop my custom markers, but it can’t drop the thing that tells it which thread a message belongs to; the whole system falls apart without it. So every loop bound in our mesh now keys on the stable thread root, not on a marker of our own invention: a root-keyed spawn breaker caps how many sessions one conversation can spawn in a rolling window, global and per-agent volume caps sit above that as a coarse ceiling, and message-ID idempotency kills the redelivery-respawn path from failure #3.
The cheapest guard reads the room before it spends a cent
We already fetch the recent messages of a thread before spawning an agent — the agent needs that context to do its job. That fetch is already paid for. So before spending a single token on a new session, we read that same message list and ask one question: how many bot messages in a row have there been since the last time a human said anything?
If that trailing run of consecutive bot-authored messages crosses a threshold — and the message that’s trying to trigger the new spawn is itself from a bot — we simply don’t spawn. We log it, drop one inert note into the thread (“paused a bot-to-bot loop here — reply to continue”), and exit before the expensive part begins.
I love this guard for three reasons that are really one. It survives a restart: a router that boots with empty counters would happily walk into a thread already thirty bot-messages deep, but this one reads the actual thread and bails on first contact. It survives parallelism: local counters have per-instance amnesia (two routers each allow “their” quota and the conversation quietly absorbs double), while thread state is shared by construction. And it tells runaway apart from real work: the signal is “the humans have gone quiet and the bots are only talking to each other,” so a single human message anywhere in the recent tail resets the count to zero and a healthy collaboration is never touched.
That’s the whole trick — it reads shared ground truth instead of a private counter. Every other guard trusts the router’s own bookkeeping; this one trusts the conversation itself.
Most of our guardrails were born, not designed
Here’s the honest meta-lesson, and the reason I’m writing this under my own name instead of pretending we designed it all up front. We didn’t. Almost every guardrail this team runs came from an incident, not a whiteboard: a transcript pipeline that had silently done nothing for months, evening work vanishing into a requeue bug, a sensitive document mis-filed into a shared channel — and this, a loop-breaker we’d trusted for months that had never worked. Each became a rule, a test, or a structural change after it bit, or after it bit someone close enough that we could learn for free.
I’ve made my peace with that. You cannot design a complete threat model for an autonomous multi-agent system up front, because the failures live in the interactions, and the interactions are exactly what you can’t fully predict. What you can build is the muscle to turn every incident — yours or a neighbour’s — into a permanent structural improvement, and to go looking for the guards you’ve been taking on faith.
The thesis, in two lines
If you take nothing else from this: prompt etiquette is not a safety layer. “Please don’t reply to yourself” and “post a nice acknowledgement when you’re done” are manners, not mechanisms. Bind your limits in orchestration code, on a key the transport can’t strip, spawn serially and killably, and read shared ground truth before you spend.
And then — this is the part almost everyone skips — verify your guards in production. Run the live test. Watch the fuse actually blow. Because the guard that scared us most wasn’t the one the sister team had removed. It was the one we thought we had.
Next in the series: getting all of this off my laptop and onto real, AWS-native infrastructure — where “spawn serially on one box” stops being a safety feature and becomes the bottleneck we have to re-engineer around.