Traffic on a fan engagement platform hit 1,500 OTP requests a minute.

Every call to SendGrid came back successful. No errors. No 429s. No rate-limit headers. The dashboard was green.

For two days, real users got nothing.

The design was fine. That was the problem.

We asked an AI to design the pipeline. It came back with per-user rate limits, global send caps in Redis, a FIFO queue with TTL, a breaker that trips when wait time exceeds five minutes, and a background worker that generates an OTP only at dequeue time.

That is a good design. It is also textbook, and textbook is what went wrong.

Every pattern in that list defends a boundary you can observe. Retries fire on errors. Backoff responds to 429s. Circuit breakers trip on failure rates. All of them are written against an API contract — and SendGrid’s contract said we were fine.

Failure one: the boundary we could not see

SendGrid accepted everything we sent. Gmail and Outlook were deferring our sending IPs, because a sudden flood from those IPs looks exactly like what spam looks like. Deferrals piled up. Backoffs snowballed. What should have been seconds became days.

Nothing in our system had a signal for this. The failure was one hop past the last thing we could instrument.

The first fix was blunt: cap what we hand to SendGrid so its onward delivery rate stays inside what Gmail tolerates for our IPs. Throttle ourselves to protect a reputation we cannot measure.

The real fix came later — provider-specific queues, roughly 800/min for Gmail and 400/min for Outlook, no single global choke point, plus a separate IP pool so transactional mail never queues behind marketing.

Even those numbers are soft. ESP limits move with domain reputation, IP reputation, spam rates and delivery errors. There is no constant to hard-code.

Failure two: the one the load test missed

At 100 requests a minute, clean.

At 1,000+, ten requests read “you are 23rd in queue” at the same moment and all ten wrote themselves to position 23. Positions climbed to 50. Then 100. Then nonsense.

A day of logs and staring at Redis like it might confess. The capacity check was a read-modify-write and it was not atomic. Only concurrency made that visible.

The fix was a Lua script. Not SETNX, not Redlock — we run Redis as a cluster on AWS, and distributed locks across slots are a problem I did not want to own. A Lua script executes atomically on the node holding the key, so the race stops existing rather than being guarded against.

That held 1.5k/min.

One correction while I am here, because the term gets thrown around: the “circuit breaker” we shipped trips when queue wait exceeds five minutes. A circuit breaker trips on downstream failure rate. Ours trips on user-facing latency. That is admission control. It is useful, and it would never have caught failure one, because nothing was failing.

The same class of mistake, in my own product

Venzio is a presence tool I built and GlobalNodes now runs on. A check-in calls ip-api.com to geolocate the request. Free tier: 45 requests a minute. The user waits on that call.

When ip-api returns 429, the check-in fails. The user sees “try again later.” A while later it works.

No timeout. No retry with backoff. No fallback to recording the check-in with the IP signal marked unavailable. And worse: a failed lookup and a genuinely mismatched IP land in the same bucket, so the data cannot tell “this person was somewhere else” apart from “we never found out.”

It has not hurt us, because we are one company and nowhere near 45 requests a minute.

That is not a design. That is a rate limit I have not reached yet.

What this means for agents

An agent calls external tools in a loop, so it inherits every failure mode above and multiplies it. Three things carry over directly.

A 200 does not mean the work happened. An agent that treats a successful call as a completed action will confidently report success it has no way to verify. That is the SendGrid failure with a language model on top of it.

A missing value and a wrong value are not the same value. Venzio’s bug is survivable in a form. In an agent it is not, because the agent will reason over whatever is in the field and produce a confident answer either way.

Degrade rather than stop. If one tool is unavailable, return what you have and name what you could not get. An agent that fails the whole request because one lookup timed out is less useful than one that hands back three-quarters of an answer and says so.

What it cost to learn

The AI got us about 60% of the way there. Every pattern it suggested was correct, and we still shipped a two-day outage — because those patterns guard the boundaries you can name, and production failures live at the ones you cannot.

I have now written 900 words about resilience, and my own product still has an unhandled 429 on its critical path.

It is fine at forty people. It stops being fine at four hundred.