The retry storm (anti-pattern)
What NOT to build: synchronized retries, an inner timeout longer than the caller's, a shared cache TTL and no jitter — watch one slow database become a full cascade.
What this design does
This is the outage you've lived through, drawn honestly. Ten thousand clients retry every 3 seconds with no jitter, so the load arrives in synchronized waves. The API's 30-second timeout outlives the client's 3 seconds, so servers keep working for callers that already gave up. Every cache key expires at the top of the hour, so the herd stampedes the database at once — and retries multiply traffic exactly when the system is sickest. Import it, then fix it: jittered backoff, a timeout ladder, retry budgets and stampede protection turn this diagram into the safe-RPC template.
The patterns it composes
- Retry storm (anti-pattern) — retries without jitter or budget multiply load ×4 during the failure
- Layered-timeout mismatch (anti-pattern) — inner 30s vs caller 3s — zombie work for callers that already left
- Thundering herd / dogpile (anti-pattern) — shared TTL expires every key at once; the miss-storm hits the DB together
- Cascading failure (anti-pattern) — the dying DB drags the API, whose queues drag Service B — failure travels the call graph
- Unbounded queue (anti-pattern) — Service B queues requests forever instead of shedding — latency grows until everything times out
Import it, attach your own numbers, and verify the design against failures — or read the engineering notes.