Anatomy of a safe RPC
One call path wearing every stability pattern: timeout, retry budget, circuit breaker, bulkhead, hedged reads, graceful fallback and deadline propagation.
What this design does
Most outages travel over a single unprotected remote call. This template is that call, armored: the gateway stamps a deadline that every hop honors and shrinks, retries are budgeted instead of infinite, the flaky dependency sits behind a circuit breaker inside its own bulkhead pool, reads hedge to a second replica when the first is slow, and when everything fails a cache serves the last good answer instead of an error page. The pattern badges on each edge aren't decoration — they ARE the design.
The patterns it composes
- Deadline propagation — 1s at the edge becomes 800ms downstream — no hop works for a caller that already gave up
- Retry with budget — bounded, jittered retries — a retry budget caps amplification at the worst moment
- Circuit breaker — the flaky PSP trips open after repeated failures; probes close it again
- Bulkhead — a dedicated pool means a slow PSP can drown 20 threads, never the whole API
- Hedged request — catalog reads fire a backup request to replica B at p95 — tail latency, tamed
- Graceful fallback — stale catalog from cache beats a 500 — degraded is a feature
Import it, attach your own numbers, and verify the design against failures — or read the engineering notes.