LLM serving gateway
One gateway for all model traffic: cheap-first routing cascade, fallback models, hedged requests, prefix/KV caching, continuous batching and a hard budget governor.
What this design does
Every team calling model APIs directly is a cost incident waiting to happen. This gateway centralizes the discipline: a router sends the easy 80% to a small fast model and escalates only what needs the frontier model; shared prompt prefixes skip prefill via KV cache; a continuous batcher keeps GPUs saturated; slow requests hedge to a second replica; and a budget governor enforces per-team quotas before the bill does. When the frontier model is down, the cascade degrades gracefully instead of erroring.
The patterns it composes
- Model router / cascade (cheap-first) — a small model handles the easy majority; escalation is the exception
- Fallback model — frontier outage → degrade to the small model, never to a 500
- Request hedging across replicas — p99 stragglers race a second replica; first token wins
- Prefix / KV-cache reuse — shared system prompts skip prefill — latency and dollars both drop
- Continuous / in-flight batching — requests join running batches; GPUs never idle between calls
- Budget / deadline governor — per-team token budgets enforced at the gate, not discovered on the invoice
Import it, attach your own numbers, and verify the design against failures — or read the engineering notes.