Lambda architecture warehouse
A batch layer for accuracy and a speed layer for freshness, merged at query time — with a medallion lake underneath.
What this design does
Lambda accepts two codebases to get both correctness and freshness: the batch layer recomputes gold tables from the bronze archive nightly, the speed layer approximates the last few hours from the stream, and the serving layer merges them. Backfills are trivial — fix logic, rerun batch, the speed layer's window of influence shrinks to hours.
The patterns it composes
- Lambda architecture — batch (accurate, slow) + speed (fresh, approximate) merged at read
- Medallion (bronze/silver/gold) — raw → cleaned → business-ready tables in the lake
- Backfill + replay — batch recompute overwrites gold; speed layer covers the gap
- Idempotent sink — both layers upsert — merge never double-counts
Import it, attach your own numbers, and verify the design against failures — or read the engineering notes.