Real-time analytics (Kappa architecture)
One stream, one processor, one serving store — the log is the source of truth, and reprocessing is just replaying it from offset zero.
What this design does
Kappa deletes the batch layer: every event lives forever in the log, a single streaming job computes windowed aggregates with watermarks for late data, and an idempotent sink makes replays safe. Need to fix a bug or add a metric? Deploy the new job, replay the log, atomically swap the serving table. No second codebase, no batch/speed drift.
The patterns it composes
- Log as source of truth — the retained stream IS the database — everything else is a view
- Tumbling & session windows — aggregates computed per time bucket, closed by watermark
- Watermarks & late-data handling — events arriving after the watermark divert instead of corrupting closed windows
- Idempotent sink — upserts keyed by window make reprocessing safe to run twice
- Backfill + replay — new logic = new consumer group reading from offset 0
Import it, attach your own numbers, and verify the design against failures — or read the engineering notes.