MLOps: training to serving
The full ML loop: one feature store feeding both training and serving, a model registry with lineage, champion-challenger shadow serving and drift-triggered retraining.
What this design does
Most ML failures aren't modeling failures — they're plumbing failures, and the worst one is training-serving skew: features computed one way offline and another way online. Here one feature store materializes the SAME definitions to both an offline store for training and an online store for sub-10ms serving. Models graduate through a registry with full lineage, new challengers serve in shadow against the champion on mirrored traffic, and a drift monitor watches live distributions — when the world changes, retraining triggers itself.
The patterns it composes
- Feature store (online + offline) — one definition, two materializations — the skew killer
- Training-serving skew mitigation — training reads point-in-time-correct features from the same registry serving uses
- Model registry & lineage — every model version knows its data, code and features — rollback is a pointer flip
- Champion-challenger / shadow model — the challenger predicts on real mirrored traffic; only the champion's answers count
- Drift-triggered retraining — input and prediction distributions are monitored; drift files the retrain ticket
Import it, attach your own numbers, and verify the design against failures — or read the engineering notes.