Production RAG platform
Retrieval-augmented generation beyond the demo: hybrid dense+sparse retrieval, a re-rank stage, ACL-aware filtering, a semantic cache and freshness re-indexing.
What this design does
Demo RAG is a vector search and a prayer. Production RAG is a pipeline: documents are chunked and embedded continuously (with a freshness loop re-indexing only what changed), queries hit BOTH a dense vector index and a sparse keyword index, a cross-encoder re-ranks the fused candidates, and retrieval is filtered by the asker's permissions — the model can't leak a document the user couldn't open. A semantic cache short-circuits the questions everyone asks. The LLM is the last, smallest box.
The patterns it composes
- Chunk-and-embed pipeline — documents become retrievable passages continuously, not in a one-off script
- Hybrid retrieval (dense + sparse) — vectors catch meaning, BM25 catches exact terms — fused, they cover each other
- Re-rank stage — a cross-encoder reorders the top-50 into a top-8 worth paying tokens for
- ACL-aware retrieval — permissions filter at query time — the index answers only what THIS user may see
- Semantic cache — similar-enough questions reuse yesterday's answer at 1/1000th the cost
- Freshness / incremental re-index — changed docs re-embed within minutes; stale answers are a data bug, not a model bug
Import it, attach your own numbers, and verify the design against failures — or read the engineering notes.