Most RAG demos follow the same script: embed a handful of clean documents, ask a question the documents obviously answer, and watch the model stitch together a confident response. It works. It also tells you almost nothing about how the system behaves once real users, real documents, and real edge cases show up.
Here are the failure modes that actually show up in production.
Retrieval quality, not model size, is the bottleneck
Teams routinely swap in a bigger, more expensive model to fix answers that are wrong for a much simpler reason: the retriever handed the model the wrong context in the first place. No model — however large — can answer correctly from irrelevant passages. Before upgrading the model, instrument retrieval itself: log what was retrieved, not just what was generated.
Chunking strategy is a design decision, not a config default
Fixed-size chunking (e.g. 512 tokens with 50-token overlap) is a reasonable starting point, not a finished architecture. It routinely splits tables mid-row, separates a claim from the citation that supports it, and breaks step-by-step instructions across chunk boundaries. Chunking should follow document structure — headings, sections, list boundaries — not just token counts.
Stale indexes produce confidently wrong answers
A vector index built once at launch quietly drifts from reality as source documents change. The system doesn't fail loudly; it keeps answering fluently, just from outdated context. Retrieval pipelines need the same "last updated" discipline as any other data pipeline — reindexing on a schedule or on change, not on a support ticket.
Common production algorithms behind a retrieval layer
- Hybrid retrieval (keyword + vector search, merged and re-ranked)
- Cross-encoder re-ranking on the top-N candidates
- Query rewriting / expansion before retrieval
- Semantic caching for repeated or near-duplicate queries
The fix is architectural, not a bigger prompt
Every failure mode above gets worse under a "just improve the prompt" strategy and better under an architectural one: better chunking, hybrid retrieval, re-ranking, freshness monitoring, and evaluation that runs continuously — not once, at launch, on a demo dataset.
That's the difference between a RAG demo and a RAG system built to hold.
