What actually breaks when you put RAG in production

Every RAG demo looks the same: ask a question, watch relevant chunks get retrieved, watch the model weave them into a clean answer. Production looks nothing like this.

The first failure mode is retrieval quality silently degrading as your corpus grows. Cosine similarity on embeddings works fine at 10,000 documents and gets noticeably worse at 500,000, not because the model got worse, but because the neighborhood of similar enough vectors gets crowded with near misses.

The second is citation drift: models confidently attributing a claim to a source that does not actually support it. We have found the fix is not a better prompt, it is a separate verification pass that checks each claim against its cited chunk before the response ever reaches a user.

The third, and most avoidable, is treating retrieval and generation evaluation as one metric. We score them separately, retrieval recall against a hand-labeled query set, and generation faithfulness against the retrieved context, because a system can nail one and quietly fail the other.