Retrieval-augmented generation has a reputation problem. The demo is a five-line script that answers questions about a PDF, so everyone assumes the hard part is done. In production, the five-line script is about 5% of the work, and most of it doesn't involve the model at all.
This is what I've learned actually deciding whether a RAG system is good enough to ship.
The retriever is the product
If retrieval returns the wrong passages, no amount of prompt engineering will save the answer. The model can only reason over what you hand it. So I treat retrieval as the primary system and the generation step as a formatting layer on top.
Concretely, that means spending time on:
- Chunking strategy — semantic, heading-aware chunks beat fixed character splits almost every time.
- Metadata filters — date, source, tenant and permissions should narrow candidates before similarity ever runs.
- Hybrid search — combine keyword (BM25-style) with vector search; pure embeddings miss exact terms like IDs and names.
- Reranking — a cheap cross-encoder rerank on the top 50 candidates fixes a surprising amount of noise.
My default stack: hybrid retrieval → rerank top 50 → pass the best 4–6 chunks. It's boring, and it wins.
You cannot improve what you don't measure
"It seemed to answer well" is not an evaluation. Before optimising anything I build a small golden set: a few dozen real questions with known-correct answers, including the ugly ones — ambiguous queries, questions with no answer in the corpus, and questions that require combining two documents.
Then I track two separate numbers:
- Retrieval recall — did the correct passage appear in the top k?
- Answer faithfulness — is the generated answer supported by the retrieved text, with no invention?
Splitting these two is the single most useful debugging move. When an answer is wrong, you immediately know whether to fix retrieval or the prompt — and usually it's retrieval.
Guardrails are not optional
A RAG system that can't say "I don't know" is a liability. Production guardrails I always ship:
- Grounding check — refuse to answer when the top similarity score is below a threshold.
- Citations — every claim links to its source passage, so users can verify.
- Permission filtering — a user must never retrieve a document they can't open directly.
- Output validation — schema-validate structured responses before they reach the app.
The permission one is a security bug waiting to happen if you skip it. Filter at the data layer, not in the prompt.
Cost and latency are design constraints
Retrieval adds work, and generation is the expensive part. Two habits keep this in budget: cache embeddings and never re-embed unchanged documents, and cap context — the best four chunks, not the best forty. Feeding the model more text is slower, pricier, and often less accurate, because you dilute the signal.
For steaming, I stream tokens so the user sees progress immediately, and I run retrieval and any cheap pre-processing in parallel where possible.
When RAG is the wrong tool
RAG is not a universal answer. If the task is classification, extraction, or summarisation of a single known document, you don't need retrieval — just pass the document. If the knowledge is a stable, structured database, query the database. Reach for RAG when the answer is spread across many documents and changes over time.
The honest summary
Good RAG is mostly information-retrieval engineering: chunk well, retrieve hybrid, rerank, measure recall and faithfulness separately, filter by permission, and make the system comfortable saying "I don't know." The model is the easy part. Everything around it is what makes it production-grade.