How to Evaluate a RAG System
Quick answer: Build a fixed evaluation set of representative questions, each with a known-good answer and the passages that should support it. Score retrieval and generation separately: retrieval on whether the right passages came back, generation on whether the answer is correct and grounded. Run the set on every prompt, model, or index change so you can tell improvement from regression.
Most RAG systems are tuned by vibes. Someone changes the chunk size, tries three questions, decides it feels better, and ships. Then quality drifts and nobody can say when or why. An evaluation set replaces the vibes with a number.
Build the evaluation set first
Before tuning anything, assemble 50 to 100 questions that reflect real use. For each one, record the correct answer and the source passages that should be retrieved to support it. Cover the full range:
- Simple single-document lookups
- Questions that need two or more documents combined
- Questions with terms, names, or codes that only keyword search will catch
- Questions whose answer is not in the corpus, where the right behavior is to say so
That last category is the one teams skip and regret. A system that confidently answers questions it has no source for is worse than one that declines.
Score retrieval and generation separately
They fail independently, so measure them independently.
| Layer | What you are asking | Metrics |
|---|---|---|
| Retrieval | Did the passages needed to answer show up? | Recall at k, precision at k, mean reciprocal rank |
| Generation | Given those passages, is the answer right and grounded? | Answer correctness, faithfulness to context, correct refusals |
If retrieval recall is low, no amount of prompt tuning helps: the model never saw the answer. Fix retrieval first (chunking, hybrid search, re-ranking) and only then work on the generation prompt.
How to grade answers at scale
Exact match works for short factual answers and structured fields. For free-text answers, an LLM judge scoring correctness and faithfulness against the reference is practical, but calibrate it: have a human label a sample, check the judge agrees, and re-check when you change the judge model. Keep some deterministic checks that do not depend on a model at all.
Keep it honest over time
- Run the full set on every change to the prompt, model, embedding model, chunking, or index. Treat a drop as a regression to investigate, not a rounding error.
- Grow the set from production failures. Every real miss that reaches a user becomes a new evaluation case so it cannot silently return.
- Watch for overfitting to the set. If scores climb but users do not notice, the set has stopped representing real use. Refresh it.
For systems that run in a controlled environment, this harness is also how you prove the system still behaves after a model or weight update.
The takeaway
A RAG evaluation set is cheap to build and it is the difference between engineering and guessing. Separate retrieval from generation, include unanswerable questions, run it on every change, and feed it from real failures.
If your RAG system works in a demo but you cannot tell whether changes help, Eximo builds the evaluation layer.
Frequently asked questions
How do you evaluate a RAG system?
Build a fixed set of representative questions with known-good answers and the passages that should be retrieved. Score retrieval and generation separately: retrieval on whether the right passages came back, generation on whether the answer is correct and grounded in them. Run the set on every change.
What metrics matter for RAG?
For retrieval: recall (did the needed passage appear in the top results) and precision (how much of what came back was relevant). For generation: answer correctness, faithfulness to the retrieved context, and how often the system correctly declines when the answer is not present.
How big should a RAG evaluation set be?
Start with 50 to 100 questions that cover the real range of use: easy lookups, multi-document questions, edge cases, and questions with no answer in the corpus. Grow it every time a real failure slips through in production.
Can you use an LLM to grade RAG answers?
Yes, an LLM judge is practical for scoring correctness and faithfulness at scale, but calibrate it against human labels on a sample first, and keep some exact-match or structured checks that do not depend on a model.
Building something like this?
Eximo builds AI ingestion pipelines, multi-agent systems, and production RAG for regulated and restricted environments. Tell us what you're working on.
Schedule a consultation