RAG evaluation: how to measure retrieval and answer quality

Retrieval-augmented generation (RAG) grounds a model's answers in your own documents. When a RAG system gives a wrong answer, the cause is usually one of two things: the right information was never retrieved, or it was retrieved and the model ignored or distorted it. Good RAG evaluation measures both separately.

Retrieval metrics

For each test question, label which documents or passages contain the answer. Then measure recall@k (did the relevant passage appear in the top k results?), precision (how much of what was retrieved is relevant?) and ranking quality such as MRR or nDCG. Low recall means fixing chunking, embeddings, filters or query rewriting — not the prompt.

Generation metrics

Faithfulness (or groundedness) asks whether every claim in the answer is supported by the retrieved context. Answer correctness compares the answer with a reference. Answer relevance checks that it actually addresses the question. Citation accuracy checks that cited sources really support the cited sentence.

Test the hard cases

Include questions whose answer is not in the corpus (the system should say it does not know), questions that need information from several documents, questions with outdated or conflicting sources, and questions phrased the way real users phrase them — including typos and vague wording.

Automated vs human scoring

Model-graded faithfulness checks scale well but should be validated against human judgments on a sample. Domain experts are essential in regulated areas such as legal, healthcare and finance, where a plausible but subtly wrong answer is the most dangerous kind.

Re-evaluate on every change

Changing chunk size, embedding model, reranker, prompt or base model can shift results in either direction. Keep a versioned RAG test set and re-run it before each release.

ZenXData designs RAG test sets and runs retrieval and answer-quality evaluations with expert human review.