A RAG pipeline can fail in two completely different places, and a single end-to-end score will not tell you which one broke. The practical approach is to score retrieval and generation separately: first measure whether the retriever surfaced the documents that actually contain the answer, then measure whether the generated answer stayed faithful to the context it was given. Only after both components pass should you run end-to-end tests.
This guide is for technically curious beginners and intermediate practitioners who already have a working RAG prototype — whether it calls a hosted API or runs a local model — and now need a repeatable way to tell whether the system is getting better or worse. The emphasis is on metrics you can compute, datasets you can build, and failure patterns you can diagnose.
Why retrieval and generation must be scored separately
RAG systems are multi-component architectures where failures at one stage cascade into others. As the Anyscale RAG evaluation documentation puts it, a retriever that floods the context with irrelevant documents can cause hallucinations even with a capable generator, while a retriever that misses critical information forces the generator to produce incomplete answers.
The same documentation makes a second point that is easy to overlook: traditional NLP metrics such as BLEU and ROUGE cannot measure retrieval quality or assess whether generated answers are faithful to retrieved context. A fluent answer that scores well on n-gram overlap can still be fabricated. Evaluating retrieval and generation separately, before testing end-to-end performance, is what lets you isolate bottlenecks and attribute failures correctly — determining whether a poor response came from finding the wrong information or from using the right information incorrectly.
Retrieval metrics: what to compute and when
Retriever evaluation measures how effectively the system identifies and ranks relevant documents, and retrieval performance sets the upper bound for final answer quality. Classical information-retrieval metrics require a ground-truth dataset containing queries and labeled relevant documents, which classifies each document as relevant or irrelevant.
Precision@k measures the proportion of relevant documents in the top k retrieved results — the number of relevant documents in the top k divided by k. Use it when accuracy matters more than completeness: when a user would rather see fewer, better sources than a longer list padded with noise.
Recall@k measures the proportion of all relevant documents that the system retrieved in the top k results — the number of relevant documents found in the top k divided by the total number of relevant documents. It is critical when missing information is costly, because a document that never reaches the context window cannot be used by the generator no matter how capable that generator is.
Ranking-sensitive metrics such as mean reciprocal rank (MRR) and normalized discounted gain (nDCG) are also standard in information retrieval when the position of the first relevant result matters more than the raw count. They are useful complements, but they do not replace Precision@k and Recall@k, which answer the simpler questions of how much of what was returned was useful, and how much of what was useful was returned.
Building the evaluation dataset
The Google Cloud guidance on RAG evaluation starts with two artifacts. The first is a test dataset of high-quality questions, assembled so that it covers a broad subset of the underlying data and includes variations in phrasing and question complexity that match real-world use cases. The same guidance recommends consulting stakeholders and end users here, to ensure the quality and relevance of that dataset.
The second artifact is a golden reference dataset of desired outputs. Google Cloud notes that although some metrics can be calculated without a reference dataset, having a set of known-good outputs allows a more comprehensive and nuanced range of evaluation.
In practice, the golden set is the part teams skip and later regret. It is also the part that decays fastest: as the underlying corpus changes, labels that were correct six months ago may now point at deleted or rewritten documents. Treat the labeled set as a maintained asset with an owner, not a one-time deliverable.
Answer faithfulness and generation quality
Faithfulness — sometimes called groundedness — asks whether every claim in the answer is supported by the retrieved context. Answer relevance asks a different question: whether the response actually addresses what the user asked. These are independent. A response can be perfectly faithful to its context and still useless because the retriever supplied the wrong context, and a highly relevant response can still contain a fabricated detail.
Reference-based metrics compare the generated answer against a golden output. Reference-free metrics assess the answer against the retrieved context alone. Google Cloud’s framing is useful here: reference-free metrics are cheaper to run at scale, while a known-good output set produces a more comprehensive and nuanced evaluation. The common compromise is to build a small golden set for high-stakes queries and rely on reference-free faithfulness checks everywhere else.
An illustrative scoring loop
The scenario below is illustrative, not a measured result from a specific deployment.
- Assemble a question set that mirrors real usage, including short queries, multi-part questions, and questions whose answers span several documents.
- For each question, label which chunks in the corpus are genuinely relevant. This is the ground truth that retrieval metrics depend on.
- Run retrieval only, log the top-k chunk identifiers, and compute Precision@k and Recall@k per question as well as in aggregate.
- Run generation with the retrieved context held fixed, then score faithfulness and answer relevance.
- Record per-question results, not just averages, and keep the raw retrieved chunks so a failure can be replayed later.
Suppose a team builds RAG over internal documentation with a top-k of 5. On a 40-question set, recall@5 is 0.92 but precision@5 is 0.34. That pattern says the retriever almost always finds the right document but buries it among irrelevant ones, so the likely fix is reranking or a tighter similarity threshold rather than a better generator. Now suppose instead that recall@5 is 0.55 and faithfulness is 0.95. The generator is behaving; the retriever is missing nearly half the answers, and no amount of prompt engineering will repair that.
A triage table for common failure patterns
| Observed symptom | Likely stage | First check |
|---|---|---|
| Answer contradicts the source documents | Generation | Confirm the correct chunk is actually in the retrieved context; if it is, the problem is faithfulness, not retrieval. |
| Answer is vague or incomplete | Retrieval | Recall@k — the needed chunk may never have reached the context window. |
| Answer is fluent but cites irrelevant material | Retrieval, then generation | Precision@k first, then check whether the prompt instructs the model to rely on the supplied context. |
| Good average score, bad individual answers | Test set design | Inspect per-question results; aggregate averages hide outliers and rare query types. |
Automating the loop and catching silent failures
Google Cloud’s guidance is blunt about the stakes: RAG systems that are not thoroughly evaluated lead to silent failures, which can undermine the reliability and trustworthiness of the system as a whole. The recommended remedy is a transparent, automated evaluation framework in which metrics are calculated in a rigorous, automated, and repeatable fashion.
That means the evaluation should run like a test suite rather than a one-off audit. Any change to chunking strategy, embedding model, index parameters, prompt template, or generator model can move retrieval quality without touching a line of application code. A regression run on each of those changes is what turns evaluation from a report into a control.
Limitations and tradeoffs
- Labeled relevance data is expensive to produce and goes stale as the corpus changes.
- Using a language model as a judge is convenient but inherits the judge model’s biases; validating the judge against a sample of human labels is the usual mitigation, and a locally hosted judge trades privacy for judge quality and extra hardware.
- Generation is nondeterministic. When comparing two configurations, fix the sampling settings or run several samples per question.
- Precision and recall trade off against each other. Raising k usually improves recall while lowering precision and diluting the context the generator receives.
- All of these are proxies. A strong score on a small test set is evidence, not proof, of production quality.
Where local models and WordPress fit
If the corpus is private, sending queries and retrieved chunks to a hosted judge may be unacceptable on its own terms. A locally hosted judge model keeps the entire evaluation loop on-premises, and the retrieval metrics — Precision@k, Recall@k, MRR, nDCG — apply unchanged because they depend on labels rather than on any particular model.
For a RAG assistant over WordPress content, the corpus changes every time someone publishes, edits, or unpublishes a post. Running the evaluation suite after content updates, and pinning a snapshot of the index for comparison, prevents ordinary editorial work from silently degrading answer quality. This is a design suggestion rather than a tested configuration, but the failure mode it guards against — a corpus that drifts away from its labels — is the most common way an evaluation suite quietly stops meaning anything.
A practical next step
Start smaller than feels comfortable: roughly 30 questions, labeled relevant chunks, Precision@k and Recall@k on retrieval, faithfulness on generation, and per-question logging. Commit that suite to continuous integration so it runs on every change to the pipeline. Then expand coverage in the places where failures cluster, and re-label the golden set on a schedule. If you want a second opinion on your setup, share your environment, your metric definitions, and a representative failure in the community — concrete retrieval logs make for far more useful discussion than a general question about accuracy.
FAQ
Can I evaluate a RAG pipeline without a labeled dataset?
Partly. Some metrics can be calculated without a reference dataset, but retrieval metrics such as Precision@k and Recall@k require a ground-truth dataset containing queries and labeled relevant documents. If you have no labels at all, start by labeling relevance for a small question set rather than trying to infer retrieval quality from answer quality alone.
Why not just use BLEU or ROUGE to score answers?
Because they measure surface overlap with a reference text, not whether the right documents were retrieved or whether the answer is faithful to the retrieved context. A fabricated answer can still score well on n-gram overlap, which is why specialized metrics per component are needed.
What is the difference between faithfulness and answer relevance?
Faithfulness asks whether the claims in the answer are supported by the retrieved context. Answer relevance asks whether the response actually addresses the user's question. A response can be faithful to its context yet irrelevant to the question if retrieval supplied the wrong documents.
How many test questions do I need?
The available sources do not specify a number. The guidance emphasizes coverage — a broad subset of the underlying data, with variation in phrasing and question complexity that matches real-world use cases — and recommends involving stakeholders and end users in assembling the set. Coverage and representativeness matter more than a fixed count.