AIForum.

How to Evaluate Local RAG Pipelines with Retrieval and Answer Metrics

On this page

THE SHORT ANSWER

Evaluating a local RAG pipeline means scoring two separate things: whether the retriever returned the right chunks, and whether the generator produced a correct, grounded answer. The practical method is to build a small labelled question set, compute deterministic retrieval metrics first, then add answer-level scoring with a judge model. This guide covers the metrics, the loop, and how to tell a retrieval failure from a generation failure.
Quick facts

Evaluating a local RAG pipeline means scoring two separate things: whether the retriever returned the chunks that contain the answer, and whether the generator turned those chunks into a correct, grounded response. Both failures look identical in the final output, but they require completely different fixes. The practical approach is to build a small labelled question set, run the pipeline against it, compute deterministic retrieval metrics first, and only then add answer-level scoring.

This guide is for practitioners running retrieval-augmented generation on their own hardware or with self-hosted models. It covers what to measure, how to build a reusable evaluation loop, and how to tell a retrieval problem from a generation problem.

Separate retrieval quality from answer quality

LangChain’s RAG evaluation tutorial describes a three-step workflow: create a dataset of questions and expected answers, run the application on those questions, and score the results with evaluators for answer relevance, answer accuracy, and retrieval quality (LangChain docs). Ragas frames the same loop as setting up an evaluation dataset, establishing metrics to measure RAG performance, building a reusable evaluation pipeline, and then analysing errors to improve the app (Ragas docs).

The structural point both workflows share is that retrieval quality and answer quality are different measurements. If the retriever never surfaces the passage that contains the answer, no amount of prompt engineering will fix the response. If the retriever surfaces the passage but the model ignores it, retrieval metrics will look healthy while answer metrics stay low. Reporting a single “RAG score” hides exactly the information you need to debug.

Build the evaluation dataset before tuning anything

Both documented workflows start with a dataset. LangChain’s tutorial uses questions paired with expected answers; Ragas uses an evaluation dataset as the first step of its improvement loop.

For retrieval scoring you additionally need to know which chunk, or which set of chunks, should have been retrieved for each question. Without that reference set you can still measure answer quality, but you cannot compute recall or precision, and you cannot tell whether a wrong answer came from missing context or from a bad generation step.

A practical dataset row for a local pipeline contains: the question text, the expected answer, the identifier of the chunk or chunks that contain the evidence, and optionally the source document and section. Keep the file in version control next to the pipeline configuration so that any metric change can be traced to a code or data change.

The tutorials above do not prescribe a dataset size, so treat any number as a starting point rather than a standard. A useful early target is a few dozen questions that cover the document types and query shapes you actually see in use, with at least a handful of questions you expect the pipeline to fail.

Retrieval metrics you can compute without a judge model

Retrieval metrics are deterministic. Given the ranked list of chunk identifiers your retriever returns and the reference identifiers for each question, you can compute them with ordinary code and no model calls at all. That makes them cheap enough to run on every change, including on a laptop with no GPU. The standard set is:

  • Recall@k — of the reference chunks for a question, how many appear in the top k results. With a single gold chunk, recall@k is 1 when that chunk is present and 0 when it is not.
  • Precision@k — the fraction of the top k results that are actually relevant. Low precision means you are filling the context window with noise.
  • MRR (mean reciprocal rank) — the reciprocal of the rank of the first relevant chunk, averaged across questions. It rewards getting the right passage near the top rather than merely somewhere in the list.
  • nDCG@k — a rank-weighted score that handles questions with several relevant chunks more gracefully than recall alone.

Report these at the k you actually pass to the generator. If your prompt includes the top five chunks, recall@5 is your operating number; recall@20 is diagnostic but is not the configuration you ship. Also log the retrieved text itself, not just the identifiers, because a chunk can be retrieved correctly and still be a poor answer source if the chunking split the relevant sentence in half.

Answer metrics and the judge problem

Answer-level metrics generally require a model to score the response. LangChain’s tutorial names answer relevance and answer accuracy among its evaluators; Ragas describes establishing metrics to measure RAG performance as part of its loop. The three categories most local pipelines track are:

  • Faithfulness (groundedness) — whether every claim in the answer is supported by the retrieved context. This is the metric that catches hallucination.
  • Answer relevance — whether the answer addresses the question that was asked, rather than a nearby question.
  • Answer correctness or accuracy — whether the answer matches the expected answer.

For a local pipeline the judge is usually another local model. This has two consequences worth planning for. First, judge quality sets a ceiling on measurement quality: a small judge model produces noisy scores, and that noise can easily exceed the difference between two configurations you are trying to compare. Second, judge runs consume time and VRAM, so it is usually practical to run deterministic retrieval metrics on every change and answer metrics on a scheduled or pre-release basis.

A reproducible evaluation loop

Evaluation only becomes useful when it is repeatable. The loop below combines the documented patterns from all three sources.

  1. Freeze the configuration. Record chunk size and overlap, embedding model and version, retriever type and k, any reranker, generation model and quantization, context length, and sampling settings. A metric without an accompanying configuration is not reproducible.
  2. Version the dataset. Questions, expected answers, and reference chunk identifiers live in a file with a revision history.
  3. Run the pipeline in batch. For each question, capture the retrieved identifiers and scores, the assembled context, and the final answer. Ragas describes building a reusable evaluation pipeline as a distinct step from running the app.
  4. Score retrieval first. Compute recall@k, precision@k, MRR, and nDCG@k at your operating k.
  5. Score answers second. Run the judge metrics over the same run. LangChain’s tutorial is a worked example of this pattern, with tracing enabled through environment variables.
  6. Store per-question results, not just averages. Aggregates hide the specific questions that fail, and those questions are where the debugging information lives.
  7. Change one variable at a time and re-run. Comparing two simultaneous changes produces a number you cannot act on.

The LangChain tutorial configures tracing with environment variables before running anything:

# Setup shown in the LangChain RAG evaluation tutorial
import os
os.environ["LANGSMITH_TRACING"] = "true"
os.environ["LANGSMITH_API_KEY"] = "YOUR LANGSMITH API KEY"
os.environ["OPENAI_API_KEY"] = "YOUR OPENAI API KEY"

Ragas’ worked example builds a minimal RAG system over a documentation dataset with a BM25 retriever and a generation client, then queries it directly. The retriever half of that setup is local keyword search; the generation half uses a hosted client in the example, so a local deployment would substitute its own model client:

# Structure from the Ragas "Evaluate and Improve a RAG App" example
import os
import asyncio
from openai import AsyncOpenAI
from ragas_examples.improve_rag.rag import RAG, BM25Retriever

os.environ["OPENAI_API_KEY"] = "<your_key>"
openai_client = AsyncOpenAI()

retriever = BM25Retriever()
rag = RAG(openai_client, retriever)

question = "What architecture is the `tokenizers-linux-x64-musl` binary designed for?"
result = asyncio.run(rag.query(question))
print(f"Answer: {result['answer']}")

Running the harness entirely offline

OpenAgent Eval’s RAG tutorial is documented as running entirely offline with mock providers, so the notebook can be worked through without an API key. It builds a minimal RAG pipeline consisting of chunking, a local keyword retriever, and a mock LLM, and configures evaluation both from YAML and programmatically. The same tutorial runs 18 retrieval, generation, performance, and cost metrics and includes sections on diagnosing failures, interpreting the report output, writing a custom metric, and comparing experiments.

Two caveats matter here. A mock LLM validates the harness — that your dataset loads, your metrics execute, and your report renders — but it tells you nothing about how your real model behaves. And an offline harness still needs a labelled dataset; mock providers do not generate ground truth for you.

Diagnosing failures: retrieval or generation?

Once both layers are measured, the failure mode is usually readable from the numbers. The following is a diagnostic heuristic, not a set of measured results:

SymptomFirst checkLikely cause
Answer misses facts that exist in the corpusRecall@k for that questionRetrieval: chunking, embedding, or query mismatch
Answer contradicts the retrieved contextFaithfulness score and the assembled contextGeneration: prompt, context ordering, or too much noise
Answer is correct but cites the wrong passagePrecision@k and nDCG@kRetrieval: too many chunks passed to the model
Retrieval metrics look fine, answer relevance is lowThe judge output and the prompt templateGeneration: the model is answering a nearby question
Scores shift between identical runsSampling settings and judge temperatureNon-deterministic configuration

OpenAgent Eval’s tutorial includes a dedicated section on diagnosing failures and interpreting report output, which is the natural place to start if you want a worked version of this reasoning.

Limitations and tradeoffs

Several constraints apply regardless of which framework you use. Judge-based metrics carry variance and cost, and on local hardware that cost is measured in GPU time rather than currency. Small datasets move in large steps, so a one-question change can swing a percentage point or more. Questions written by reading the same chunks you used to tune chunking will overstate performance. And a high faithfulness score is a proxy, not a guarantee: an answer can be perfectly grounded in the retrieved context and still be useless to the reader.

It is also worth being explicit about what the sources above are. They are documentation for three evaluation toolchains — LangChain’s LangSmith tutorial, Ragas, and OpenAgent Eval — not independent comparative benchmarks. Their metric lists tell you what the tooling can compute, not which metrics are validated for your workload. Treat them as implementation references and measure your own pipeline.

What to do next

Start with retrieval. Assemble a few dozen questions with reference chunk identifiers, compute recall@k at the k you actually pass to the generator, and improve retrieval until that number is acceptable before spending GPU time on judge metrics. Then add faithfulness and answer correctness with a local judge, store per-question results alongside the full configuration, and change one variable per experiment.

If you are running this on a specific stack, the most useful thing you can share with other practitioners is your environment: operating system, GPU and VRAM, model and quantization, retriever and k, and the metric you are tracking. Post that alongside your results in the community so others can compare against a configuration rather than a number in isolation.

FAQ

Can I evaluate a local RAG pipeline without an API key?

Yes, for part of it. Retrieval metrics such as recall@k, precision@k, MRR, and nDCG@k are deterministic and need no model calls at all. OpenAgent Eval's RAG tutorial is also documented as running entirely offline with mock providers, so the harness itself can be exercised without keys. Answer-level metrics that rely on a judge model still need some model to run, which can be a local one.

What is the difference between answer relevance and answer accuracy?

LangChain's RAG evaluation tutorial lists both as evaluators. Relevance asks whether the response addresses the question that was asked; accuracy asks whether the response matches the expected answer. An answer can be relevant but wrong, or accurate but phrased so that it does not directly answer the user's question.

Should I measure retrieval or answer quality first?

Retrieval first. It is deterministic, cheap, and diagnosable. If the correct chunk never reaches the model, answer metrics will be depressed for reasons that prompt changes cannot fix, so establishing recall@k at your operating k gives you a stable base before adding judge-based scoring.

Does a mock LLM tell me how my real model performs?

No. A mock provider validates that your dataset loads, your metrics execute, and your report renders. It does not predict the behaviour of your actual generation model, so treat offline mock runs as harness verification rather than as a quality measurement.

References

  1. https://docs.langchain.com/langsmith/evaluate-rag-tutorial
  2. https://docs.ragas.io/en/stable/howtos/applications/evaluate-and-improve-rag/
  3. https://openagenthq.github.io/openagent-eval/examples/rag/

Related terms

RAG
Retrieval-Augmented Generation: a technique that enhances a language model by supplying it with relevant external knowledge retrieved at query time.
Recall@k
The proportion of reference chunks for a question that appear within the top k retrieved results.
Faithfulness (groundedness)
An answer metric measuring whether the claims in a response are supported by the retrieved context rather than invented.
LLM-as-judge
Using a language model to score another model's output against criteria such as relevance, correctness, or grounding.
BM25
A classical keyword-based ranking function used as a local retriever in the Ragas example; it requires no embedding model or vector store.
Mock provider
A stand-in model used in offline evaluation harnesses so that pipeline and metric code can be tested without API keys or real inference.

Leave a Reply

Your email address will not be published. Required fields are marked *