How to Evaluate RAG Retrieval Quality with a Small Test Set

Evaluating RAG retrieval quality with a small test set means building a labeled list of real user questions paired with the chunk IDs that should answer them, running only the retriever against that list, and asserting on retrieval metrics such as recall@k and MRR. No generation model is involved, so the suite runs offline in seconds on every commit and fails the build when retrieval regresses — before a wrong chunk reaches a user as a confident, fluent answer.

This guide is for practitioners who already have a working RAG pipeline and want a regression test that catches its most common defect. The design follows the two-layer pattern described in OpsResolve’s tutorial on building a small RAG evaluation dataset for CI and Autonoma’s guide to testing whether RAG retrieves the right context.

Why retrieval regressions hide from end-to-end tests

A RAG pipeline changes in a dozen places that have nothing to do with each other: the chunker, the embedding model, the index, the retrieval filter, the reranker, the prompt, the generation model. Any one of them can make answers worse without anything failing. Unit tests do not see it, because no single function is wrong.

The failure mode is specific and easy to misdiagnose. The retriever pulls the wrong chunk, hands it to the generator, and the generator does exactly what it was asked: it summarizes the document it was given, faithfully and confidently, even though that document is irrelevant. Teams then tune the prompt for three iterations while the actual defect sits in the vector search. Testing retrieval as its own step, isolated from generation, is what makes the defect visible.

What “small” means: 30 to 80 cases shaped like real traffic

A useful retrieval set is small enough to read in one sitting — the OpsResolve tutorial puts the practical range at 30 to 80 cases — and shaped like the questions users actually ask. Pull them from logs, not from imagination. A set of 44 labeled queries is enough to gate a real pipeline, which is the size of the fixture in Autonoma’s runnable example.

Cover the question shapes the system has to handle, not just the easy ones:

• Lookup: the answer is one sentence in one document.
• Multi-chunk: the answer needs two passages, often from different documents.
• Filtered: the right answer depends on a metadata filter such as tenant, version, or date.
• Unanswerable: the corpus does not contain the answer, and the correct behavior is to say so.
• Near-miss: a document exists on a closely related topic and must not be chosen.

A set containing only lookup questions passes every commit and tells you nothing when the reranker breaks multi-chunk answers. The near-miss and unanswerable cases are the ones that catch over-eager retrieval, where a similarity score is high enough to return something but not high enough to be correct.

Label the evidence, not just the answer

For each question, record the chunks that actually contain the evidence — not a reference answer string. A query-to-expected-chunk-ID pair is what lets you score the retriever alone, without any LLM call in the loop. If you only store a golden answer, you cannot tell whether a failure came from retrieval or from generation.

Two practical requirements follow from this:

• Stable chunk labels. Chunk IDs must be deterministic and reproducible from the source document, not database row numbers that shift on every re-index. A human-readable composite such as api-keys#4 survives re-chunking better than an autoincrement ID.
• Multiple acceptable chunks. Store a list, not a single ID. Real corpora often contain two passages that both answer the question, and a test that insists on one of them will produce false failures.

Pin the evaluation index and freeze the corpus

The test set is only meaningful against a fixed index. Pin the evaluation index to a known snapshot of the corpus and embedding model, and version it alongside the labels. If you swap the embedding model or re-chunk the corpus, the pinned index must be rebuilt deliberately — and the retrieval scores will move, which is exactly the signal you want rather than a surprise in production.

This is also why the retrieval layer can run offline on a clean checkout with no API key: the index is a fixture, not a live service.

The metrics worth asserting

Retrieval metrics are cheap to compute and easy to explain to a reviewer. The table below summarizes the ones that carry their weight in a small suite.

Assert each metric separately rather than collapsing them into one score. A reranker can improve precision while quietly hurting recall, and a single blended number will hide that trade-off.

A minimal pytest layout

The following is an illustrative skeleton, not a tested artifact. It shows the shape of a suite that runs the retriever only and asserts on aggregate thresholds.

# tests/test_retrieval.py
import pytest
from rag.retriever import Retriever

# Pinned fixture index; no network calls, no API key.
retriever = Retriever(index="eval-index-2026-10-01")

CASES = [
    {"query": "What is the refund window for annual plans?",
     "expected_chunk_ids": ["billing-policy#3"]},
    {"query": "How do I rotate an API key?",
     "expected_chunk_ids": ["api-keys#1", "api-keys#4"]},
    # ... 30 to 80 real questions from logs
]

def recall_at_k(ranked_ids, expected_ids, k):
    top_k = set(ranked_ids[:k])
    return len(top_k & set(expected_ids)) / len(expected_ids)

def reciprocal_rank(ranked_ids, expected_ids):
    for rank, chunk_id in enumerate(ranked_ids, start=1):
        if chunk_id in expected_ids:
            return 1 / rank
    return 0.0

@pytest.mark.parametrize("case", CASES, ids=lambda c: c["query"][:40])
def test_case_retrieves_expected_chunk(case):
    ranked = retriever.search(case["query"], k=10)
    assert recall_at_k(ranked, case["expected_chunk_ids"], k=10) > 0

def test_aggregate_recall_at_5():
    scores = [
        recall_at_k(retriever.search(c["query"], k=10),
                    c["expected_chunk_ids"], k=5)
        for c in CASES
    ]
    assert sum(scores) / len(scores) >= 0.90

def test_near_miss_is_not_returned():
    ranked = retriever.search("How do I cancel a monthly plan?", k=5)
    assert "billing-policy#9" not in ranked

The 0.90 threshold above is a placeholder. Record your current scores first, then set the gate slightly below the baseline so it catches regressions without failing on normal variance.

Gate on aggregate thresholds, and protect the near-misses

Per-query assertions are useful for debugging, but the build should fail on aggregate thresholds rather than a single unlucky case. Autonoma’s example describes exactly this split: a labeled fixture, a regression suite that gates on aggregate thresholds, and a CI workflow that fails the build when retrieval regresses.

Keep a small number of hard per-query assertions for the cases where any failure is unacceptable — the near-miss that must not be retrieved, the filtered query that must respect tenant boundaries. Those are correctness properties, not statistics.

Keep the generation layer for nightly runs

Retrieval testing does not replace generation evaluation. It isolates one layer so that the slower, more expensive layer can run less often. The two-layer design recommended by OpsResolve is: a small, versioned, deterministic retrieval set on every commit, and a judge-based generation evaluation nightly or at release. A suite that calls a judge model a thousand times cannot run on every commit; a retrieval suite that never calls a model can.

Limitations and trade-offs to accept up front

• Small sets miss things. Thirty to eighty cases cannot cover every query shape. They catch most regressions because most regressions are retrieval regressions, not because they are exhaustive.
• Chunk labels are brittle. Change the chunker and every label needs review. Budget for that review as part of the chunking change, not as an afterthought.
• Retrieval metrics do not measure answer faithfulness. A perfect recall@k score says the evidence was available; it says nothing about whether the generator used it correctly.
• Tuning against the set overfits it. If you repeatedly adjust the retriever until the suite passes, hold back a portion of the cases or refresh them from newer logs.

A useful next step

Start by exporting 30 recent questions from your logs, labeling the chunk IDs that answer each one, and writing a single aggregate recall@k assertion against a pinned index. Run it, record the number, and only then decide what threshold to gate on. If you are running local models or a self-hosted vector store, this suite is also the cheapest way to compare embedding models and chunk sizes on your own corpus rather than on a public benchmark.

If you build a set like this, share your question mix, your baseline scores, and any near-miss cases that caught a real bug in the community discussion — concrete numbers from other people’s corpora are more useful than another tutorial.