Why a RAG System Retrieves the Wrong Documents

When a retrieval-augmented generation (RAG) system answers with the wrong document, the instinct is to blame the language model. That instinct is usually misplaced. If the correct document exists in the knowledge base, the failure almost always happens earlier in the pipeline — in how documents were split, how the query was turned into a search, or how candidate passages were ranked before the model ever saw them. As one debugging write-up from the AWS Builder Center puts it, the generation step was working perfectly: it was “faithfully summarizing garbage.” [1]

This guide is for practitioners who can already run a RAG pipeline — local or hosted — and now need to debug one that retrieves the wrong material. It covers how to classify the failure, how to trace a single bad answer through the pipeline, and which fixes address which cause.

Classify the failure before changing anything

Retrieval failures are not all the same, and the type of failure determines which investigation path is worth taking. A retrieval debugging playbook from unrag.dev proposes a taxonomy of five cases: [2]

• False negatives — relevant content exists in the index but is not retrieved. The user asks about topic X, the knowledge base covers X, and the system returns content about Y or nothing at all.
• False positives — irrelevant content is retrieved and used. The returned passage mentions the query topic only tangentially, and the model answers from it anyway.
• Permission failures — the right content exists but the user cannot access it. This looks like a false negative to the user, but the cause is access control rather than retrieval quality.
• Freshness failures — outdated content is retrieved instead of the current version. The material was correct at some point and is now stale.
• Coverage gaps — the content does not exist. This is a content problem, not a retrieval problem, and the correct response is an explicit “I don’t have information about that.”

Two of these are commonly mistaken for each other. A coverage gap cannot be fixed by tuning a retriever, and a permission failure cannot be fixed by swapping an embedding model. Classifying the failure first prevents a long detour.

Capture a diagnostic trace

Without instrumentation, RAG debugging becomes trial and error. The same playbook recommends capturing a diagnostic trace for any specific failure: what happened at each stage of the pipeline, starting with the query itself and whether it was rewritten. [2] A useful trace logs, in order:

1. The raw user query, verbatim.
2. The query actually sent to the retriever, after any rewriting, expansion, or decomposition.
3. The embedding model and version used for both indexing and querying.
4. The index, namespace, or collection queried, plus any metadata filters applied.
5. The top-k candidate chunks with their identifiers, similarity scores, and source documents.
6. Any reranker output and the final ordering.
7. The exact context string passed to the language model.
8. The generated answer.

The trace answers the single most useful question in RAG debugging: was the answer-bearing chunk present in the final context? If it was, the problem is downstream, in prompting or generation. If it was not, no model change will fix it.

Where the pipeline actually breaks

The AWS Builder Center article frames the system not as three boxes — documents, model, answer — but as roughly five places where things can go wrong, with three that break most often: chunking, retrieval, and ranking. [1]

Chunking cuts the answer in half

Chunking happens before anything is embedded. It is invisible, unglamorous, and responsible for a large share of bad answers. The article’s illustration is a refund policy table: item categories down the left, return windows across the top. Fixed-size chunking with a 300-token limit cuts the table down the middle. One chunk ends up holding “Electronics 14 days restocking fee applies” with no header explaining what those numbers refer to; another holds the header row with no data. Neither chunk answers “how long do I have to return a laptop?” Neither is wrong exactly — they are amputated. [1]

The tell-tale symptom of a chunking failure is a retrieved passage that shares vocabulary with the query but does not contain the answer. The retriever did its job; the answer was never assembled into a single retrievable unit. Typical causes are fixed-size splits that ignore document structure, tables and lists separated from their headers, and chunks that begin mid-sentence after an overlap boundary.

The right chunk exists but is never retrieved

This is the classic false negative. The content is in the index, but the query and the document do not meet in embedding space. Dense embeddings compress meaning, which is an advantage for paraphrase and a liability for rare tokens: product codes, error strings, acronyms, part numbers, and names that appear once in the corpus. A user who types the exact error message may get a semantically similar passage about a different error.

Query-side problems produce the same symptom. Multi-part questions, conversational follow-ups that rely on pronouns, and queries that were rewritten into something broader or narrower than the original can all push the correct chunk out of the candidate set. This is why the trace must record the query that was actually searched, not just the one the user typed. [2]

The right chunk is retrieved but ranked below the cutoff

“Retrieved” in a vector store means “inside the top-k that was returned.” Anything below that line is invisible to the model, no matter how relevant it is. If the correct chunk sits at rank 15 of 20 and the pipeline passes the top 5 to the model, the answer will be wrong while the index is perfectly healthy.

Small top-k values, near-duplicate documents that crowd the candidate list, and the absence of a reranking stage all contribute. A reranker scores a wider candidate set with a more expensive model and reorders it before the context is assembled; skipping that stage is a common reason a system retrieves plausible-but-wrong passages.

Causes that are not retrieval quality at all

Some wrong-document failures have nothing to do with similarity search. Permission filters that exclude the correct document, metadata filters with a mismatched value, and an index that was never rebuilt after a document was updated all produce the same user-visible symptom. [2] A less obvious case is changing the embedding model without re-indexing: vectors produced by different models are not comparable, so the query and the stored documents stop sharing a meaningful space.

A diagnostic procedure, from cheap checks to invasive ones

1. Reproduce with the exact query. Capture the raw query and the rewritten query. If you cannot reproduce the failure, you do not yet have enough diagnostic information.
2. Search the index for the answer directly. Use an exact phrase or a distinctive keyword from the document. If the content is not in the index at all, this is a coverage gap, an ingestion failure, or a permission filter — not a retrieval-quality problem.
3. Inspect the top-k candidates. Is the correct chunk present but ranked low? Treat it as a ranking problem. Is it absent entirely? Treat it as a query or embedding problem.
4. Run a keyword-only search on the same query. If lexical search finds the chunk and dense search does not, the two sides disagree on vocabulary. Hybrid search, which combines lexical and dense signals, is the usual remedy.
5. Read the retrieved chunk text. If it shares terms with the query but lacks the answer, look at how the document was split.
6. Check the final context string. Confirm whether the answer-bearing chunk reached the model. This separates retrieval failures from generation failures.
7. Only then consider the model. Swapping to a larger model or rewriting the prompt does not change which passages were retrieved.

Matching symptoms to fixes

These are hypotheses, not verdicts. Each symptom has more than one possible cause, and the trace is what confirms which one applies in your system.

Verify the fix against more than one query

A change that repairs the failing example can quietly degrade others. Re-run the original query and confirm two things: the answer-bearing chunk now appears in the final context, and the generated answer is correct. Then run a small regression set of queries whose correct source chunks you already know, and compare retrieval before and after. Similarity scores are not comparable across embedding models or index configurations, so track whether the right chunk appears in the final context rather than chasing a specific score.

Limits of what you can conclude

The answer alone rarely identifies the cause. A wrong answer is consistent with a chunking failure, a false negative, a false positive, a stale index, and a permission filter — the same output, different pipelines. Without a trace that records the query, the candidates, and the final context, any diagnosis is a guess. If you cannot reproduce the failure on demand, the first fix is instrumentation, not tuning.

It is also worth stating what the sources do not settle. The AWS Builder Center article describes chunking, retrieval, and ranking as the three stages that break most often, but it does not claim they are the only ones; it notes roughly five places for failure overall. [1] The unrag.dev taxonomy is a classification aid, not a measurement of how frequently each failure type occurs. [2]

What to do next

Instrument the trace before changing any component. Log the raw query, the rewritten query, the top-k candidates with identifiers and scores, and the final context string. That single change converts RAG debugging from prompt-and-pray into a procedure you can repeat. Then classify the failure, work through the checks in order, and verify the fix on a regression set rather than a single example.

If you are working through a retrieval failure right now, the community is a good place to compare notes — share your pipeline stage, the query that failed, and what your trace showed at each step. Concrete traces are far more useful to other practitioners than a description of the wrong answer.