Retrieval-Augmented Generation: A Practical Guide to RAG Systems

Retrieval-Augmented Generation (RAG) is a design pattern that lets a language model answer a question using evidence fetched from a document collection at request time, instead of relying only on what it absorbed during training. A RAG system joins two subsystems: a retriever that locates relevant passages in your data, and a generator, usually an instruction-following language model, that writes an answer grounded in those passages.

This guide is for technically curious readers who can read a little code and want to move from having heard of RAG to running a pipeline they can evaluate. It covers the mental model, the design decisions that change answer quality, a realistic first build, how to measure whether the system works, and where the approach breaks down. It does not cover model training, and it does not promise that RAG eliminates hallucination: retrieval removes some failure modes and introduces others.

The Problem RAG Addresses

A language model’s parameters hold a compressed, frozen snapshot of its training data. Four consequences follow. The model cannot know about events or documents created after training. It cannot see your private files, internal wikis, or customer records. It can produce fluent statements that are simply wrong, because fluency and factuality are separate properties. And updating its knowledge normally means retraining or fine-tuning, which is expensive and still static once finished.

RAG addresses these by changing the input rather than the weights. At query time the system searches a corpus you control, injects the best-matching passages into the prompt, and asks the model to answer from that evidence. The book A Practical Approach to Retrieval Augmented Generation Systems describes this as blending the strengths of information retrieval and natural language generation, and uses the familiar chat-with-your-PDF scenario as its running example [1]. The engineering guide Retrieval-Augmented Generation: An Engineer’s Guide to Building RAG Systems with Your Own Data frames the same idea around a harder problem: most teams can get a notebook demo working, but the gap between this works on my laptop and this runs reliably in production is wide [2].

How RAG differs from neighbouring techniques

RAG is often confused with four things that are not the same.

• Fine-tuning modifies model weights so the model behaves differently. RAG leaves the weights alone and supplies evidence. Fine-tuning is good for style, format, and task behaviour; RAG is good for facts that change and data you cannot ship into a training run. The two are complementary.
• Semantic search is retrieval without generation. A vector search over documents is a component of RAG, not a RAG system.
• Prompt engineering shapes how the model responds. It cannot supply knowledge the model never had, and it cannot cite a source that was never retrieved.
• Long-context prompting pastes a large document into the prompt. It works for a single short file, but it scales badly in cost and latency, and models do not attend equally to every position in a very long context. Retrieval is how you decide what deserves the space.

RAG is also not always the right answer. If your task is to change how the model writes rather than what it knows, fine-tuning or careful prompting is cheaper. If your data fits comfortably in a single prompt and changes rarely, long-context prompting may be simpler. RAG earns its complexity when the corpus is larger than a prompt, changes over time, or must be cited.

The Mental Model: Four Stages and a Feedback Loop

Every RAG system, from a weekend prototype to a production service, decomposes into four stages. Three of them run at query time or near it; the fourth runs continuously.

1. Ingestion and indexing. Documents are loaded, parsed, split into chunks, converted into embeddings, and stored in an index alongside metadata.
2. Retrieval. A user question is turned into a query, the index is searched, and a ranked set of candidate passages is returned.
3. Generation. The top passages are assembled into a prompt, and the language model writes an answer, ideally with citations back to the source chunks.
4. Evaluation and observability. Queries, retrieved chunk identifiers, scores, latency, and answers are logged so you can tell whether a change helped.

It helps to trace one question through all four stages. The example below is illustrative rather than a case study.

Suppose you have indexed a 300-page employee handbook. A user asks: how many days of parental leave do I get, and does it differ for adoption?

During ingestion, the handbook was split by heading, so the parental-leave section became a handful of chunks with metadata recording the section title and page number. At query time the question is embedded and matched against those chunks. A dense retriever may surface the parental-leave section but miss a separate adoption clause filed under a different heading; a keyword retriever may catch the word adoption but miss a paraphrase. This is exactly why hybrid retrieval and reranking exist, and why chunk boundaries matter.

The generator then receives perhaps four chunks and an instruction to answer only from them and to name the section for each claim. If the adoption clause was never retrieved, a well-behaved system should say the retrieved policy does not cover adoption, rather than inventing a number. That behaviour is a property of the prompt and the model, not of the retriever, which is why the two must be evaluated separately.

Ingestion and indexing in more detail

Ingestion is the stage teams under-invest in. The engineer’s guide notes that a production pipeline has to handle real PDFs with OCR artifacts, encoding issues, and dirty markdown [2]. A document that arrives as a scanned PDF may contain no usable text layer at all; a markdown export may have broken tables; a CSV may have shifted columns. If parsing is wrong, no amount of downstream cleverness recovers the information.

After parsing, text is split into chunks. Each chunk is embedded, meaning it is converted into a numeric vector such that semantically similar passages sit close together in vector space. Vectors and their metadata are written to an index, which is usually a vector database or a vector index library.

Retrieval in more detail

Retrieval turns a question into one or more queries, searches the index, and returns candidates. The simplest version embeds the question and performs a nearest-neighbour search. More capable versions add keyword search, metadata filters, query rewriting, and a second-stage reranker. The output of retrieval is not an answer; it is a shortlist of evidence.

Generation in more detail

Generation formats the shortlist into a context block and asks the model to answer from it. Good prompts do three things: they constrain the model to the supplied evidence, they require citations or explicit uncertainty, and they define the output shape. The model’s job is synthesis and phrasing, not recall.

The evaluation loop

The fourth stage is what separates a demo from a system. Log every query, the chunk identifiers retrieved, their scores, the final answer, and the latency. Without this record you cannot tell whether a prompt change improved faithfulness or merely made the prose more confident.

Design Decisions That Determine Answer Quality

Most RAG failures trace back to a small number of decisions. The engineer’s guide is explicit that chunking strategy should be chosen on retrieval evidence rather than intuition, and that vector store choice should follow your actual constraints [2]. The sections below treat each decision the same way.

Chunking: the decision that most often decides retrieval quality

A chunk is the unit of retrieval. If chunks are too small, they lose the context needed to be meaningful; if they are too large, they dilute the embedding and crowd the prompt with irrelevant text. Common strategies include fixed-size windows with overlap, recursive splitting on paragraph and sentence boundaries, structure-aware splitting on headings and sections, and semantic splitting that groups sentences by topic shift.

Practical guidance: start with structure-aware splitting, because headings are free information about where a topic begins. Add overlap of roughly ten to fifteen percent of the chunk size so that a sentence spanning a boundary is not lost. Store metadata with every chunk, at minimum the source file, section heading, and position, because that metadata is what makes citation and filtering possible.

Embedding models

An embedding model maps text to a vector. Choices differ in dimension, maximum input length, language coverage, domain fit, and whether they run locally or behind an API. Two rules of thumb are worth stating plainly. First, the embedding model and the retrieval task must match: a model trained on general web text may underperform on legal or medical phrasing. Second, changing the embedding model invalidates the existing index, because vectors from different models are not comparable. Re-embedding the whole corpus is the cost of that change.

Vector stores: FAISS, pgvector, and Qdrant

The engineer’s guide names FAISS, pgvector, and Qdrant as the three to pick between based on actual constraints [2]. They occupy different positions rather than competing on a single axis.

There is no universal winner. A single-server internal tool with a few thousand chunks is well served by an in-process index. A multi-tenant product with per-user permissions is usually better served by a database that can enforce filters and survive restarts.

Retrieval mode: dense, keyword, or hybrid

Dense retrieval embeds the query and compares vectors. It handles paraphrase well and exact strings poorly. Keyword retrieval, classically BM25, matches terms and handles rare identifiers, error codes, and product names well. Hybrid retrieval runs both and fuses the rankings. The engineer’s guide lists hybrid search among the techniques to add when they earn their place [2], and that framing is useful: hybrid search adds moving parts, so introduce it when you can show a retrieval-quality gain on your own test set.

Reranking and query transformation

A reranker takes the shortlist from first-stage retrieval and reorders it using a more expensive model that reads the query and each candidate together. Because it only scores a few dozen candidates, it is affordable and often produces a larger quality gain than swapping the embedding model. Query transformation rewrites or expands the user question before retrieval, which helps when questions are vague or conversational. Both are optional stages; both should be justified by measurement.

Top-k and the context budget

How many chunks you retrieve is a budget decision, not a correctness decision. Retrieving more raises the chance that the right passage is present but also raises prompt length, cost, latency, and the chance that marginal passages distract the model. A common pattern is to retrieve a wider candidate set, rerank it, and then pass only the top few chunks to the generator. Whatever you choose, record it: top-k is one of the first parameters to revisit when answers degrade.

A decision summary

A Realistic First Build

The goal of a first build is not a product. It is a pipeline small enough to understand and instrumented enough to improve. The steps below assume you can install Python packages and that you have a document set you are permitted to process.

Prerequisites. A folder of 20 to 50 representative documents in formats you can parse; a working Python environment; an embedding model, either a local one or a hosted API; and a generator model, again local or hosted. If your documents contain personal or confidential data, decide before you start whether embeddings and prompts may leave your machine.

1. Assemble a representative corpus. Choose documents that cover the questions you actually expect. A corpus that is easy to parse but unrepresentative will produce misleadingly good results.
2. Parse and clean. Extract text, repair encoding, and strip repeated headers and footers. Inspect a sample by eye. If a PDF has no text layer, it needs OCR first; if tables matter, check whether they survived parsing.
3. Chunk with structure. Split on headings and paragraphs, then apply a size window with overlap. Attach source, section, and position metadata to every chunk. Success check: print ten random chunks and confirm each one is understandable on its own.
4. Embed and index. Convert chunks to vectors and write them to your chosen store. Success check: run a query you know the answer to and confirm the correct chunk appears in the top results.
5. Write a labelled question set. Twenty to thirty questions, each with the passage that should answer it and a short reference answer. This set is the instrument you will use for every later decision.
6. Build the retrieval function. Return the top k candidates and log their identifiers and scores. Success check: measure how often the correct passage appears in the top k.
7. Assemble the prompt. Instruct the model to answer only from the supplied context, to cite the source of each claim, and to say when the context is insufficient. Success check: ask a question your corpus cannot answer and confirm the system declines rather than guesses.
8. Measure and iterate. Change one thing at a time, re-run the question set, and keep the change only if it improves a metric you care about.

The following is illustrative pseudocode, not a tested script. It shows the shape of the pipeline rather than the API of any specific library.

# Illustrative pseudocode only.
# --- Ingestion ---
chunks = []
for doc in load_documents('corpus/'):
    text = normalize(doc.text)                 # encoding, headers, footers
    for block in split_by_structure(text):     # headings and paragraphs
        for piece in window(block, size=800, overlap=120):
            chunks.append({
                'text': piece,
                'source': doc.path,
                'section': block.heading,
            })

vectors = embed(chunks)                        # local model or hosted API
index = VectorIndex(dim=vectors.dim)
index.add(vectors, metadata=chunks)

# --- Query time ---
def answer(question, k=8):
    query_vector = embed([question])[0]
    hits = index.search(query_vector, k=k)
    hits = rerank(question, hits)              # optional second stage
    context = format_with_citations(hits[:4])
    return generate(prompt=PROMPT, context=context, question=question)

Notice what the code does not do. It does not decide chunk size for you, it does not guarantee that the right passage was indexed, and it does not check whether the generated answer is faithful. Those are the parts that require judgement and measurement.

Evaluating a RAG System

The engineer’s guide makes a point that is worth repeating because it is so often skipped: evaluate retrieval quality separately from generation quality, and prove that your changes help [2]. Mixing the two produces a single number that tells you something is wrong but not what.

Retrieval metrics

With a labelled question set you can compute recall at k, which is the fraction of questions whose correct passage appears in the top k results, and precision at k, which is the fraction of returned passages that are relevant. Mean reciprocal rank captures how high the first correct result appears. These are cheap to compute and they change immediately when you adjust chunking or retrieval mode.

Generation metrics

Generation quality is about faithfulness to the retrieved context, relevance to the question, and correctness of citations. Faithfulness is best assessed by a human on a sample, or by a model-based grader whose judgements you spot-check. Citation correctness can be checked mechanically: does the cited chunk actually contain the claim?

Building the labelled set

The labelled set is the foundation of everything above. Write questions the way real users ask them, including vague and conversational phrasings, and include several questions your corpus genuinely cannot answer. For each question, record the chunk or passage that should be retrieved. If two people label the same question differently, the question is ambiguous; rewrite it or accept that the system cannot be scored reliably on it.

Failure modes to look for

RAG systems fail in recognisable ways. The engineer’s guide introduces seven failure points in its first chapter and traces them through every subsequent chapter [2]; the categories below are general patterns you can test for directly.

• Nothing relevant was indexed. The answer exists in the source material but the chunk was never created, or parsing destroyed it. Diagnostic: search the index for a distinctive phrase from the source.
• The right chunk was not retrieved. It exists but ranks below the cut. Diagnostic: increase k and see whether it appears.
• The right chunk was retrieved but ranked low. Diagnostic: add a reranker and re-measure.
• Context dilution. Too many marginal chunks crowd out the useful one. Diagnostic: reduce k and compare.
• The model ignored the context. It answered from parametric memory instead. Diagnostic: ask a question whose correct answer contradicts common knowledge and see which one appears.
• The index is stale. The document changed but the chunk did not. Diagnostic: compare the chunk text against the current source.
• Permission leakage. A user retrieved a chunk they should not see. Diagnostic: attempt a query as a restricted user and inspect the retrieved identifiers.

Troubleshooting Common Problems

The table below pairs a symptom with a plausible cause, a diagnostic step, and an action. Causes are hypotheses until your logs confirm them.

Privacy, Cost, and Operational Reality

Three practical constraints shape RAG deployments more than model choice does.

Data handling. Embeddings are derived from your documents, and prompts contain retrieved passages. If either leaves your infrastructure, you have exported content. Decide deliberately whether embeddings and generation run locally or through a hosted API, and enforce access control at retrieval time, not only in the interface. A retrieval layer that ignores user permissions will happily surface a confidential chunk to the wrong person.

Cost. RAG cost has three components: embedding the corpus once, embedding each query, and generating an answer from a prompt that may be several thousand tokens long. Because context length dominates generation cost, retrieving eight chunks instead of four can roughly double the prompt. Any cost figure depends on your chunk size, your k, your prompt template, and current provider pricing, so treat estimates as assumptions to be measured rather than constants.

Freshness and observability. An index is a cache of your documents, and caches go stale. Plan for incremental updates keyed on modification time or content hash, and plan for deletion, including the case where a document must be removed for legal reasons. Log queries, retrieved identifiers, scores, and latency so that a complaint can be traced to a specific retrieval decision.

Running RAG Locally

Local RAG keeps documents, embeddings, and generation on hardware you control. The architecture is identical; only the models change. Embedding models are typically small enough to run on a CPU, while generation is the demanding part.

When estimating memory, do not count model weights alone. A running system holds the weights, the key-value cache that grows with context length and batch size, and runtime overhead from the framework. A long retrieved context increases the KV cache, so a configuration that fits comfortably at short prompts may fail at the prompt lengths RAG actually produces. Measure with your real chunk size and k rather than a short test prompt.

Practical sequence for a local setup: start with a small corpus and a modest quantised model, confirm the pipeline works end to end, then increase model size only if answer quality is the limiting factor. If retrieval is the bottleneck, a larger generator will not help. The engineer’s guide’s emphasis on choosing components from constraints rather than defaults applies directly here [2].

Local deployment also changes your evaluation obligations. Hosted models are opaque and change under you; local models are fixed, so a regression after an upgrade is your own change and can be bisected. Record which model file, quantisation, and runtime you used alongside your retrieval scores, or you will not be able to reproduce a good result later.

RAG on a WordPress Site

Content sites are a natural RAG fit because they already hold a large, structured, permissioned corpus. Two use cases dominate: a documentation assistant that answers reader questions from published articles, and an editorial assistant that surfaces related material while drafting.

A workable architecture has four parts. First, ingest published content through the WordPress REST API on a schedule, storing the post identifier and modification time so that unchanged posts are skipped and duplicates are not created. Second, chunk and index that content with metadata recording the post identifier, URL, and section. Third, expose retrieval and generation through a small service that the site calls, keeping credentials server-side. Fourth, render answers in the front end with citations back to the source posts, and log every query.

Two editorial controls matter. Generated answers should be presented as derived from published content, with links to the originals, so readers can verify. And any workflow that drafts content for publication should route through human review rather than publishing directly; the retrieval step improves grounding but does not remove the need for an editor. If the site serves multiple audiences, enforce visibility rules at retrieval time so that drafts and private pages never enter the context.

Learning Path and Next Action

A useful progression runs in three stages.

Beginner. Build the pipeline described above on 20 to 50 documents, with 20 to 30 labelled questions. The deliverable is a working query loop and a recall number you trust. Do not add hybrid search, reranking, or query rewriting yet; you have no baseline to compare against.

Intermediate. Improve retrieval on evidence. Try a different chunk size, add hybrid search, then add a reranker, measuring recall at k and answer faithfulness after each change. The deliverable is a short record of what helped and what did not on your corpus.

Advanced. Add the operational layer: query routing when you have several corpora, conversational memory when follow-up questions matter, and observability tooling that lets you replay a bad answer. The book A Practical Approach to Retrieval Augmented Generation Systems treats query routing, memory, and observability as the natural next concerns after the core pipeline works [1].

If you are deciding where to start today, pick the smallest corpus that contains a real question you actually need answered, and build the labelled question set before you tune anything. Retrieval quality, not model size, is usually what limits a first RAG system.

If you would like help interpreting your results, share your goal, your environment, the documents you indexed, the steps you attempted, and the retrieval scores you observed in the community discussion. Concrete numbers make troubleshooting far faster than descriptions of answers that felt wrong.