If you are about to reduce RAG token costs — lowering top-k, shrinking chunks, swapping in a cheaper embedding model, or removing a reranker — measure retrieval quality first. The Databricks AI Search retrieval quality guide is direct about the ordering: before optimizing retrieval you “must have a reproducible evaluation system,” and if you do not have evaluation in place, “stop here and set it up first,” because “optimizing without measurement is guesswork.” The same rule applies to cost work. Almost every token-saving change is also a retrieval change, and retrieval changes tend to fail quietly.
This guide is for practitioners who already have a working RAG pipeline and are under pressure to reduce prompt size or per-query spend. It covers what to measure, how to build a small evaluation set, which metrics map to which cost decisions, and how to keep quality and cost in the same harness so a saving can be distinguished from a regression.
Why cost cuts are retrieval changes in disguise
In most RAG pipelines, the prompt dominates the bill. Retrieved context tokens are roughly top-k × average chunk size, on top of the system prompt, conversation history, and any query-expansion or reranking overhead. Reducing any of those factors reduces cost — and simultaneously removes candidate evidence from the model’s view.
The failure mode is not an error message. The Google Cloud RAG evaluation guidance describes RAG systems that are not thoroughly evaluated as producing “silent failures” that undermine the reliability and trustworthiness of the whole system. A model given four chunks instead of ten will usually still produce a fluent, confident answer. It will simply be wrong more often on questions whose evidence was in chunk five.
That is why the cost conversation has to start with a measurement question: what retrieval quality do we currently have, and how much of it can we spend?
Define the contract: latency targets and the metric that matters
Before touching configuration, write down two things.
Latency targets. The Databricks guide recommends establishing clear latency targets by use case, giving two examples: RAG agents measured by Time to First Token (for example, under 2 seconds) and search bars measured by end-to-end latency to display results (for example, under 100 milliseconds). Those figures are illustrations from the guide, not universal thresholds — but the principle holds: any optimization must meet the latency requirement, and some cost optimizations (adding a reranker, adding a query-rewriting LLM call) trade tokens for time.
The quality metric that matches your risk. The Databricks guide frames metric choice by use case and notes that recall matters most when missing information is costly, naming RAG agents, pharmaceutical clinical trial matching, and financial compliance search as examples. If a missed chunk means a wrong answer, an ineligible patient, or a missed regulation, recall is your gate. If your queries are single-fact lookups and the model is drowning in irrelevant context, precision and context-token count may matter more.
Build the smallest evaluation set that can detect a regression
The Databricks guide lists three practical ways to get automated evaluation running: an existing golden dataset of labeled query-answer pairs; a synthetic evaluation set generated from your own documents; or ground-truth-free evaluation using Agent Evaluation judges. Its key point is that perfect data is not required — what matters is having some automated way to measure change, and focusing on relative improvements between runs.
The Google Cloud guidance adds structure to the question set itself: the test dataset should cover a broad subset of the underlying data and include variations in phrasing and question complexity that match real-world use cases, and it is worth consulting stakeholders and end users to check relevance. It also recommends assembling a golden reference dataset of desired outputs, noting that while some metrics can be computed without references, known-good outputs allow a more comprehensive and nuanced evaluation.
In practice, the cheapest source of realistic questions is usually already in your logs: site search queries, support tickets, and chat transcripts. Add a handful of questions your system should refuse to answer, so that cost cuts cannot hide behind confident improvisation.
Metrics that map to specific token decisions
The metrics below are standard information-retrieval measures; tooling names and implementations vary by stack, so confirm what your framework actually computes before trusting a number.
| Metric | What it measures | Token decision it protects |
|---|---|---|
| Recall@k | Share of relevant chunks present in the top-k results | Lowering top-k, shrinking chunk size, narrowing the candidate pool |
| Precision@k / context precision | Share of retrieved chunks that are actually relevant | Trimming context, removing reranking, tightening filters |
| MRR | Rank position of the first relevant chunk | Switching to a cheaper embedding model or dropping reranking |
| nDCG@k | Graded ranking quality across the result list | Reordering or truncation strategies |
| Hit rate | Fraction of queries with at least one relevant chunk | Aggressive metadata filters and hard result caps |
| Context tokens per query | Actual retrieved-context size sent to the model (report p50 and p95) | The cost figure you are trying to change |
| Answer faithfulness / groundedness | Whether the generated answer is supported by the retrieved context | Catching cases where a cost cut converts a retrieval miss into a hallucination |
A reproducible baseline procedure
- Freeze the configuration and version it. Record the embedding model and version, chunk size and overlap, index type, top-k, reranker settings, prompt template, generation model, and sampling settings. A run without a config record is not a baseline.
- Freeze the question set. Start smaller than feels comfortable. The Databricks guide notes that even a small synthetic dataset can reveal whether reranking improves quality by a meaningful margin for your use case.
- Label relevance at the chunk level, not only the answer level. If you only score final answers, you cannot tell whether a quality drop came from retrieval or from generation.
- Run the baseline and capture both quality and cost. Store retrieved chunk IDs, per-query context token counts, latency, and the metric scores together.
- Change one variable at a time. Top-k, chunk size, reranker, and embedding model interact; changing three at once produces a result you cannot attribute.
- Re-run and compare. Treat differences smaller than your set’s natural variation as inconclusive rather than as wins.
A minimal run record might look like this:
run_id: baseline-2026-10
embedding_model: <name-and-version>
chunk_size: 512
chunk_overlap: 64
top_k: 10
reranker: none
prompt_template: rag_v3
generation_model: <name-and-version>
metrics: [recall@10, precision@10, mrr, ndcg@10, context_tokens_p50, context_tokens_p95, ttft_p95]
An illustrative worked example
The following is a constructed scenario, not a measured result. Suppose a documentation assistant runs at top-k = 10 with 800-token chunks. Retrieved context is roughly 8,000 tokens per query before the system prompt. Dropping to top-k = 4 with 400-token chunks reduces that to about 1,600 tokens — roughly an 80% reduction in retrieved context.
Whether that change is acceptable depends entirely on what the frozen evaluation set says. If the question set is dominated by single-fact lookups whose answer sits in one chunk, recall@4 may hold. If it contains multi-hop questions whose evidence spans two documents, recall@4 will likely fall, and the “saving” is really a quality trade you did not price. The arithmetic above is illustrative and depends on your own chunking, reranking, and query distribution.
Put cost accounting in the same harness
Log per-query context tokens, total prompt tokens, completion tokens, embedding calls, reranker calls, and wall-clock latency — and report the tail, not just the mean. A p95 context length is what causes surprise bills and timeout errors.
Then report quality and cost as a pair. “Recall@10 down 3 points, context tokens down 60%” is a decision a team can make. “It’s cheaper now” is not. Keep token counts as the stable unit and apply provider pricing separately, since pricing and caching behaviour change independently of your retrieval configuration.
Where local models and WordPress automation change the framing
If you run models locally, the bill is compute and time rather than per-token dollars — but longer context still costs KV cache memory and time-to-first-token, so the same evaluation gate applies. Local models are often smaller and more sensitive to noisy context, which shifts the emphasis toward precision@k and context-token reduction rather than raw recall alone.
For WordPress-based assistants — documentation search, support chat, internal knowledge bases — the evaluation harness can run against real site-search logs, and the retrieval configuration can be tested before deployment rather than after. If your automation pipeline rebuilds an index or changes chunking on a schedule, run the frozen question set as a check in that pipeline. Keep a human review step for any content or configuration change that affects what the assistant can cite.
Failure modes worth naming
- Scoring only the final answer. A strong generation model masks retrieval misses, so the metric looks stable while the retriever degrades.
- Tuning and testing on the same questions. The set stops measuring generalisation and starts measuring memorisation.
- Ignoring latency. A reranker can cut context tokens while blowing a time-to-first-token budget.
- Comparing runs with different configurations. Without a frozen config record, differences are unattributable.
- No unanswerable questions. Without them, a system that hallucinates under thin context scores the same as one that abstains.
- Optimising the mean only. Cost and quality problems usually live in the tail.
The gate to adopt before your next cost change
Agree on a rule and write it down: no cost change ships unless your primary quality metric stays within an agreed tolerance on the frozen evaluation set, and latency stays inside its target. Then re-measure cost, and record the new configuration as the next baseline. That single practice turns token optimisation from guesswork into a sequence of reviewable decisions — and it is the ordering both the Databricks and Google Cloud guidance point toward.
If you are working through this now, it helps the community to compare notes. Share your setup — retrieval stack, top-k, chunk size, the metrics you track, and what changed when you cut context — so others can see which trade-offs held up in practice.
FAQ
Can I evaluate retrieval quality without a labeled dataset?
Yes. The Databricks guide lists ground-truth-free evaluation using Agent Evaluation judges as one of three approaches, alongside an existing golden dataset and a synthetic evaluation set generated from your documents. Google Cloud's guidance similarly notes that some metrics can be calculated without a reference dataset, while a set of known-good outputs allows a more comprehensive and nuanced evaluation.
How many test questions do I need before I can trust a comparison?
Neither source specifies a number. The Databricks guide's position is that the key is having some automated way to measure changes and that perfect data is not required — even a small synthetic dataset can indicate whether a change helps your specific use case. Start small, then expand the set as you find questions it fails to distinguish.
Does reducing top-k always reduce answer quality?
Not necessarily. The effect depends on where the evidence sits in your corpus and how your questions are phrased. The only reliable way to know is to measure recall@k and your primary quality metric on a frozen question set before and after the change, rather than assuming either direction.
Is token cost the only cost that matters when tuning retrieval?
No. Reranking, query expansion, and additional LLM calls consume compute and add latency. The Databricks guide recommends establishing latency targets up front and requiring any optimization to meet them, which means a change that lowers token count but breaks a time-to-first-token budget is not a net win.