How to Evaluate and Reduce Hallucinations in LLM Workflows

Hallucinations in LLM workflows are best treated as two linked engineering problems: measuring them with a labelled evaluation set, and reducing them with grounded retrieval plus an output guardrail. Teams that do only one of the two tend to ship either a system that sounds confident but is wrong, or a system that is accurate but so heavily filtered that users stop trusting it.

This guide is for practitioners who already have a working LLM feature — a support agent, a documentation assistant, an automation step — and need a repeatable way to find out how often it invents things, and what to do about it. It draws on two public references: the OpenAI Cookbook walkthrough on developing hallucination guardrails and Microsoft’s partner best-practices document on mitigating hallucinations with Azure AI services.

Why evaluation comes before tuning

The OpenAI Cookbook frames guardrail development as three sequential activities: build a strong evaluation set, identify specific criteria to measure hallucinations, and only then improve the accuracy of the guardrail with techniques such as few-shot prompting. The ordering matters. If you change a prompt, swap a retrieval strategy, or add a verification step without a fixed set of labelled examples, you have no way to tell whether the change helped or merely moved the failures somewhere you are not looking.

An evaluation set is therefore not documentation of past performance. It is the instrument you use to make every subsequent decision.

Defining what counts as a hallucination in your workflow

“Hallucination” is too broad to score directly. Before labelling anything, narrow it to a criterion that a reviewer can apply consistently. In the Cookbook’s illustrative scenario, a customer support team feeds an automated agent knowledge-base policies for handling tickets such as returns, refunds, and feedback, and expects the model to follow those policies when responding to customers. In that setting, a hallucination is a response that contradicts, invents, or silently extends a policy.

That is meaningfully different from other failure modes:

• Unhelpful — the answer is correct but does not resolve the request.
• Off-topic — the model drifts away from the question.
• Unsafe — the content violates a policy unrelated to factual accuracy.

Microsoft’s guidance treats these as separate layers. Its best-practices document recommends a layered evaluation and security approach, with content safety handled alongside accuracy rather than folded into a single score. Conflating them produces a metric that no one can act on.

Building the evaluation set

A useful eval set has three properties: it covers the topics your system actually answers, each item has a reference answer or a policy excerpt that defines correctness, and each item carries a label stating whether the model output was acceptable.

In the Cookbook’s approach, the team first uses GPT-4o to draft the set of policies the assistant must follow, then builds the evaluation data around them. Synthetic generation is a starting point, not the finish line: generated examples should be reviewed against real user questions, because synthetic queries rarely match the phrasing, ambiguity, and edge cases that arrive in production.

For a support workflow, a workable first pass might include:

1. Common questions with a single unambiguous policy answer.
2. Questions that touch two policies and require the model to combine them.
3. Questions whose correct answer is “this is not covered — escalate.”
4. Questions that resemble a covered topic but are actually out of scope.

Category 4 is where most hallucination guardrails earn their keep, because it is where a model is most likely to pattern-match onto a nearby policy and answer confidently.

Scoring with precision and recall

The Cookbook’s implementation imports precision_score and recall_score from sklearn.metrics and reports both against the labelled set. That pairing is the right default for a hallucination check, because each metric describes a different failure:

• Recall answers: of the outputs that really were hallucinated, how many did the guardrail catch?
• Precision answers: of the outputs the guardrail flagged, how many were actually hallucinated?

A guardrail that flags everything achieves perfect recall and terrible precision. In production that means every correct answer is routed to a human, which is usually worse than the problem you set out to solve. A guardrail that flags almost nothing achieves the reverse. Reporting a single accuracy number hides this tradeoff entirely.

Track both figures over time, and record which examples moved when you change something. A prompt revision that raises recall by two points while dropping precision by fifteen is a regression, not a win.

Reducing hallucinations through retrieval grounding

Evaluation tells you how often the system fails. Grounding is the main lever for reducing that rate. Microsoft’s best-practices document organises retrieval-augmented generation (RAG) work into three areas, and the ordering is again deliberate.

Data preparation and organisation

• Clean and curate the source data before it reaches the index.
• Organise content into topics so search results are not diluted by unrelated material.
• Audit and update grounding data regularly, because outdated content produces confidently wrong answers just as reliably as missing content.

Search and retrieval techniques

• Compare keyword, vector, hybrid, and semantic search rather than assuming one is correct for your corpus.
• Use metadata filtering — recency tags, source reliability — to prioritise higher-quality passages.
• Apply chunking so retrieved units are coherent and small enough to be useful in context.

Query engineering and post-processing

• Use prompt engineering to specify which data source or section the model should draw from.
• Apply query transformation, such as sub-queries, for complex multi-part questions.
• Employ reranking to improve the quality of what finally reaches the model.

The practical implication is that most hallucination reduction happens before generation, not after it. If the correct passage never reaches the context window, no amount of instruction-following will produce a grounded answer.

Prompt-level controls that support grounding

Retrieval supplies evidence; the prompt tells the model what to do with it. Standard controls include instructing the model to answer only from the supplied context, to cite the passage it used, and to state plainly when the context does not contain the answer. The last instruction is the one teams most often omit, and it is the one that converts an invisible hallucination into a visible escalation.

These controls are necessary but not sufficient. A model can still misread a retrieved passage or blend two policies, which is why an output check remains useful.

Guardrails as a runtime layer

The Cookbook defines a guardrail as a set of rules and checks designed to ensure that LLM outputs are accurate, appropriate, and aligned with user expectations, and develops an output guardrail specifically for hallucination. In practice this is a second model call that receives the question, the retrieved context, and the candidate answer, and returns a judgement against your written criteria.

Few-shot prompting improves the accuracy of that judgement, according to the Cookbook’s walkthrough. Including several labelled examples — both clear hallucinations and borderline cases that should pass — reduces the rate at which the guardrail misfires. The guardrail itself should be scored with the same precision and recall discipline as the primary system.

Design the failure path explicitly. A flagged answer should route to a human reviewer, a safe fallback message, or a retrieval retry — not simply be deleted. Silent suppression hides a defect that will otherwise repeat.

A repeatable improvement loop

Combining both sources into a single working cycle:

1. Assemble a labelled evaluation set covering each policy or topic your system handles.
2. Measure a baseline: precision, recall, and the raw counts behind them.
3. Inspect the false negatives — hallucinations the guardrail missed — and classify the cause.
4. Fix the cause at the right layer: retrieval quality, prompt instruction, or guardrail criteria.
5. Re-run the full set and compare. Change one variable at a time.
6. Add newly discovered failure cases back into the set so the instrument improves with the system.

Step 3 is where most of the value sits. A false negative caused by a missing document is a data problem. One caused by a vague instruction is a prompt problem. One caused by a guardrail that cannot distinguish a paraphrase from a contradiction is a criteria problem. Treating all three as “the model hallucinated” leads to prompt tinkering that fixes nothing.

Where local and self-hosted setups change the picture

If you run models locally or self-hosted, the same loop applies, but two constraints shift. First, the guardrail is itself a model call, so it consumes VRAM and adds latency on top of generation; budget for it rather than assuming it is free. Second, a smaller local judge model may be less reliable at fine-grained criteria, which makes a well-labelled eval set more important, not less — it is the only way to know whether your local judge is trustworthy. In automation pipelines, the same reasoning applies to any workflow step that publishes or acts on model output: place the verification before the action, not after.

Limitations and tradeoffs

• No metric eliminates hallucination. Precision and recall describe behaviour on your eval set, not a guarantee about unseen inputs.
• Eval sets drift. As your product, policies, and user base change, previously representative examples become stale. Schedule reviews.
• Guardrails cost latency and money. Every additional check is another inference call.
• Retrieval quality sets a ceiling. Better prompting cannot compensate for grounding data that is wrong, outdated, or missing.
• Accuracy and safety are different problems. Microsoft’s guidance keeps content safety as a distinct layer rather than a side effect of the hallucination check.

What to do next

If you have no evaluation set, that is the first task — not a prompt rewrite. Assemble a few dozen labelled examples from real usage, write down your hallucination criterion in one sentence, and measure a baseline. Only then start changing retrieval, prompts, or guardrails, and re-measure after each change. The teams that reduce hallucinations most reliably are not the ones with the cleverest prompt; they are the ones that can tell, with evidence, whether last week’s change worked.

If you are working through this on a specific stack — a local model, a WordPress automation, or a RAG pipeline — the Ai-Forum community is a good place to share your evaluation setup, the criteria you settled on, and the failure cases you cannot classify. Concrete examples from real workflows are more useful to other practitioners than general advice.