Guardrails for a local AI chatbot are the checks that sit on either side of the model: filters that inspect a user’s request before it reaches the chat model, and validators that inspect the model’s response before it reaches the user. Running the chatbot locally does not remove the need for them. It moves responsibility for them out of a hosted provider’s console and into your own stack, where you can implement them as separate model services, prompt-driven classifiers, or plain code.
This guide is for practitioners who already have a local chat model serving requests and want to add a defensible layer of control around it. It covers what guardrails are in a local context, the two architectures most teams choose between, a step-by-step pipeline assembled from a filter model, a chat model, and a refusal model, and the verification and troubleshooting work that determines whether the guardrail actually does anything useful.
What a guardrail is, and what it is not
The OpenAI cookbook describes a guardrail as “a generic term for detective controls that aim to steer your application,” noting that “greater steerability is a common requirement given the inherent randomness of LLMs” (How to use guardrails). That framing matters: a guardrail detects and redirects, it does not make a model safe by construction.
The same source draws the distinction that most local deployments inherit:
- Input guardrails aim to stop inappropriate content from reaching the model at all. The cookbook names three common cases: topical guardrails that identify off-topic questions and redirect the user, detection of jailbreaking attempts that try to hijack the model and override its prompting, and detection of prompt injection where a user hides malicious instructions in content the model will process.
- Output guardrails validate what the model produced before it reaches the customer.
One caveat about the source: the OpenAI recipe is explicitly marked as archived and may reference outdated models or APIs. The architectural patterns it describes have held up, but specific model identifiers in that notebook are historical rather than current recommendations.
Two architectures for a local stack
There are two broad ways to place a guardrail in front of a self-hosted chatbot.
A gate model. A second, usually smaller prompt model receives the user’s request and returns an accept or reject decision based on defined filtering criteria. Only accepted requests are forwarded to the chat model. This is the approach Seldon’s LLM Module tutorial takes: it deploys mistralai/Mistral-7B-Instruct-v0.2 locally alongside “two additional prompt models,” where the first accepts or rejects the request and the second generates the answer if the query passes the checks (Guardrails | LLM Module). The advantage is that the policy lives in natural language and can be edited without redeploying code. The cost is an extra model call on every request.
A code-level validator. Rules, regexes, allowlists, embedding similarity against a topic list, or a small dedicated classifier run inside your application before and after the model call. This is cheaper and more predictable, but the policy is expressed in code and drifts as your product changes.
Most production local deployments combine both: cheap deterministic checks first, a gate model for the ambiguous middle, and output validation on the way back. For a refusal path, the Seldon tutorial adds a third component — a custom MLServer model that ignores the input and returns a fixed warning message.
Prerequisites and a reference stack
Before wiring anything together, confirm you have:
- A local runtime capable of serving more than one model. The Seldon tutorial assumes the Local Runtime, the Prompt Runtime, and an instance of MLServer are running.
- A chat model with a standard chat template. The tutorial uses
mistralai/Mistral-7B-Instruct-v0.2. - A filtering model, or a rule set, that produces an accept/reject decision.
- A refusal path — either a small model or a static response served by a lightweight component.
- Per-model configuration files. In the Seldon setup these are
model-settings.jsonfiles, one per deployed model.
If your hardware cannot hold several models in memory at once, plan for sequential loading or quantized weights rather than assuming everything fits. Weight size is not the whole memory budget: context and KV cache grow with conversation length, and each runtime adds its own overhead.
Building the pipeline step by step
Step 1: Write the policy down before writing code
List the categories you intend to block, the topics that are in scope, and the exact text the user should see when a request is refused. A written policy is what makes the filter model’s prompt testable and the false-positive rate measurable.
Step 2: Deploy the chat model
Serve your chosen instruct model through your local runtime and confirm you can complete a normal chat turn end to end. Do this before adding any guardrail, so that any later failure is attributable to the new component rather than the base setup.
Step 3: Deploy the filter model
Configure a second prompt model whose only job is to return an accept or reject decision against your written criteria. Keep its temperature at or near zero so the same input produces the same decision, and keep its output schema minimal — a single token or short label is easier to route on than free text.
Step 4: Deploy the refusal response
The Seldon tutorial implements the refusal as a custom MLServer model. It ignores the input entirely and returns a fixed message:
from mlserver.model import MLModel
from mlserver.codecs import StringCodec
from mlserver.types import InferenceRequest, InferenceResponse
class MessageModel(MLModel):
async def load(self) -> bool:
self.ready = True
return self.ready
async def predict(self, payload: InferenceRequest) -> InferenceResponse:
return InferenceResponse(
model_name=self.settings.name,
model_version=self.settings.version,
outputs=[
StringCodec.encode_output(
name="role",
payload=["assistant"],
),
StringCodec.encode_output(
name="content",
payload=["I cannot talk about this."],
),
StringCodec.encode_output(
name="type",
payload=["text"],
),
],
)Note the output shape: the response carries role, content, and type fields, which is what lets the refusal be dropped into the same chat stream as a normal answer. Match that schema to whatever your client expects, or the refusal will render as an error rather than a message.
Step 5: Wire the routing logic
Your application, not the model, decides what happens next. The logic is short: send the request to the filter model, read its decision, and either forward the request to the chat model or return the refusal response. Log the decision and the matched policy category alongside a request identifier.
Step 6: Add output validation
Input filtering alone is not enough. Add a second pass on the generated answer that checks for leaked system prompts, disallowed content, malformed structure, or empty output. The cookbook treats output guardrails as a distinct category for exactly this reason.
Step 7: Log, measure, and review
Record which requests were blocked, which policy category matched, and how often. Without this, you cannot tell a working guardrail from one that refuses everything.
Verifying that the guardrail actually fires
A guardrail that has never been tested against a hostile input is an assumption, not a control. Build a small regression set and run it after every policy change:
- A clearly in-scope question, which must pass through to the chat model and produce a normal answer.
- A clearly out-of-scope question, which must return the refusal text.
- A jailbreak-style instruction that tries to override the system prompt, which must be blocked.
- A prompt injection payload embedded in retrieved or pasted content, which must not cause the model to follow the hidden instruction.
- A borderline request, to observe where your filter sits and whether the threshold is defensible.
For each case, check two things: that the routing decision was correct, and that the refusal response did not itself leak the system prompt or internal policy text. Verifying only the decision and not the response text is a common gap.
Tradeoffs, limits, and licensing
Latency. A gate model adds a full inference call before the chat call. On local hardware this is often the dominant cost of adding guardrails, and it is the main reason teams start with deterministic rules and escalate to a model only for ambiguous cases.
False positives. An over-strict filter is a product problem, not a safety win. Every blocked legitimate request erodes user trust, and the refusal message is often the only explanation the user receives.
Guardrails are bypassable. The cookbook’s framing — detective controls — is accurate. A determined user can probe for gaps, and a filter model can be talked around just as a chat model can. Treat guardrails as one layer among several, not as a guarantee.
Maintenance. Policy language drifts. A filter prompt written for one product scope becomes wrong when the product changes, and nobody notices until users complain.
Licensing. Open weights, open source, and permission for commercial use are three different things. Before shipping a guardrail model, read the license that applies to that specific model rather than assuming it matches your chat model’s terms.
Framework options. The cookbook points readers to NeMo Guardrails and Guardrails AI as official libraries that distribute pre-built guardrail components. If you would rather not assemble the pipeline by hand, those are the documented starting points; the cookbook itself focuses on the underlying patterns rather than those libraries.
Troubleshooting common failures
The filter blocks everything. The filter prompt is usually too broad or the criteria list is too long. Narrow the policy to specific categories, add a few labeled examples of acceptable requests, and re-run the regression set.
The filter blocks nothing. Check that the routing logic is actually reading the filter’s output field and not a default value. A schema mismatch — reading content when the model returned a label — fails silently in the permissive direction, which is the dangerous one.
The refusal renders as an error. The refusal component’s output fields do not match what the client expects. Confirm the role, content, and type fields are present and correctly named.
Out-of-memory when loading the pipeline. Too many models are resident at once. Quantize, load sequentially, or collapse the filter and chat roles into a single model that self-checks before answering.
Inconsistent decisions on identical input. Sampling is still enabled on the filter model. Set temperature to zero.
Refusals leak internal text. The refusal is being generated by the chat model rather than returned by a fixed component. Route refusals to a static response instead.
Where to go next
Start with the smallest version that is testable: one deterministic rule, one refusal message, and a five-case regression set. Add the gate model only when you can point to requests that the rules handle badly. Then extend the same pattern to output validation, since the response path is where leaked prompts and malformed content actually reach users.
If you are running this on your own hardware, the useful thing to share with the community is your environment and your result: which runtime, which models, how much memory the full pipeline consumed, and which requests your filter got wrong. That kind of record is what turns a guardrail design into a reproducible setup.
FAQ
Do I need a second language model to run guardrails locally?
Not necessarily. Deterministic rules, allowlists, and small classifiers can handle the clear cases. Seldon's tutorial does use a second prompt model to accept or reject requests, plus a custom MLServer model that returns a fixed refusal message, but that is one design among several.
Can a local guardrail be bypassed?
Yes. The OpenAI cookbook frames guardrails as detective controls, not guarantees. A filter model can be talked around in much the same way a chat model can, so guardrails should be one layer among several rather than the only protection.
Does running guardrails locally keep user data private?
It keeps requests inside your own infrastructure, provided every component in the pipeline — including the filter model and any refusal component — runs locally. Logging and monitoring can still store request content, so review what your pipeline persists.
What should I check first when the guardrail does not block anything?
Verify that the routing logic reads the correct output field from the filter model. A schema mismatch, such as reading a content field when the model returned a short label, fails silently in the permissive direction.