AIForum.

How to Use System Prompts and Chat Templates with Local AI Models

On this page

THE SHORT ANSWER

System prompts and chat templates are two separate layers: the system prompt is what you tell the model, and the chat template is how that message becomes tokens. Local open-weight models are far less forgiving of a mismatch than cloud APIs — the wrong control tokens can cost 20–50% of instruction-following quality. This guide explains how the major model families differ, how to inspect a template before running anything, and how to diagnose the symptoms of a mismatch.
Quick facts

A system prompt and a chat template are two different layers of the same problem. The system prompt is what you tell the model; the chat template is how that message is serialized into tokens before the model ever sees it. Cloud APIs hide the second layer. With local models you own both, and getting the second one wrong can cost you a large fraction of the model’s instruction-following quality — the Local LLM Prompting Guide puts the loss at 20–50% when you feed a model the wrong template tokens [1].

This guide is for anyone running open-weight models through llama.cpp, Ollama, LM Studio, vLLM, or Transformers and wondering why a prompt that works on GPT-5 or Claude produces rambling or ignored instructions locally. It covers what chat templates are, how the major model families differ, how to inspect a template before you run anything, and how to diagnose the symptoms of a mismatch.

What a chat template actually is

Every causal language model, chat-trained or not, does one thing: it continues a sequence of tokens. Chat fine-tuning does not change that. It trains the model on data formatted as a sequence of messages, so the model learns to expect control tokens such as <|user|>, <|assistant|>, or <|end_of_message|> that mark where each turn begins and ends [2].

The chat template is the piece of code that converts a list of {"role": ..., "content": ...} dictionaries into that exact token sequence. It is not a suggestion or a style preference. It is the format the model was trained on, and the model has no way to know you meant something else.

Two models fine-tuned from the same base checkpoint can still use different templates. Hugging Face’s documentation demonstrates this with two Mistral-7B derivatives and notes that different models may use different formats or control tokens even when they share a base model [2]. That is why copying a prompt between local models is not the same as copying it between cloud APIs.

Where the system prompt lives in the token stream

Most modern chat templates accept a system role and place it at the start of the conversation, before the first user turn. That is the conventional meaning of “system prompt”: persistent instructions that stay in context for the whole exchange, separate from the user’s actual question.

Not every family supports it natively. Gemma’s template uses <start_of_turn> markers and has no native system role [1]. In practice, system-level instructions for Gemma get folded into the first user turn, because there is no dedicated slot for them. DeepSeek R1 is a more aggressive case: the Local LLM Prompting Guide reports that R1 silently degrades if you add a system prompt at all [1]. “Silently” is the important word — you do not get an error, you get worse output.

This yields the first practical rule: before writing a long system prompt, confirm that the model family you are running has a system role and that your runtime is actually using it.

Family-by-family template reference

The table below summarizes the template tokens and tool-calling conventions documented for the major local model families. Treat it as a starting point for verification, not a substitute for reading the model card — vendors revise templates between releases.

FamilyTemplate tokensSystem roleTool-call format
Llama 3<|begin_of_text|>, <|start_header_id|>YesRaw JSON in the assistant turn
Qwen / Phi-4ChatML: <|im_start|>YesHermes-style <tool_call>{...}</tool_call>
Gemma<start_of_turn>No native system roleNot specified in the source
Mistral[INST]...[/INST]Varies by versionOpenAI-compatible
DeepSeek R1Hermes-style tool blocksDegrades with a system prompt<tool_call>{...}</tool_call>

Two entries deserve elaboration. First, Qwen 3 expects its own /think toggle, which is separate from the chat template but lives in the same prompt-construction layer [1]. Second, the tool-calling column is where most integration bugs originate: three families, three output shapes, for the same conceptual request.

Inspect the template before you run anything

If you are using the Transformers library, you do not have to guess. The tokenizer carries its chat template, and apply_chat_template renders it to a string you can read [2].

from transformers import AutoTokenizer

tok = AutoTokenizer.from_pretrained("mistralai/Mistral-7B-Instruct-v0.1")

chat = [
    {"role": "user", "content": "Hello, how are you?"},
    {"role": "assistant", "content": "I'm doing great. How can I help you today?"},
    {"role": "user", "content": "I'd like to show off how chat templating works!"},
]

# The raw template string the tokenizer will use
print(tok.chat_template)

# The rendered prompt, with the generation prompt appended
print(tok.apply_chat_template(chat, tokenize=False, add_generation_prompt=True))

For the Mistral example above, the rendered output begins with <s>[INST] Hello, how are you? ... [2]. If your runtime produces something different for the same model, your runtime is not using the model’s template.

Expected result: the printed template contains the control tokens listed in the table for your family, and the rendered prompt ends with the assistant-turn opener that add_generation_prompt=True inserts. If it does not, stop and fix that before tuning anything else.

If you are not using Transformers

llama.cpp, Ollama, LM Studio, and vLLM each ship their own template handling. The check is the same in spirit: find the template string in the runtime’s model configuration and compare it against the model card. Many runtimes let you override the template explicitly, which is the correct fix when a bundled template is outdated relative to the model weights.

Sampler defaults are part of the same problem

Template tokens decide whether the model understands the structure. Sampler settings decide whether the output is coherent once it does. The Local LLM Prompting Guide lists vendor-specified defaults that differ substantially between families:

ModelDocumented sampler defaults
Qwentemperature 0.7, top_p 0.8, top_k 20
Mistral 3.2 (tool calls)temperature 0.15, top_p 1.0
Phi-4temperature 0.7, top_p 0.95

The guide’s warning is specific: use the wrong defaults and you get either repetition loops or incoherent output [1]. Note the Mistral tool-calling figures in particular — a temperature of 0.15 is far below what most people use for chat, and it exists because tool calls need deterministic JSON rather than creative prose.

Writing system prompts that survive the template

Once the plumbing is correct, the system prompt itself still has to do work. A few practices follow from how local models behave:

  1. Keep the system prompt short and structural. Local models are described as less forgiving than cloud models with vague prompts [1]. Short lists of explicit rules outperform long narrative descriptions.
  2. Do not duplicate instructions across roles. If the system prompt says “always answer in JSON” and the user turn says “explain in detail,” you have created a conflict the model must resolve arbitrarily.
  3. Match the format to the family. If the family has no native system role, put the persistent instructions at the top of the first user message and keep them there for every turn.
  4. Separate thinking from answering. For families with an explicit reasoning toggle, the system prompt should not try to imitate that toggle in prose.
  5. Version your system prompt. Because template mismatches and prompt changes produce similar-looking bad output, keeping the prompt in a file with a version label makes diagnosis much faster.

Diagnosing a mismatch

The failure modes of a wrong template and a bad prompt overlap, which is why the diagnostic order matters. Work from the cheapest check to the most expensive.

  1. Symptom: the model ignores instructions entirely. Check the rendered prompt first. If the control tokens do not match the family, that is almost certainly the cause.
  2. Symptom: output is coherent but drifts off-format after a few turns. Check whether the system prompt is re-sent on every turn and whether the template places it correctly.
  3. Symptom: repetition loops. Check sampler settings against the vendor defaults above before rewriting the prompt [1].
  4. Symptom: tool calls arrive as malformed JSON. Check which tool-call convention the model emits. A parser expecting Hermes-style <tool_call> blocks will fail on Llama 3’s raw JSON, and vice versa [1].
  5. Symptom: quality dropped after switching models. Re-render the prompt with the new tokenizer. This is the most common cause and the easiest to rule out.

A reproducible check

A useful experiment is to hold the conversation content constant and vary only the template. Render the same three-message chat with the correct tokenizer and with a deliberately wrong one, then compare outputs at a fixed seed and identical sampler settings. This is an illustrative protocol rather than a reported measurement — its value is that it isolates the template variable, which is otherwise hard to see. Record the model revision, quantization, context length, sampler settings, and the exact rendered prompt alongside the output so someone else can reproduce it.

Limitations and what the sources do not cover

The template and sampler details above come from a community-maintained prompting catalog and Hugging Face’s chat templating documentation [1][2]. The catalog is explicit that its badges distinguish vendor model card claims from behavior verified on the authors’ own hardware, and not every entry has been verified. Model cards are revised; a template that was correct for one release may not be correct for the next. Where a family is missing from the table above — for example, Gemma’s tool-calling format — the source does not specify it, and it should be confirmed against the model card rather than assumed.

There is also no universal “best” system prompt. Templates and samplers are mechanical constraints you can verify; prompt content is a design decision that depends on your task, your evaluation criteria, and the model you actually run.

Next steps

If you are setting up a local model for the first time, render its chat template and read it before writing a single instruction. If you already have a working setup, compare your runtime’s template against the model card, then compare your sampler settings against the vendor defaults. Those two checks resolve a large share of “this model is bad at following instructions” reports. If your setup still misbehaves, share the model name and revision, runtime and version, the rendered prompt, sampler settings, and the exact output in the community — that combination is what makes a problem reproducible for someone else.

FAQ

Why does my local model ignore my system prompt?

The most common cause is a template mismatch: the model is not receiving the control tokens it was trained on. Render the prompt with the model's own tokenizer and compare the output against the model card. A second possibility is that the family has no native system role, as with Gemma, or degrades when one is supplied, as reported for DeepSeek R1.

Can I use the same prompt on Llama 3 and Qwen?

The message content can be similar, but the serialized prompt cannot be identical. Llama 3 uses and , while Qwen uses ChatML's . Let each model's own tokenizer or runtime template handle the conversion rather than pasting a pre-rendered prompt.

Does DeepSeek R1 support a system prompt?

The Local LLM Prompting Guide reports that R1 silently degrades if you add a system prompt at all. There is no error message, only worse output, so persistent instructions are better placed at the start of the user turn.

How do I find out which chat template my model uses?

In Transformers, print tokenizer.chat_template to see the raw template and apply_chat_template(chat, tokenize=False) to see the rendered prompt. In other runtimes, locate the template string in the model configuration and compare it against the model card, overriding it if it is outdated.

Do sampler settings matter as much as the template?

They address a different failure. A wrong template makes the model misread the conversation structure; wrong samplers produce repetition loops or incoherent text. Vendor defaults differ substantially between families, so check both when output quality drops.

References

  1. https://www.runlocalai.co/prompting
  2. https://huggingface.co/docs/transformers/chat_templating

Related terms

Chat template
The code that converts a list of role/content messages into the exact token sequence a chat model was trained on.
Control token
A special token such as or that marks turn boundaries in the token stream.
System role
The message role used for persistent instructions that stay in context across the whole conversation. Not every model family supports it natively.
ChatML
A chat format built around and markers, used by Qwen and Phi-4 among others.
Hermes-style tool call
A tool-calling convention that wraps the call payload in {...} blocks, used by Qwen and DeepSeek R1.
apply_chat_template
A tokenizer method that renders a message list into the model's expected prompt string or token IDs.

Leave a Reply

Your email address will not be published. Required fields are marked *