AIForum.

How Temperature, Top-p, and Seed Affect Local Model Output

On this page

THE SHORT ANSWER

Temperature scales how random token selection is, top-p limits which tokens are eligible, and seed fixes the sampler's random starting point. Together they make local model runs far more repeatable — but not identical, because sampling is only one of several sources of variation in an inference stack.
Quick facts

If you have ever re-run the same prompt against a local model and received a noticeably different answer, the sampling parameters are the first place to look — and usually not the last. Temperature, top-p, and seed each change a different part of the token-selection process, and understanding which one does what turns a vague sense of “the model is unpredictable” into a set of settings you can reason about and record.

This guide is for anyone running models locally through runtimes such as llama.cpp, Ollama, LM Studio, or vLLM, and for anyone who needs to explain why two runs that look identical on paper produced different text. It covers what each parameter does, how they interact, and the practical limits of reproducibility.

What each parameter actually controls

Temperature

Temperature is used for sampling during response generation and controls the degree of randomness in token selection. Lower values suit prompts that need a less open-ended or creative response; higher values produce more diverse output. A temperature of 0 means the highest-probability tokens are always selected, which makes responses for a given prompt mostly deterministic — but, as the documentation notes, a small amount of variation is still possible (Google Cloud documentation).

Two practical notes from that same documentation are worth keeping. First, if a model returns something too generic, too short, or a fallback response, the suggested fix is to increase temperature. Second, if generation runs away into an infinite loop, raising temperature to at least 0.1 may improve results. In other words, temperature 0 is not automatically the “safe” setting.

Ranges are model-specific. For Gemini models the documented range is 0.0 to 2.0 with a default of 1.0, and 1.0 is described as the recommended starting value. Do not assume the same bounds apply to a local checkpoint; check the model card or the runtime’s defaults.

Top-p

Top-p, or nucleus sampling, changes how the model selects tokens rather than how strongly it weights them. Instead of considering every token in the vocabulary, it keeps the smallest set of tokens whose cumulative probability reaches p and discards the rest. At top_p = 1.0, no truncation happens and the full distribution is available.

Top-p and temperature are not substitutes. Temperature reshapes the distribution; top-p cuts off its tail. A high temperature with a low top-p is a common combination for keeping output varied without letting very unlikely tokens through.

Top-k

Top-k is the cruder cousin of top-p: it caps the candidate list to the k highest-probability tokens regardless of how the probability mass is distributed. Many runtimes let you disable it explicitly. In the vLLM request shown below, top_k: -1 means top-k is turned off.

Seed

The seed parameter fixes the random number generator that the sampler draws from. With a fixed seed and unchanged sampling settings, the sequence of random draws is reproducible, which is what makes a run repeatable in the first place.

A documented configuration example

The following request shape appears in a write-up on non-determinism in LLM serving as the recipe for seedable sampling in vLLM. It is reproduced here as a documented example, not as a result measured for this article.

curl http://host:8000/v1/chat/completions -d '{
  "model": "your-model",
  "messages": [{"role": "user", "content": "..."}],
  "temperature": 0.0,
  "seed": 42,
  "top_p": 1.0,
  "top_k": -1
}'

Read that block as a template: greedy-ish decoding (temperature 0), no nucleus truncation, top-k disabled, and a fixed seed. It is the most conservative starting point for a reproducible local run — and, as the next section explains, it is still not a guarantee.

Why temperature 0 is not the same as deterministic

The claim that temperature=0 guarantees identical output is one of the most persistent assumptions in local AI work, and it is wrong. The same prompt can produce different tokens across runs for at least seven independent reasons, only the first of which is sampling.

  1. Sampling. With temperature above zero, different RNG seeds select different tokens. This is the easy one to fix, if you know where the seed lives.
  2. Ties at the argmax. Even at temperature 0, two tokens can share the exact same logit. Which one wins depends on the CUDA kernel’s tie-break rule and sometimes on warp scheduling.
  3. Non-associative floating-point math. Floating-point addition is not associative, so the order in which operations are reduced can change results.
  4. Batch-size dependence. Outputs can shift depending on how requests are batched together.
  5. Scheduler-induced drift. How the serving scheduler interleaves work can affect what gets computed and when.
  6. Quantisation and hardware differences. Different quantisation levels and different GPUs do not produce bit-identical arithmetic.
  7. Library and kernel version drift. Upgrading a runtime, a CUDA version, or an attention kernel can change outputs with no change to your prompt or parameters at all.

The tie-breaking point deserves emphasis because it is invisible from the outside. A model can be perfectly “deterministic” in the sense that no randomness is injected, and still return a different token because two candidates were exactly equal and the kernel resolved the tie differently under a different scheduling pattern.

What a seed does and does not fix

A seed fixes the sampler’s random draw. It does not fix the arithmetic, the batching, the scheduler, the quantisation, or the runtime version. That distinction has a concrete consequence: the same seed on two different machines is not a reproducibility guarantee. Even on one machine, changing the batch size or upgrading the inference library can change output while the seed stays at 42.

This is why reproducibility in local inference is best treated as a property of the whole recorded environment — model revision, quantisation, runtime and version, hardware, batch configuration, and sampling parameters — rather than as a property of one field in a request body.

Choosing settings by task

The right settings depend on what you are trying to get out of the run.

  • Reproducible evaluations and regression tests. Temperature 0, top_p 1.0, top-k disabled, a fixed seed, a pinned runtime and model revision, a fixed quantisation, and one request at a time where possible. Record all of it alongside the results.
  • Structured output such as JSON. Low temperature helps, but it does not guarantee valid JSON. Validate the output and retry on failure rather than assuming determinism will hold.
  • Creative writing. Higher temperature is the point. Seed variation is a feature, not a bug.
  • Production chat and agents. Some run-to-run variability is harmless and often desirable.
  • Multi-turn search or retrieval workflows. What matters is final answer quality, not token-by-token identity.

That last group is worth stating plainly: chasing bit-identical output is only worth the effort when you actually need it. For evaluations, CI regression tests, audit or compliance reproduction, and ablation studies, you do. For a chat assistant, you usually do not.

A practical reproducibility checklist

  1. Set temperature, top_p, top_k, and seed explicitly rather than relying on runtime defaults.
  2. Pin the model revision and quantisation, and record both.
  3. Pin the runtime and its version, including CUDA or ROCm version where relevant.
  4. Keep batch composition stable — ideally one request at a time — when comparing outputs.
  5. Re-run the same prompt several times before concluding that settings are stable. A single matching pair proves very little.
  6. When output changes after an upgrade, check the runtime and kernel versions before blaming the parameters.

Where to go next

Start by fixing the parameters you can control: set an explicit seed, decide deliberately between temperature 0 and a small positive value, and pick a top-p that matches how much variety you actually want. Then treat the rest of the stack — quantisation, batching, runtime version — as part of the experiment rather than as background noise. If you are comparing two local models or two prompt variants, record the full configuration next to each result so that a future difference can be attributed rather than guessed at.

If you are working through a reproducibility problem of your own, the community is a good place to compare notes: share your runtime and version, model and quantisation, hardware, the exact sampling parameters, and the two outputs that differed. Those five pieces of information are usually enough to narrow down which of the sources above is responsible.

FAQ

Does temperature 0 guarantee the same output every time?

No. Temperature 0 means the highest-probability tokens are always selected, which makes responses mostly deterministic, but a small amount of variation is still possible. Ties between equal logits, floating-point ordering, batching, and runtime versions can all change the result.

What should top-p be set to?

top_p 1.0 disables nucleus truncation and leaves the full distribution available, which is the usual choice for reproducible runs. Lower values narrow the candidate pool and are typically paired with a higher temperature when you want variety without very unlikely tokens.

Will the same seed give the same output on two different machines?

Not reliably. A seed fixes the sampler's random draw, but it does not control quantisation, hardware arithmetic, batch composition, scheduler behaviour, or library and kernel versions. Reproducibility requires recording the whole environment.

Why did my output change after I upgraded the inference library?

Library and kernel version drift is a recognised source of output variation. Upgrading a runtime, CUDA version, or attention kernel can change results even when the prompt, model, and sampling parameters are unchanged.

References

  1. https://docs.cloud.google.com/gemini-enterprise-agent-platform/models/prompts/adjust-parameter-values
  2. https://brendanjameslynskey.github.io/Local_LLM_09_Determinism/

Related terms

Sampling
The process of choosing the next token from the model's probability distribution. Temperature and seed apply here.
Top-p (nucleus sampling)
A filter that keeps the smallest set of tokens whose cumulative probability reaches p, discarding the rest before selection.
Top-k
A filter that limits candidates to the k highest-probability tokens. Often disabled with a value of -1 in serving runtimes.
Greedy decoding
Always selecting the single highest-probability token. Approximated by temperature 0, though ties can still be resolved differently.
Non-determinism
Run-to-run variation in output that persists even when prompts and parameters are held constant.

Leave a Reply

Your email address will not be published. Required fields are marked *