How to Record a Reproducible Local Model Benchmark

Recording a reproducible local model benchmark means writing down everything another person would need to reproduce your run: the exact model bytes, the runtime and its version, the hardware and power state, the context and cache settings, the full command, and the individual measurements rather than one headline number. Without those fields a result is an anecdote. It may be interesting, but it cannot be compared or verified.

This guide is written for technically curious beginners and intermediate practitioners who already run local models and want results that survive a rerun. It follows two published protocols: the Local AI Stats reproducible test worksheet and Hermanity’s local LLM benchmarking guide. Neither is a source of benchmark scores. Both describe how to record them, and both are explicit that the record matters as much as the number.

Start with the decision, not the model

A benchmark is only meaningful relative to a use case. Hermanity’s guide puts this first: write down the decision the benchmark must support, then choose tasks and metrics that can actually change that decision. Typical decisions for local model users include:

• Which local model is reliable enough for private summarization?
• Which checkpoint is best for code generation on a single 24 GB GPU?
• Does a larger quantized model outperform a smaller higher-precision model?
• How much latency does a long context add?
• Where should an agent route structured JSON, debugging, or creative work?
• Is the local model a credible substitute for a metered API on this workload?

These questions determine which measurements matter. A latency question needs time to first token and generation speed at a fixed context. A memory question needs peak RAM and VRAM. A routing question needs quality and failure rates on the same prompts. Deciding this after the run usually means running it again.

The configuration manifest: what to record before you run

The Local AI Stats worksheet asks for a long list of fields, and the reason is that each one is a variable that can move the result. Group them into a manifest you fill in before and during the run.

The worksheet also instructs you to leave unknown fields blank rather than guess. That instruction matters: a blank field is an honest gap, while a plausible guess becomes a false variable the next person cannot reproduce.

Hermanity’s guide explains why the hash and version fields exist. Without them, a later rerun can silently test different bytes under the same model name. Quantization level, runtime version, prompt template and sampling parameters all change results, so “a 30B model at a low-bit quantization” is not a sufficient identity for anything you intend to publish.

Measure each workload separately

The Local AI Stats protocol is short and specific. Follow it in order.

1. Close unrelated workloads. Record idle memory and power before the test so the baseline is known.
2. Warm up the model. Repeat each fixed workload at least five times, then keep the individual results, the median and the spread. The protocol notes its own limit: five runs do not by themselves prove statistical significance.
3. Separate prompt processing from token generation. Do not compare aggregate multi-user throughput with single-user chat.
4. Test short and longer contexts separately. Record time to first token, peak RAM and VRAM, swapping and failed runs.
5. Use the same task prompts for quality checks. Faster generation does not establish better answers.

Prompt processing and token generation are different numbers

Prompt processing (prefill) is the work of reading the input; token generation (decode) is the work of producing output one token at a time. They scale differently with context length and hardware, and collapsing them into a single “tokens per second” figure hides the difference. The same caution applies to throughput: aggregate multi-user throughput is not comparable with single-user chat, because the two measure different workloads.

A minimal llama-bench example

The official llama-bench documentation describes repetitions, prompt and generation lengths, and machine-readable output. The Local AI Stats guide gives this example:

llama-bench -m /path/to/model.gguf -p 512 -n 128 -r 5 -o json > result.json

Read the caveats attached to that example, because they are essentially the point of this article. It measures prompt processing and generation under the tool’s defaults. It does not specify a long conversation context and does not fix GPU offload. The effective settings must be recorded alongside the output, and the output must not be labelled as an end-to-end application benchmark. Pin a runtime version and check its help output before relying on any flag.

If you use this command, the JSON file is your raw measurement log. Keep it. The manifest and the raw log together are the record; a summary written from memory is not.

Context length and memory deserve their own sweep

Context length changes both latency and memory, so a single context setting answers very little. The Local AI Stats guide suggests building a context-versus-memory sweep at 2K, 8K and 32K tokens, recording peak RAM and VRAM, swapping and failed runs at each point.

One practical warning applies here. The size of the weight file is not the total memory a run needs. The KV cache grows with context length and cache precision, and the runtime adds its own overhead on top. A model that fits comfortably at a short context can swap or fail at a long one, which is exactly the kind of result a sweep is designed to surface.

Quality, latency, memory and failure rates

Hermanity’s guide frames a useful benchmark as answering more than “how fast did one prompt run?”, and lists quality, latency, memory, model identity, task fit and failure rates as the dimensions to cover. Two of those are easy to skip.

Failure rate is a measurement, not noise. A run that crashes, hangs, produces malformed output, or silently falls back to a different served model is data about whether the setup is usable for the intended task. Record it instead of quietly rerunning until it works.

Task fit is the dimension that connects numbers to the decision. The same checkpoint can be a good choice for summarization and a poor one for structured output. Using the same task prompts for the quality check, as the worksheet protocol requires, is what makes two runs comparable at all.

Hermanity describes its own Model Bench as combining custom operational tasks with recognized industry suites, with local checkpoints run on an RTX 3090 Ti through pinned runtimes and hosted or subscription models accessed through their real paths. That is a description of one project’s method, not a result you can copy. The transferable parts are the pinning and the separation of access paths.

An illustrative worksheet layout

The table below is an illustrative layout for a single run, not a set of results. Fill it with your own values, and leave fields blank when you do not know them.

What breaks a benchmark record

• Recording the model name but not the checkpoint, quantization or file hash, so a rerun tests different bytes under the same name.
• Reporting one run, or a median with no individual values and no spread.
• Combining prompt processing and token generation into one throughput figure.
• Comparing aggregate multi-user throughput with single-user chat.
• Testing one context length and then generalizing to long conversations.
• Reporting weight-file size as total memory use.
• Dropping failed runs from the record.
• Presenting a microbenchmark as an end-to-end application benchmark.
• Omitting the runtime version, drivers or power mode, so the environment cannot be recreated.

Limits and honest reporting

Two limits are stated plainly by the sources and are worth repeating. First, five repetitions per workload is a floor for a repeatable protocol; the Local AI Stats guide notes that five runs do not by themselves prove statistical significance, so spread and individual values should be published rather than a single figure. Second, results are environment-specific. Hardware, drivers, power mode, runtime version and context settings all move the numbers, which is why Hermanity’s guide frames the goal as building evidence for routing decisions rather than crowning a universal winner.

There is also a reporting limit. If you did not measure something, say so. A blank field and a stated verification gap are more useful to the community than an estimate presented as a measurement.

Next steps

Pick one decision you actually need to make, fill in the manifest fields you can supply today, and run the protocol on a single workload at two context lengths. Keep the raw JSON or CSV log next to the manifest. If a field is unknown, leave it blank and note it as a gap.

If you publish the record, publish the environment with it. Readers in the Ai-Forum.org community can compare setups, spot missing variables, and share their own manifests, including the runs that failed, which are often the most useful entries. Share your goal, environment, result or error in the community discussion so others can reproduce or challenge it.