Benchmarking a local LLM is really two measurements wearing one coat. Speed is a runtime property you can observe directly: time to first token (TTFT), tokens per second, and their percentiles under a given concurrency. Cost per 1K tokens is not a price list at all for a local model — it is an amortized estimate derived from your hardware, electricity, and utilization. Getting a useful answer means measuring the first honestly and calculating the second transparently.
This guide is for technically curious beginners and intermediate practitioners running models through Ollama, vLLM, LM Studio, llama.cpp server, or another OpenAI-compatible endpoint. The goal is a benchmark you can rerun and compare, not a single impressive number.
What you are actually measuring
LLM inference performance splits into two phases. Prefill processes the input prompt and determines TTFT; decode generates output tokens one at a time and determines tokens per second. A long prompt hurts TTFT. A long generation hurts total completion time. Concurrency affects both, because batching trades per-request latency for aggregate throughput.
The metrics worth recording, as described in the llm-bench documentation, are time to first token, time per output token, and p50/p90/p95/p99 percentiles alongside mean and standard deviation. Percentiles matter because averages hide the tail that users actually feel. A model with a 400 ms median TTFT and a 3 s p99 will feel worse than one with a 600 ms median and a 900 ms p99.
Cost per 1K tokens for a hosted API is a published rate. For a local model it is an allocation: total cost of ownership divided by tokens served. That distinction is the single most common source of confusion in local benchmarking threads.
Prerequisites before you run anything
- A fixed environment. Record OS, GPU or CPU, driver version, runtime version, model name, quantization, and context length. Changing any one of these invalidates comparisons.
- A serving endpoint. Ollama, vLLM, LM Studio, llama.cpp server, or any OpenAI-compatible URL. llm-bench explicitly targets OpenAI, Anthropic, Ollama, OpenRouter, Gemini, and OpenAI-compatible servers.
- A benchmark tool. llm-bench is a Rust CLI and library that sends the same prompts to several models and measures TTFT, latency percentiles, tokens per second, cost, and success rate. LLMPerf and vLLM’s
benchmark_serving.pyare Python alternatives aimed at one serving endpoint at a time. - A warm-up run. Discard the first request. Model loading and cache state skew early results; llm-bench exposes a
warmupsetting for exactly this.
A reproducible speed protocol
The following is a suggested protocol, not a recorded test result. Substitute your own model and endpoint.
- Fix the prompt set. Use identical prompts across every model you compare. Mix a short prompt and a long-context prompt so prefill cost is visible.
- Fix sampling. Keep temperature, top-p, and max output tokens constant. Varying max tokens makes tokens-per-second comparisons meaningless.
- Run multiple repetitions. Ten runs per prompt with concurrency 2 is a reasonable starting point; increase concurrency separately to study throughput.
- Enable streaming. TTFT is only measurable when the response streams token by token.
- Write a summary file. Percentiles and standard deviation are more useful than a single run.
A minimal llm-bench invocation looks like this:
llm-bench run
--models gpt-4o-mini,claude-haiku-4-5,ollama:qwen2.5:7b
--prompts "Say hi"
--stream
--summary-file summary.json
For a purely local comparison, drop the hosted models and point at your local server. The library form is compact enough to embed in a script:
use llm_bench::{runner::BenchRunner, stats::model_reports, types::{BenchConfig, ProviderConfig}};
let ollama = ProviderConfig {
name: "ollama".into(),
model: "qwen2.5:7b".into(),
api_key: String::new(),
base_url: String::new(),
max_tokens: 128,
};
let config = BenchConfig {
prompts: vec!["Say hi".into()],
runs_per_prompt: 10,
concurrency: 2,
providers: vec![ollama],
stream: true,
retries: 2,
warmup: 1,
};
let (results, failures) = BenchRunner::new()?.run_detailed(&config, |_, _| {}).await?;
for m in model_reports(&results, &failures) {
println!("{}: TTFT p95 {:?} ms", m.model, m.ttft_ms.map(|s| s.p95));
}
Expected observable result: a summary file containing per-model TTFT percentiles, time per output token, tokens per second, and success rate. If a model shows a high failure rate, fix that before interpreting speed — a fast model that drops requests is not fast.
Turning speed into cost per 1K tokens
There is no vendor invoice for a local model, so build the estimate from parts and show your assumptions:
- Hardware cost. Purchase price of the machine or GPU, divided by expected service life in hours.
- Electricity. Measured or rated power draw in kW multiplied by your local tariff and by hours run.
- Utilization. The fraction of those hours actually serving tokens. A box that runs four hours a day at full load costs far more per token than one under continuous load.
- Tokens served. Take the throughput you measured, in tokens per second, and multiply by productive seconds.
Divide total cost by total tokens, then multiply by 1,000. Because utilization is an assumption, always publish the number alongside it. A per-1K-token figure quoted without utilization and power assumptions is not reproducible.
For context on hosted pricing, the Runpod benchmarking article notes that OpenAI API endpoints are charged by tokens in and tokens out, with figures around $5 per 1M input tokens and $2.5 per 1M output tokens at the time of writing. That is roughly $0.005 per 1K input tokens and $0.0025 per 1K output tokens — useful as a comparison anchor, but check current pricing before relying on it.
Where local benchmarking goes wrong
Comparing across quantizations. A 4-bit model and an 8-bit model of the same family are different systems. Label them separately.
Ignoring context length. Throughput at a 512-token context says little about behavior at 32K. KV cache growth also increases memory use, so weight size alone is a poor memory estimate.
Reporting one run. Thermal throttling, background processes, and model loading make single runs unreliable. Percentiles from repeated runs are the minimum credible evidence.
Mixing prompt lengths. A short prompt inflates tokens per second because prefill is cheap. Hold the prompt set constant.
Treating cost as a single number. Local cost per 1K tokens depends on amortization choices. Two people with identical hardware can honestly report different figures.
Choosing a tool
llm-bench is the more convenient option if you want hosted and local models in one table, with built-in prices for a set of OpenAI and Anthropic models and streaming percentiles. LLMPerf and vLLM’s benchmark_serving.py are better suited if you are benchmarking a single serving endpoint and want Python integration. The metric families overlap; the difference is workflow, not physics.
What to do next
Pick one model, one quantization, and one endpoint. Run the protocol above, save the summary file, and record your hardware and power assumptions next to it. Then change exactly one variable — concurrency, context length, or quantization — and run again. That single-variable discipline is what turns a benchmark into evidence you can act on. If your numbers look odd, share your environment, prompt set, and summary file with the community so others can reproduce or challenge them.
FAQ
Can I compare a local model's cost per 1K tokens directly to an API price?
Only with caveats. API pricing is a published per-token rate, while local cost per 1K tokens is an amortized estimate that depends on hardware purchase price, electricity tariff, and utilization. Publish those assumptions alongside the number or the comparison is not reproducible.
How many runs do I need for a trustworthy local benchmark?
There is no universal minimum, but a single run is not reliable because of model loading, thermal throttling, and background load. Ten runs per prompt with one discarded warm-up and a concurrency setting is a common starting point, and percentiles from those runs are more informative than a mean alone.
Why does my tokens-per-second number drop when I raise concurrency?
Batching increases aggregate throughput but can increase per-request latency, because concurrent requests share compute. Measure per-request tokens per second and aggregate throughput separately, and report the concurrency level with both.