AIForum.

How to Benchmark Local LLM Speed and Cost per 1K Tokens

On this page

THE SHORT ANSWER

Benchmarking a local LLM means measuring two different things: how fast it generates tokens and what those tokens cost you in hardware, electricity, and time. This guide explains the metrics, shows a reproducible protocol using tools such as llm-bench and LLMPerf, and explains why per-1K-token cost for local models is an amortized estimate rather than a vendor price.
Quick facts

Benchmarking a local LLM is really two measurements wearing one coat. Speed is a runtime property you can observe directly: time to first token (TTFT), tokens per second, and their percentiles under a given concurrency. Cost per 1K tokens is not a price list at all for a local model — it is an amortized estimate derived from your hardware, electricity, and utilization. Getting a useful answer means measuring the first honestly and calculating the second transparently.

This guide is for technically curious beginners and intermediate practitioners running models through Ollama, vLLM, LM Studio, llama.cpp server, or another OpenAI-compatible endpoint. The goal is a benchmark you can rerun and compare, not a single impressive number.

What you are actually measuring

LLM inference performance splits into two phases. Prefill processes the input prompt and determines TTFT; decode generates output tokens one at a time and determines tokens per second. A long prompt hurts TTFT. A long generation hurts total completion time. Concurrency affects both, because batching trades per-request latency for aggregate throughput.

The metrics worth recording, as described in the llm-bench documentation, are time to first token, time per output token, and p50/p90/p95/p99 percentiles alongside mean and standard deviation. Percentiles matter because averages hide the tail that users actually feel. A model with a 400 ms median TTFT and a 3 s p99 will feel worse than one with a 600 ms median and a 900 ms p99.

Cost per 1K tokens for a hosted API is a published rate. For a local model it is an allocation: total cost of ownership divided by tokens served. That distinction is the single most common source of confusion in local benchmarking threads.

Prerequisites before you run anything

  • A fixed environment. Record OS, GPU or CPU, driver version, runtime version, model name, quantization, and context length. Changing any one of these invalidates comparisons.
  • A serving endpoint. Ollama, vLLM, LM Studio, llama.cpp server, or any OpenAI-compatible URL. llm-bench explicitly targets OpenAI, Anthropic, Ollama, OpenRouter, Gemini, and OpenAI-compatible servers.
  • A benchmark tool. llm-bench is a Rust CLI and library that sends the same prompts to several models and measures TTFT, latency percentiles, tokens per second, cost, and success rate. LLMPerf and vLLM’s benchmark_serving.py are Python alternatives aimed at one serving endpoint at a time.
  • A warm-up run. Discard the first request. Model loading and cache state skew early results; llm-bench exposes a warmup setting for exactly this.

A reproducible speed protocol

The following is a suggested protocol, not a recorded test result. Substitute your own model and endpoint.

  1. Fix the prompt set. Use identical prompts across every model you compare. Mix a short prompt and a long-context prompt so prefill cost is visible.
  2. Fix sampling. Keep temperature, top-p, and max output tokens constant. Varying max tokens makes tokens-per-second comparisons meaningless.
  3. Run multiple repetitions. Ten runs per prompt with concurrency 2 is a reasonable starting point; increase concurrency separately to study throughput.
  4. Enable streaming. TTFT is only measurable when the response streams token by token.
  5. Write a summary file. Percentiles and standard deviation are more useful than a single run.

A minimal llm-bench invocation looks like this:

llm-bench run 
  --models gpt-4o-mini,claude-haiku-4-5,ollama:qwen2.5:7b 
  --prompts "Say hi" 
  --stream 
  --summary-file summary.json

For a purely local comparison, drop the hosted models and point at your local server. The library form is compact enough to embed in a script:

use llm_bench::{runner::BenchRunner, stats::model_reports, types::{BenchConfig, ProviderConfig}};

let ollama = ProviderConfig {
    name: "ollama".into(),
    model: "qwen2.5:7b".into(),
    api_key: String::new(),
    base_url: String::new(),
    max_tokens: 128,
};

let config = BenchConfig {
    prompts: vec!["Say hi".into()],
    runs_per_prompt: 10,
    concurrency: 2,
    providers: vec![ollama],
    stream: true,
    retries: 2,
    warmup: 1,
};

let (results, failures) = BenchRunner::new()?.run_detailed(&config, |_, _| {}).await?;

for m in model_reports(&results, &failures) {
    println!("{}: TTFT p95 {:?} ms", m.model, m.ttft_ms.map(|s| s.p95));
}

Expected observable result: a summary file containing per-model TTFT percentiles, time per output token, tokens per second, and success rate. If a model shows a high failure rate, fix that before interpreting speed — a fast model that drops requests is not fast.

Turning speed into cost per 1K tokens

There is no vendor invoice for a local model, so build the estimate from parts and show your assumptions:

  1. Hardware cost. Purchase price of the machine or GPU, divided by expected service life in hours.
  2. Electricity. Measured or rated power draw in kW multiplied by your local tariff and by hours run.
  3. Utilization. The fraction of those hours actually serving tokens. A box that runs four hours a day at full load costs far more per token than one under continuous load.
  4. Tokens served. Take the throughput you measured, in tokens per second, and multiply by productive seconds.

Divide total cost by total tokens, then multiply by 1,000. Because utilization is an assumption, always publish the number alongside it. A per-1K-token figure quoted without utilization and power assumptions is not reproducible.

For context on hosted pricing, the Runpod benchmarking article notes that OpenAI API endpoints are charged by tokens in and tokens out, with figures around $5 per 1M input tokens and $2.5 per 1M output tokens at the time of writing. That is roughly $0.005 per 1K input tokens and $0.0025 per 1K output tokens — useful as a comparison anchor, but check current pricing before relying on it.

Where local benchmarking goes wrong

Comparing across quantizations. A 4-bit model and an 8-bit model of the same family are different systems. Label them separately.

Ignoring context length. Throughput at a 512-token context says little about behavior at 32K. KV cache growth also increases memory use, so weight size alone is a poor memory estimate.

Reporting one run. Thermal throttling, background processes, and model loading make single runs unreliable. Percentiles from repeated runs are the minimum credible evidence.

Mixing prompt lengths. A short prompt inflates tokens per second because prefill is cheap. Hold the prompt set constant.

Treating cost as a single number. Local cost per 1K tokens depends on amortization choices. Two people with identical hardware can honestly report different figures.

Choosing a tool

llm-bench is the more convenient option if you want hosted and local models in one table, with built-in prices for a set of OpenAI and Anthropic models and streaming percentiles. LLMPerf and vLLM’s benchmark_serving.py are better suited if you are benchmarking a single serving endpoint and want Python integration. The metric families overlap; the difference is workflow, not physics.

What to do next

Pick one model, one quantization, and one endpoint. Run the protocol above, save the summary file, and record your hardware and power assumptions next to it. Then change exactly one variable — concurrency, context length, or quantization — and run again. That single-variable discipline is what turns a benchmark into evidence you can act on. If your numbers look odd, share your environment, prompt set, and summary file with the community so others can reproduce or challenge them.

FAQ

Can I compare a local model's cost per 1K tokens directly to an API price?

Only with caveats. API pricing is a published per-token rate, while local cost per 1K tokens is an amortized estimate that depends on hardware purchase price, electricity tariff, and utilization. Publish those assumptions alongside the number or the comparison is not reproducible.

How many runs do I need for a trustworthy local benchmark?

There is no universal minimum, but a single run is not reliable because of model loading, thermal throttling, and background load. Ten runs per prompt with one discarded warm-up and a concurrency setting is a common starting point, and percentiles from those runs are more informative than a mean alone.

Why does my tokens-per-second number drop when I raise concurrency?

Batching increases aggregate throughput but can increase per-request latency, because concurrent requests share compute. Measure per-request tokens per second and aggregate throughput separately, and report the concurrency level with both.

References

  1. https://docs.rs/llm-bench/latest/llm_bench/
  2. https://www.runpod.io/blog/benchmark-local-llm-inference-performance

Related terms

TTFT (time to first token)
The delay between sending a request and receiving the first streamed output token; dominated by prompt prefill.
Tokens per second
Decode throughput — how many output tokens the model produces per second during generation.
Percentile latency
The value below which a given share of measurements fall, such as p95; it exposes tail behavior that averages hide.
Quantization
Reducing the numeric precision of model weights (for example to 4-bit or 8-bit) to lower memory use, usually with some quality and speed tradeoff.
KV cache
Memory used to store attention keys and values for the conversation context; it grows with context length and adds to total memory use beyond model weights.

Leave a Reply

Your email address will not be published. Required fields are marked *