How to Choose Quantization Settings for Local Models Without Breaking Output Quality

Start with Q4_K_M. It is the format the Hugging Face GGUF quantization guide labels “Recommended,” and it sits at the point on the documented curve where a 7B model file shrinks from 13.0 GB at FP16 to 4.1 GB while measured perplexity moves from 5.9565 to 6.0565 — an increase of about 1.68%. For most local setups, that is the best available balance of quality, memory, and tokens per second.

The rest of this article is about knowing when to deviate from that default, and how to verify that the deviation did not cost you the output quality you actually care about. It is written for technically curious beginners and intermediate practitioners running GGUF models through llama.cpp-family runtimes, and it stays close to documented figures rather than folklore.

What quantization actually changes

Every parameter in a neural network is a floating-point number, and the numeric format decides how many bytes it occupies. FP32 uses 32 bits — one sign, eight exponent, twenty-three mantissa — which is precise to roughly seven significant digits at four bytes per number. FP16 halves that to two bytes but keeps a cramped exponent: its largest representable value is 65,504, and training computations occasionally exceed that. BF16 makes the opposite trade, keeping FP32’s full eight-bit exponent while cutting the mantissa to about two or three significant digits. Because networks tolerate limited precision far better than limited range, BF16 became the standard for training and full-quality inference. FP8 formats are now used in parts of training and serving, and MXFP4 is a 4-bit floating format.

GGUF quantization goes further than simply picking a narrower float. Weights are grouped into blocks that share scale factors, and the K-quant family mixes bit widths across different tensors inside the same file. That is why two files with similar-looking names can behave differently, and why the suffix matters as much as the number in front of it.

Let the model repository choose first

Generic tables are a fallback, not a starting point. The quantization guide is explicit about this: open the model repository with the llama.cpp app query parameter and read the Hardware compatibility section, which lists the exact quant labels and file sizes the publisher recommends.

https://huggingface.co/<repo>?local-app=llama.cpp

Then confirm that the filenames you intend to download actually exist in the repository tree:

https://huggingface.co/api/models/<repo>/tree/main?recursive=true

Only when the repository page does not expose a clear recommendation should you fall back on the generic heuristics below. This ordering matters because quant naming, availability, and suitability vary by architecture. A rule of thumb derived from a dense 7B model does not transfer cleanly to a mixture-of-experts model, a heavily fine-tuned variant, or a model whose publisher deliberately skipped certain formats.

The documented quality, size, and speed curve

The reference table below comes from the Hugging Face skills quantization guide. The perplexity column uses a 7B model as the baseline, and the size and throughput columns describe that same class of model. Treat the numbers as a shape, not as a promise about your specific model or hardware.

Three readings stand out. First, Q8_0 is effectively lossless in this table — a 0.03% perplexity increase at roughly half the FP16 file size. Second, the curve is not linear: moving from Q5_K_M down to Q4_K_M costs about 1.3 percentage points of perplexity but saves 0.7 GB, while moving from Q3_K_M down to Q2_K costs more than nine percentage points for only 0.6 GB. Third, the guide’s own use-case framing is blunt: Q3_K_M is described as suitable for small models only, and Q2_K is not recommended.

How to read the trade-off without over-reading it

Perplexity is a proxy, not a verdict. A 1.68% increase does not mean answers are 1.68% worse; it means the model assigns slightly higher surprise to a held-out text sample. Some tasks — translation, code completion, strict JSON formatting — tend to notice quantization earlier than open-ended chat, because a single wrong token can invalidate an entire structured output. Others absorb it invisibly.

Two further caveats apply. The published throughput figures depend on hardware, runtime version, and how many layers are offloaded to the GPU, so they should not be used to predict your own tokens per second. And degradation does not scale the way intuition suggests across model sizes: smaller models have less redundancy to spare, which is why the guide flags Q3_K_M for small models rather than recommending it broadly.

Memory planning: the weights are only part of the bill

The file sizes in the table describe the weights alone. A running model also needs memory for the key-value cache, which grows with context length and batch size, plus runtime and framework overhead. A 4.1 GB Q4_K_M file can still exhaust an 8 GB card once a long context is loaded.

Plan in this order: pick the quant from the repository’s recommendation, estimate the KV cache for the context length you actually intend to use, add headroom for the runtime, and only then decide how many layers to offload. If the total does not fit, reducing context length is often cheaper in quality terms than dropping another quant level, because context reduction is predictable and reversible while quantization loss is not.

Importance matrices

The guide documents importance matrices, usually called imatrix, as a way to make quantization aware of which weights matter most. Instead of quantizing every tensor uniformly, the process measures activation importance using a calibration corpus and uses that information to protect sensitive weights. In practice this can improve quality at a given bit width, which makes it attractive when you are forced below Q4. Its limits are practical: it requires a representative calibration text, an extra processing step, and it does not rescue very low bit widths such as Q2_K, where the loss comes from the format itself rather than from how the tensors were selected.

A reproducible quality check

The following is a protocol, not a set of measured results. Run it yourself before trusting any quant for production work.

1. Fix the variables. Record the model repository and revision, the exact quant filename, runtime name and version, context length, and sampling settings.
2. Build a small prompt set that reflects your real workload. A handful of prompts is enough if they cover your failure-sensitive cases, such as structured output, multi-step arithmetic, or long-document summarization.
3. Reduce randomness. Use greedy decoding or a fixed seed so differences come from the quant rather than the sampler.
4. Run the same prompts against at least two quant levels, ideally including a high-quality reference such as Q8_0 or FP16.
5. Score for failure modes, not vibes. Malformed JSON, dropped instructions, repeated phrases, wrong units, or invented citations are more informative than a general impression.
6. Repeat on a second day or with a second seed to check that a difference is stable rather than noise.

If you want a numeric comparison rather than a behavioral one, the guide includes a quality testing section, and llama.cpp-family runtimes generally expose perplexity measurement tooling that requires a suitable text corpus. The two approaches answer different questions, and the behavioral check is usually the one that predicts whether your workflow breaks.

Common symptoms and what to check first

These are hypotheses to test in order, not confirmed diagnoses.

• Output degrades into repetition or drift. Check the quant level first, especially if you are at Q3 or below. Compare against Q5_K_M on the same prompts.
• Structured output breaks. This is a frequent early symptom of aggressive quantization. Confirm with a higher quant before rewriting prompts.
• The model loads but generation is slow. Throughput is usually a hardware and offload question rather than a quant question, though lower-bit formats do reduce memory bandwidth pressure.
• Out-of-memory despite a small file. Recalculate with the KV cache and context length included, then reduce context or offloaded layers.
• Quality differs from the repository’s own example. Verify you downloaded a filename listed in the tree endpoint; similarly named files are not interchangeable.

Choosing by scenario

• Default local desktop or single consumer GPU: Q4_K_M.
• Memory is the binding constraint and you accept some loss: Q4_K_S, then re-run your behavioral check.
• Quality-critical work with headroom: Q6_K, or Q8_0 if the file still fits comfortably.
• Models below roughly 3B parameters: stay at Q5 or above; small models have the least redundancy to give up.
• Long-context workloads: budget for the KV cache before spending bits on weights.

Next step: open your model’s repository page with the llama.cpp query parameter, note the recommended quant and its size, then confirm the filename in the tree endpoint. If the recommendation conflicts with your available memory, move one step down the table and validate with the behavioral protocol above rather than guessing. Readers who want to compare notes can share their model, hardware, chosen quant, and any observed output differences in the community discussion.