Choosing a local AI model is a hardware-fit problem before it is a model-quality problem. The useful decision sequence is: define the job, find the memory budget, choose a runtime and file format, estimate weights plus KV cache plus runtime overhead, then pick the smallest model and quantization that fits with headroom. Hardware sets the ceiling, the task sets the shape, and the first model you run should be small enough to get working before you optimize (Local AI Zone).
Start with the job, not the leaderboard
The first step is to write down the job. Local AI Zone gives four examples: ‘Summarise my notes’, ‘help me write code’, ‘answer questions about a pile of documents’, and ‘talk to me’. Each of these rewards a different model. A general chat model is a reasonable default if the job is fuzzy, but if you already know the work is code, choose a model trained for code (Local AI Zone).
Three terms appear on almost every download page. Parameters describe a model’s size. Quantization describes how compressed its weights are. Inference is the act of running the model. You do not need to understand these deeply before you begin, but they determine whether a model will fit and how fast it will run (Local AI Zone).
Find the real memory budget
Your memory budget is the VRAM on your graphics card, or on an Apple Silicon machine the unified memory shared between CPU and GPU. Take the memory you have and subtract what the rest of the system needs. On a discrete card, the operating system and desktop already hold some VRAM. On a unified-memory laptop, the model competes with everything else you have open. Treat the number you are left with as a hard ceiling, not a target (Local AI Zone).
Check whether the model will fit comfortably in the memory available to your runtime. That means accounting for the KV cache, runtime allocations, and the rest of the system, not just the model weights (Kilo Blog).
You can estimate weight memory with a simple formula (Kilo Blog):
weights_memory ≈ parameters × bits_per_weight ÷ 8By that calculation, a 27B model at FP16 needs roughly 54 GB of VRAM or unified memory, before accounting for the additional memory required to hold context and runtime overhead (Kilo Blog).
The KV cache can be estimated separately. For a model using an FP16 or BF16 KV cache, use 2 bytes per element; an 8-bit KV cache uses roughly 1 byte per element. Multiply by batch size if you process more than one sequence at a time (Kilo Blog).
kv_cache ≈ 2 × layers × context_tokens × kv_heads × head_dimension × bytes_per_elementFor example, a model with 48 layers, 8 KV heads, a head dimension of 128, and a 32,768-token context requires about 6 GB for an FP16 KV cache. Add that to the model’s loaded size, then leave further headroom for the runtime and operating system (Kilo Blog).
A discrete GPU has its own high-bandwidth VRAM, so models that fit entirely within it are generally preferable for local inference. Unified memory can run larger models than a small discrete GPU, but it is shared with the rest of the system, so the practical ceiling is lower than the total memory figure suggests (Kilo Blog, Local AI Zone).
Choose the runtime before the model file
Picking the app first helps, because the engine inside it decides what file formats you can use (Local AI Zone). Once you know the runtime, you can select the right quantization level and model file format for that engine (Kilo Blog).
Quantization level changes the bits per weight, which directly changes the weight memory estimate. Fewer bits per weight usually lowers memory use, but it is a tradeoff: you should verify quality on your own task rather than assuming a smaller file is equivalent. The right choice balances performance against inference speed, because a model that is too slow for your workflow is not useful even if it is more capable (Kilo Blog).
A repeatable selection procedure
- Write down the job and what a successful result looks like. Use the categories from Local AI Zone: summarisation, code, document question answering, or conversation (Local AI Zone).
- Choose a runtime or app. Note which model file formats it supports, because this limits your choices (Local AI Zone).
- Calculate your memory budget. Subtract operating system, desktop, and other open applications from total VRAM or unified memory. Treat the result as a hard ceiling (Local AI Zone).
- Estimate weight memory with the parameters × bits per weight ÷ 8 formula (Kilo Blog).
- Estimate KV cache for the context length you actually need. Longer context increases the cache (Kilo Blog).
- Add headroom for runtime allocations and the operating system (Kilo Blog).
- Pick the smallest model and quantization that fit, meet your task, and run at an acceptable speed.
- Test with representative prompts. Check both output quality and tokens per second or another speed measure that matters to you.
- If the model does not fit or is too slow, reduce model size, reduce context length, or choose a lower-bit quantization. If quality is poor, move in the opposite direction.
Illustrative example: a 16 GB memory budget
This is an illustrative calculation, not a tested benchmark. Suppose you have 16 GB of memory available to the runtime after subtracting the operating system and other applications. A 27B FP16 model needs roughly 54 GB for weights alone, so it is not viable on that budget (Kilo Blog). You would need a smaller model or a lower-bit quantization.
Even when weights fit, context can consume significant memory. The example KV cache from Kilo Blog uses about 6 GB for a 48-layer model with 8 KV heads, head dimension 128, and a 32,768-token context at FP16 (Kilo Blog). On a 16 GB budget, that leaves little room for weights and runtime overhead, so a shorter context or a smaller model may be necessary. The exact numbers depend on the model architecture and the runtime, so use the formulas as an estimate and verify with your own setup.
Tradeoffs that change the decision
Speed matters as much as capability. No matter how smart an AI model is, there is no point if it takes hours to complete a simple task. For local inference, choose a model that balances performance against inference speed (Kilo Blog).
Context length is a memory decision, not just a quality decision. The KV cache grows with context tokens, layers, KV heads, and head dimension, so a long-context workflow can rule out a model that would otherwise fit (Kilo Blog).
Unified memory is flexible but shared. On an Apple Silicon laptop, the model competes with everything else you have open, so the practical budget is lower than the total memory figure (Local AI Zone).
Quantization is a quality tradeoff. Lower-bit quantization reduces the weight memory estimate, but you should test whether the compressed model still performs well enough for your task (Kilo Blog).
Before deploying a model, verify its license terms for your intended use. Open weights, open source, and commercial-use permission are not interchangeable, and the model card or license is the authoritative source.
What to do next
Pick one model that is small enough to get working before you start optimizing. Write down your job, calculate your memory budget, estimate weights and KV cache, choose a runtime and file format, and test with real prompts. If you are unsure, start with a general chat model and a modest context length, then adjust based on measured speed and quality (Local AI Zone).
If you share your goal, hardware, runtime, and the result or error you see, the Ai-Forum community can help you narrow the next step.
FAQ
How much VRAM do I need for a local model?
Estimate weight memory, KV cache, and runtime overhead, then compare the total with your available VRAM or unified memory. Weights alone are not enough.
Is a larger model always better?
No. A larger model may be more capable, but if it does not fit or runs too slowly for your task, a smaller model can be more useful.
What is quantization?
Quantization describes how compressed a model's weights are. Fewer bits per weight lower the weight memory estimate, but you should verify quality on your task.
Do I need a discrete GPU to run a local model?
Not necessarily. Apple Silicon unified memory can run models, but the model competes with everything else open on the machine. A discrete GPU has its own high-bandwidth VRAM.
How do I choose my first local model?
Write down the job, find your memory budget, choose a runtime, estimate weights and KV cache, then pick one model small enough to get working before optimizing.