Choose the GPU with the most dedicated VRAM you can afford, then confirm that your runtime, driver, and operating system actually support it. For local LLMs, VRAM capacity decides which models you can load at all; compute speed only decides how fast they answer once loaded. A slower card with more memory runs the same model as a faster card with less memory — the faster card simply cannot run the larger model at all.
This guide is for anyone buying or upgrading a GPU to run text models locally. It covers the arithmetic that turns a model name into a memory requirement, the tier of hardware each budget realistically buys, and the checks that prevent a purchase that fits on paper but fails in practice.
VRAM capacity sets your ceiling; speed sets your pace
Hardware guides for local inference converge on one rule: VRAM capacity is the single most important specification for running LLMs locally. It determines which models you can run, at what quantization level, and with how much context. GPU cores, clock speed, Tensor cores, and RT cores are secondary to that decision, according to CanItRun’s GPU selection guide.
The practical consequence is counterintuitive for anyone used to shopping on benchmark scores. A $200 used RTX 3060 with 12 GB of VRAM can run models that a $2000 RTX 4080 with 8 GB cannot, because the model literally does not fit in 8 GB. The cheaper card is slower, but it produces output; the faster card produces an out-of-memory error.
The weight arithmetic: parameter count and quantization
Model size in memory is not the same as parameter count. Quantization compresses each weight to fewer bits, and the compression ratio is what makes a model fit on consumer hardware. At Q4_K_M, a widely used 4-bit quantization format, each billion parameters needs roughly 0.6 GB of VRAM for weights alone.
That single ratio scales predictably across common model sizes:
| Model size | Approximate weight memory at Q4_K_M |
|---|---|
| 7B | 3.9 GB |
| 14B | 7.9 GB |
| 32B | 18.0 GB |
| 70B | 39.4 GB |
Those figures are weights only. They exclude the KV cache, format overhead, and runtime allocations, all of which come out of the same pool of VRAM.
Quantization level is the lever you adjust when a model almost fits. Moving a 14B model from Q4_K_M to a more aggressive 3-bit format such as Q3_K_M reduces the weight footprint enough to change which tier of card is sufficient. The tradeoff is fidelity: lower-bit quantization discards more precision, and the quality cost varies by model and task. There is no universal rule for how much quality a given step down costs — it has to be evaluated per model.
KV cache and context: the memory you forget to budget
Weights are a fixed cost. The KV cache is not. It grows with context length, and it is the reason two users with the same model file and the same GPU can have very different experiences.
Budget an additional 0.5–2 GB for the KV cache depending on context length. A short chat with a few paragraphs sits at the low end. An agent carrying repository files and tool output sits at the high end, and can exceed it. The distinction matters because a chat workload and an agent workload are genuinely different hardware requirements even when they use the same model.
This is also why tier tables should be read as starting points rather than guarantees. A 12 GB card does not “run 14B models” in the abstract; it runs a specific 14B file at a specific quantization with a specific context window.
What else on the spec sheet is worth checking
Once capacity is settled, memory bandwidth is the next most relevant attribute, because it governs how quickly weights can be streamed during generation. At the budget end, the Arc B580 12GB is noted for adding excellent bandwidth for its price, which is why it appears alongside the RTX 3060 12GB in the entry tier despite being a different architecture.
Advertised AI throughput figures and gaming position are not substitutes for capacity and software checks. A card can be fast in a benchmark and still be unusable if your inference runtime does not support it well.
What each budget tier realistically unlocks
The following tiers reflect current guidance for consumer hardware, from used cards up through dual-GPU and Apple Silicon setups above $2500:
| Budget | GPU | VRAM | What it unlocks |
|---|---|---|---|
| Under $200 (used) | RTX 3060 12GB | 12 GB | 7–8B comfortably at Q4_K_M; 14B at Q3_K_M with short context |
| $200–400 | RTX 3060 12GB (new) or Arc B580 12GB | 12 GB | Same as above; Arc B580 adds excellent bandwidth for the price |
| $400–800 | RTX 4060 Ti 16GB | 16 GB | 14B models at Q4_K_M with comfortable context |
| $800–1500 | RTX 3090 24GB (used) | 24 GB | 32B model class at Q4 |
A second source frames the same landscape as investigation tiers rather than promises: 6 GB suits a small 3B–4B quantized model; 8 GB suits selected 7B–8B 4-bit models with modest context; 12–16 GB suits selected 14B-class compressed models depending on the exact file and context; 24–32 GB suits selected 32B-class compressed models; and more than 32 GB opens larger models or longer context based on actual requirements rather than the tier alone.
Note the wording: selected models, depending on the exact file and context. Capacity tiers describe a range of possibilities, not a compatibility guarantee.
Software support decides whether that capacity is usable
Dedicated VRAM determines whether the required weights and conversation memory can stay on the GPU. Application, driver, and operating-system support determine whether that capacity is usable at all. Both checks are necessary, and a card that passes the first can still fail the second.
Before committing to a card, confirm that the inference application you intend to use runs on your operating system, that current drivers exist for your platform, and that other users report working setups with that combination. Community reports are useful here precisely because support gaps tend to surface as specific, reproducible failures rather than as documented limitations.
An illustrative memory budget
The following scenario is constructed to show how the numbers interact. It is not a measured result.
Suppose you own a 12 GB card and want to run a 14B model. At Q4_K_M, the weights alone are about 7.9 GB. That leaves roughly 4 GB for the KV cache, format overhead, and everything else the runtime allocates. With a short context and a single request, that may fit. With a long context, several concurrent requests, or an agent pipeline, it likely will not — which is consistent with the published guidance placing 14B at Q4_K_M in the 16 GB tier for comfortable context, and reserving 14B on a 12 GB card for a more aggressive quantization with a short context.
The lesson is that the gap between “fits” and “fits comfortably” is often only a few gigabytes, and it is exactly the gap that determines whether a workflow is usable day to day.
How to verify before you buy
- Pick one exact model download. Not a model family — a specific file at a specific quantization level.
- Pick one application and one context length. Decide whether you need a single conversation or several requests at once, and whether document search or an agent is part of the job.
- Run the numbers with a RAM and VRAM calculator using those exact choices.
- Compare requirements against dedicated VRAM and system RAM separately. Do not combine them into one interchangeable capacity.
- Check software support for the card, driver, operating system, and runtime combination you plan to use.
- Test a small model on hardware you already own first. For occasional short prompts, CPU-only use may be sufficient and avoids a purchase you do not need.
Tradeoffs and uncertainty
Two caveats affect every estimate above. First, architectures differ: some model families need different memory at the same parameter count, so a parameter-count table is a planning aid rather than a precise predictor. Second, quantization quality loss is model-dependent and task-dependent, so the cheapest way to fit a model is not always the best one.
Prices and availability in the used market move, and the tier table reflects guidance at the time it was published rather than a permanent price list. Treat the tiers as a way to reason about capacity, and verify current prices and support status yourself.
What to do next
Write down your intended model file, application, and context length, then work out the memory requirement before looking at any card. If you are already running local models, share your GPU, model, quantization, and context length in the Ai-Forum community so others can compare real configurations against these estimates — and report the exact error message if a model fails to load.
FAQ
Is VRAM more important than GPU speed for running local LLMs?
Yes, by capacity. VRAM determines which models you can run, at what quantization, and with how much context; cores, clock speed, and Tensor cores affect speed once the model is loaded. A slower card with more VRAM runs models a faster card with less VRAM cannot run at all.
How much VRAM does a 7B model need at Q4_K_M?
About 3.9 GB for the weights alone. Add roughly 0.5–2 GB for the KV cache depending on context length, plus runtime overhead, before comparing against your card's dedicated VRAM.
Can I use system RAM instead of VRAM?
System RAM and dedicated VRAM should be compared separately rather than combined into one interchangeable capacity. Offloading layers to system RAM may let a model load, but the memory tiers used for planning are based on dedicated VRAM.
Can I run a 14B model on a 12 GB card?
Sometimes. At Q4_K_M the weights are about 7.9 GB, leaving roughly 4 GB for context and overhead, which suits short-context use. Published guidance places 14B at Q4_K_M with comfortable context in the 16 GB tier, and reserves 12 GB cards for 14B at a more aggressive quantization with short context.