How Context Length Affects Local AI Memory Usage

Setting a local model’s context window to 32,768 tokens feels like a free upgrade. It is not. In a controlled sweep of Qwen2.5-Coder-7B-Instruct at Q4_K_M on an RTX 5060 Ti, going from a 2K to a 32K context changed generation speed by essentially nothing — 80.2 tok/s versus 80.3 tok/s — while VRAM rose from 4,559 MiB to 6,269 MiB, roughly +1.7 GB [1]. The bill is paid in memory, and it arrives the moment the server starts, before your first prompt is processed.

This article explains where that memory goes, how to estimate it before launching a model, and which settings actually change the cost. It is aimed at people running models locally with llama.cpp, Ollama, or a similar runtime on a single GPU or on Apple Silicon unified memory. The figures cited here come from two published measurement efforts rather than a test run for this article; where the evidence is thin, that is stated plainly.

Three things compete for memory in a local model

Memory use in local inference is often described as ‘the model size’, but that is only the first of three allocations:

• Weights. The parameters themselves, stored at whatever quantization you chose. This figure is fixed by the model file and does not change when you adjust the context window.
• The KV cache. The attention keys and values for every token in context, stored per layer. This is the allocation that scales with the context setting, and it is the subject of this article.
• Runtime overhead. Compute buffers, the CUDA or Metal context, and the server process itself. This is roughly constant for a given build and backend, but it is not zero, which is why a ‘6 GB model’ can exceed 6 GB of VRAM.

Because the weights and the overhead are effectively fixed, the context window is the one dial a user can turn that moves total memory in a predictable direction. That predictability is useful — and it also means that turning the dial ‘just in case’ has a cost that shows up immediately.

The measured cost of 2K versus 32K

InventiveHQ Lab measured this directly: one model, one quantization, one machine, with only the context-window flag changing between runs. Each configuration was launched as a fresh llama-server process on an RTX 5060 Ti using the llama.cpp CUDA build, and generation speed, VRAM footprint, and a time-to-first-token proxy were recorded [1].

Source: InventiveHQ Lab, Qwen2.5-Coder-7B-Instruct Q4_K_M, RTX 5060 Ti, llama.cpp CUDA build [1].

The contrast is the whole finding. Generation speed is flat across a sixteen-fold increase in the context window. VRAM climbs in a straight line, adding about 1.7 GB between the smallest and largest setting. The authors describe this as a context-length tax paid in memory rather than throughput [1].

The practical consequence: a context setting that fits comfortably at 8K can push a model over the edge of your card’s VRAM at 32K, even though the model file, the quantization, and the sampler settings are identical. If you have been chasing intermittent out-of-memory failures or sudden CPU offloading, the context flag is one of the first places to look.

Why the KV cache grows with the context window

Transformer attention needs the key and value vectors of every token it can still attend to. The runtime caches those vectors so it does not recompute them on every step. The cache therefore has to be sized for the maximum number of tokens the model is allowed to hold — not for the number of tokens you have actually sent.

A rough estimate for the cache, in bytes, is:

2 × layers × KV heads × head dimension × tokens × bytes per element

The leading 2 accounts for keys and values. The terms that matter most to a user are layers, KV heads, and bytes per element. Two models with the same parameter count can have very different per-token cache costs if one uses grouped-query attention (GQA) with few KV heads and the other uses full multi-head attention. That is why a rule of thumb copied from one model rarely transfers cleanly to another.

The other term is tokens, and it is linear. Doubling the context window doubles the KV cache. In llama.cpp the allocation happens at load time for the full configured window [1], which is why the memory increase is visible before any prompt is processed.

As an illustrative example — not a measurement — suppose a model has 32 layers, 8 KV heads, a head dimension of 128, and stores cache values in 16-bit format (2 bytes). The cache for a single token would be 2 × 32 × 8 × 128 × 2, or about 128 KiB. At 2,048 tokens that is roughly 256 MiB; at 32,768 tokens it is about 4 GiB. The measured slope in the CUDA sweep is smaller than this illustration, which is expected: the real per-token cost depends on the model’s layer count, KV head count, and cache data type, and the numbers above are chosen only to show how the arithmetic behaves.

KV-cache quantization changes the slope

The bytes-per-element term is adjustable. A separate profiling project measured peak process RSS for llama.cpp and Ollama on macOS Apple Silicon across context lengths from 1K to 32K, comparing KV-cache types f16, q8_0, and q4_0 [2].

The headline result: at 32K tokens, peak memory was about 5.8 GB with an f16 KV cache and about 3.2 GB with q4_0 — a reduction of roughly 45% at long context [2]. At short contexts the gap is smaller, because the cache is a smaller share of the total. This is the same tradeoff as weight quantization: fewer bits per value, less memory, and some effect on output quality that you should evaluate on your own workload rather than assume.

llama.cpp exposes KV-cache type options (commonly --cache-type-k and --cache-type-v; check your build’s --help output, since flag names have changed across versions). If you need long context on a card that is nearly full, quantizing the cache is often a better first move than pushing layers to CPU.

Note that the two studies are not measuring the same quantity. The CUDA sweep reports VRAM as seen by the driver [1]; the Apple Silicon work reports process RSS as a practical proxy for unified memory [2]. On Apple Silicon there is no separate VRAM pool, so RSS is a reasonable stand-in, but the absolute numbers should not be compared across the two platforms.

Runtime defaults can silently cap your context

A context window you did not actually get is its own class of bug. In the profiling runs, Ollama used a fixed 4096-token context (OLLAMA_CONTEXT_LENGTH=4096), so longer prompts were truncated and memory stayed flat as the requested context grew. The report notes that setting a higher context length when starting the server is what makes Ollama’s memory scale with context [2].

Two failure modes follow. If a long document seems to be partially ignored, the context may be capped below what you assumed. If memory does not rise when you increase the context setting, the setting may not be reaching the runtime. Both are worth checking before concluding that a model ‘handles long context poorly’.

A protocol for measuring your own setup

The published sweeps are useful because they changed exactly one variable. You can reproduce that discipline on your own hardware in a few minutes. The following is a suggested protocol, not a script that was tested for this article.

1. Fix everything except the context window: same model file, same quantization, same backend, same GPU offload setting.
2. Launch a fresh server process for each context size. Do not reuse a running server; the cache is allocated at load time [1].
3. Let the model fully load, then record memory before sending any prompt.
4. Send the same prompt and record generation speed and time to first token.
5. Repeat at two or three context sizes that bracket your intended setting, then compare the readings.

# Suggested protocol, not a tested script for this article.
# One context size per fresh server launch.

llama-server -m ./Qwen2.5-Coder-7B-Instruct-Q4_K_M.gguf -c 8192 -ngl 99

# In a second terminal, sample GPU memory once the model has loaded:
nvidia-smi --query-gpu=memory.used --format=csv,noheader

# Then stop the server, relaunch with -c 32768, and compare the two readings.

On Apple Silicon, substitute a process-level memory reading for nvidia-smi, matching the RSS approach used in the macOS profiling work [2]. Record the numbers somewhere. A two-row table of context size versus memory is enough to make sizing decisions for every future model on that machine.

What the available evidence does not cover

Both sources are single-model, single-machine studies. The CUDA sweep covers one 7B model at one quantization on one GPU [1]; the Apple Silicon work covers a small set of runtimes and KV-cache types on macOS [2]. Neither should be treated as a general formula for your hardware.

Three gaps are worth naming. First, architectures differ: GQA models have much smaller per-token caches than full multi-head models of similar size, so the slope in the table above is model-specific. Second, the TTFT proxy in the CUDA sweep stayed in a narrow band of 85.8 to 92.5 ms [1], but that reflects short prompts; a genuine 32K-token prefill is a different workload and will take substantially longer. Third, neither source measures output quality as a function of context length or KV-cache quantization, so any claim about accuracy at long context needs separate evaluation.

Choosing a context size you can defend

Sizing comes down to matching the window to the workload and leaving room for everything else on the machine:

• Know your actual maximum. If your longest real input is a 6K-token file, a 32K window buys nothing and costs roughly the difference measured above.
• Leave headroom. The operating system, the display, and other applications need memory too. A configuration that barely fits at idle will fail under load.
• Quantize the cache before you sacrifice layers. If you need long context on a nearly full card, q8_0 or q4_0 KV caching reduced peak memory substantially at 32K in the Apple Silicon measurements [2].
• Verify the setting took effect. Confirm that the runtime is using the context length you configured, particularly with server-based tools that carry their own defaults [2].

The next useful action is small: measure your current setup at two context sizes and write down the delta. That single measurement converts an abstract tradeoff into a number you can plan around, and it will tell you more about your machine than any general recommendation.

If you run the sweep, the community is interested in the environment as much as the result — model, quantization, runtime version, context sizes, and the memory readings you observed. Sharing the full configuration is what makes a number reusable by someone else.