When you run a large language model (LLM) on your own hardware, two types of memory dominate the conversation: VRAM (video memory on your GPU) and system RAM (the main memory for your CPU). The short answer is that VRAM is the performance-critical resource for GPU inference, while system RAM is the capacity resource that keeps the rest of the system running and allows you to load, offload, or run models on the CPU. Confusion often arises because hardware requirements are frequently stated as a single number—“you need 30GB of RAM”—without specifying which memory is meant. This article explains what each type does, how they interact, and how to make practical decisions for local AI workloads.
The Two Memory Systems in a Local AI Setup
Modern computers have at least two distinct memory hierarchies relevant to LLM inference: GPU memory (VRAM) and system memory (RAM). They are not interchangeable. VRAM is physically attached to the graphics card and is designed for high-bandwidth parallel access. System RAM is attached to the motherboard and is optimized for capacity and general-purpose CPU access.
According to Dewan’s Blog, during GPU-based inference, VRAM holds the performance-critical parts of the model: model weights in their inference format (FP16, INT8, INT4, etc.), activation buffers used during forward passes, the KV cache that stores attention state for the active context, and temporary buffers used by fused kernels and matrix multiplications. Modern inference runtimes such as vLLM, llama.cpp, TensorRT-LLM, and PyTorch-based stacks try very hard to keep as much of this as possible in VRAM because GPU memory offers dramatically higher bandwidth and better effective latency for GPU workloads than system RAM. [Source: Dewan’s Blog]
System RAM, by contrast, serves several supporting roles. It holds the operating system, the inference runtime, and the application that drives the model. When you start a local LLM, the model weights are typically loaded from disk into system RAM first, then copied to VRAM if a GPU is used. If you run inference on the CPU, system RAM becomes the primary memory for the model and its KV cache. And if the model does not fit entirely in VRAM, the runtime can offload some layers or tensors to system RAM, fetching them as needed. That works, but it is slower because system RAM has much lower bandwidth than VRAM. [Source: Dewan’s Blog]
Why VRAM Is the Performance-Critical Resource
The rule of thumb is simple: if the full model—including its KV cache and runtime buffers—fits in VRAM, you get fast, predictable throughput. If it does not, you are forced into offloading strategies where parts of the model or attention state live in system RAM. Offloading can make a model runnable on hardware that would otherwise be unable to load it, but it introduces a performance penalty. Data must move back and forth between system RAM and VRAM over the PCIe bus, which is orders of magnitude slower than the internal bandwidth of VRAM. [Source: Dewan’s Blog]
This is why VRAM capacity is often the first specification to check when planning a local AI build. A GPU with more VRAM can hold larger models, longer context windows, or bigger batch sizes without offloading. VRAM bandwidth also matters: higher bandwidth reduces the time spent moving weights and activations through the memory hierarchy, which directly affects tokens per second.
What System RAM Actually Does
System RAM is not just a passive bystander. It is essential for several tasks:
- Model loading: Before inference begins, the model file is read from disk into system RAM. From there, it may be copied to VRAM. If the model is larger than VRAM, it stays in system RAM and is streamed as needed.
- CPU inference: If you run a model entirely on the CPU (for example, on a laptop without a discrete GPU), system RAM holds the weights, KV cache, and activations. CPU inference is generally much slower than GPU inference, but it can work for small models or low-throughput use cases.
- Offloading: When VRAM is insufficient, runtimes like llama.cpp can offload some layers to system RAM. The GPU processes the layers it holds, while the CPU processes the offloaded layers. This hybrid approach allows larger models to run, but at reduced speed.
- Supporting the OS and applications: The operating system, Python environment, and any other software also consume system RAM. If system RAM is exhausted, the system may swap to disk, which is catastrophic for performance.
The Puget Systems hardware primer notes that for CPU inference, the number of memory channels in the platform affects performance, because more channels provide higher memory bandwidth to the CPU. [Source: Puget Systems] This is a reminder that system RAM is not just about capacity; its configuration and bandwidth matter for CPU-bound workloads.
How Model Size, Quantization, and Context Affect Memory
To estimate memory needs, you must consider more than the raw parameter count. The three main factors are:
- Model weights: The size of the weights depends on the number of parameters and the quantization format. In FP16, each parameter takes 2 bytes; in INT8, 1 byte; in INT4, roughly 0.5 bytes. So a 7-billion-parameter model would require about 14 GB in FP16, 7 GB in INT8, or 3.5 GB in INT4 for the weights alone. These are rough estimates; actual file sizes vary by architecture and implementation.
- KV cache: The KV cache stores attention keys and values for the active context. Its size grows with context length and batch size. Longer conversations or larger batches require more KV cache memory. This memory is typically allocated in VRAM for GPU inference, and it can be substantial for long contexts.
- Runtime buffers: Inference runtimes allocate temporary buffers for operations like matrix multiplication and fused kernels. These buffers add overhead on top of weights and KV cache.
Because of these factors, the total VRAM required is always greater than the weight size alone. A model that “fits” in 4 GB based on weight size may still fail to run if the KV cache and runtime buffers push it over the limit. This is a common source of confusion when users compare their GPU’s VRAM to the model’s file size.
A Practical Example: Running a 7B Model
Consider an illustrative scenario: you want to run a 7B parameter model at 4-bit quantization on a desktop with a GPU that has 8 GB of VRAM and 32 GB of system RAM. The model weights might occupy about 3.5–4 GB of VRAM. With a context length of 4096 tokens, the KV cache might add 1–2 GB, and runtime buffers another 0.5–1 GB. The total VRAM usage could land around 5–7 GB, which fits within 8 GB. In this case, inference runs entirely on the GPU, and system RAM is used mainly for loading the model and running the OS.
Now imagine the same model on a GPU with only 4 GB of VRAM. The weights alone might nearly fill VRAM, leaving little room for KV cache and buffers. The runtime would need to offload some layers to system RAM. Inference would still work, but tokens per second would drop significantly because the GPU would frequently wait for data from system RAM. This example is illustrative and not based on a specific benchmark; actual results depend on the model architecture, runtime version, and configuration.
Common Confusion: “30GB of RAM” — Which One?
Dewan’s Blog highlights a common confusion: a Reddit thread about running Qwen3-Next locally mentioned a requirement of “30GB of RAM,” and readers were unsure whether that meant VRAM, system RAM, or both. [Source: Dewan’s Blog] The answer depends on the context. For GPU inference, “RAM” often refers to VRAM. For CPU inference, it means system RAM. For offloading, it can mean a combination. When you see a memory requirement, always ask: is this for the GPU or the CPU? And does it include the KV cache and overhead? Without that clarification, you may buy the wrong hardware or misjudge whether your system can run a model.
How to Check Your Memory Usage
Before making changes, measure what you have. On NVIDIA systems, the command nvidia-smi shows VRAM usage and processes. On Linux, free -h or htop shows system RAM usage. On Windows, Task Manager’s Performance tab displays “Dedicated GPU memory” for VRAM and “Memory” for system RAM. On macOS, Activity Monitor’s Memory tab shows system memory, and for Apple Silicon, memory is unified, meaning the CPU and GPU share the same pool—so the VRAM/system RAM distinction is less clear. Monitoring these metrics while running a model helps you see whether you are VRAM-limited or system RAM-limited.
Choosing Hardware: What to Prioritize
If your goal is fast GPU inference, prioritize VRAM capacity and bandwidth. VRAM is the bottleneck for most local LLM workloads because it determines the largest model and context you can run without offloading. System RAM should be sufficient to hold the model file during loading, run your OS and tools, and provide headroom for offloading if needed. For CPU-only inference, system RAM capacity and memory bandwidth (via multiple channels) become the primary considerations. [Source: Puget Systems]
There is no universal recommendation, because model sizes, quantization levels, and context lengths vary widely. A practical approach is to start with the model you want to run, find its quantized weight size, add an estimate for KV cache and runtime overhead, and compare that to your available VRAM. If it fits, you are in good shape. If not, consider a smaller model, more aggressive quantization, a shorter context, or hardware with more VRAM.
Limitations and Tradeoffs
VRAM is expensive and limited. System RAM is cheaper and often expandable. Offloading to system RAM lets you run larger models on modest GPUs, but it reduces throughput. Unified memory architectures, such as Apple Silicon, blur the line by allowing the GPU to access the same memory pool as the CPU, but they still have finite capacity and bandwidth constraints. Understanding these tradeoffs helps you set realistic expectations and choose the right balance for your workload.
Next Steps
Start by identifying the model you want to run and its quantized size. Check your VRAM and system RAM using the tools mentioned above. Try running the model with a small context length and monitor memory usage. If you hit VRAM limits, reduce context, lower quantization, or enable offloading. If system RAM is the bottleneck, close other applications or add more memory. Share your hardware configuration and results in the Ai-Forum.org community to help others learn from your experience.
FAQ
Can I run a local LLM with only system RAM and no GPU?
Yes, CPU inference uses system RAM to hold the model and KV cache, but it is typically much slower than GPU inference. Small models and low-throughput use cases may still be practical.
What happens if my model doesn't fit in VRAM?
The runtime can offload some layers or tensors to system RAM. Inference still works but slows down because data must move over the slower PCIe bus between system RAM and VRAM.
How much VRAM do I need for a 7B model?
It depends on quantization, context length, and runtime overhead. At 4-bit quantization, weights are roughly 3.5–4 GB, but you must add KV cache and buffers. Check the model card and runtime documentation for specific estimates.