AI Hardware Explained: RAM, VRAM, Quantization, and Model Size

Running a large language model locally is mostly a memory-planning problem. The model’s parameter count sets the raw size, quantization sets how many bytes each parameter occupies, and the available VRAM or system RAM determines whether the model can load and how much context it can handle. This guide explains those relationships, shows how to estimate memory with a worked example, and helps you choose a realistic hardware target for local inference.

It is written for technically curious beginners who are moving from hosted AI tools to running open-weight models on their own machine. The focus is local inference, not training. Training has different memory requirements and is outside this guide’s scope. The same memory math also applies when you run a local model behind a WordPress automation workflow, although the model server and the web server are usually separate machines.

The mental model: parameters, precision, and memory

A language model is a large mathematical function with billions of adjustable numbers called parameters. Model size is usually written as a number of parameters in billions: 7B means about seven billion parameters, 70B means about seventy billion. The parameter count is the first input to any hardware estimate because every parameter must be stored somewhere when the model runs.

Precision, also called data type or quantization level, determines how many bytes each parameter uses. At full FP16 precision, each parameter takes 2 bytes. At Q8, each parameter takes about 1 byte. At Q5_K_M, each parameter takes about 0.625 bytes. At Q4_K_M, each parameter takes about 0.5 bytes. These values come from a published VRAM planning formula that uses bytes per parameter as the core multiplier (LLMHardware.io methodology summary).

Quantization is the process of reducing the precision of model weights so the model uses less memory. It does not change the number of parameters. A 70B model is still a 70B model at Q4; it simply stores each parameter in roughly half a byte instead of two bytes. That is why quantization is the main lever for fitting a model onto consumer hardware, and why it also creates a quality tradeoff that you must evaluate for your own tasks.

RAM, VRAM, and storage are different jobs

Local AI hardware discussions often blur three kinds of memory. Storage is the disk space that holds model files, runtimes, and datasets. System RAM is the main memory used by the CPU and operating system. VRAM is the dedicated memory on a graphics card or integrated GPU. A hardware overview from ApXML describes these components separately: the CPU handles general computation, RAM holds active data for the CPU, the GPU accelerates parallel math, and VRAM is the GPU’s dedicated memory (ApXML hardware components).

For LLM inference, VRAM is usually the primary bottleneck. A local AI hardware guide states this directly: VRAM is the primary bottleneck, and GPU selection is the most important decision for most users (LocalAI.computer LLM hardware guide). System RAM still matters. It holds the operating system, the runtime, and any model layers that are offloaded to the CPU. If you run a model entirely on the CPU, system RAM becomes the main constraint. If you run on a GPU, system RAM must still be large enough to load the model and support the runtime.

Storage matters less for inference speed but still matters for practicality. A 70B Q4 model can be tens of gigabytes on disk, and multiple model files add up quickly. Fast storage reduces load time, but it does not replace VRAM or RAM once the model is running.

Why quantization changes the hardware conversation

Without quantization, a 70B model at FP16 would need roughly 140 GB just for weights: 70 billion parameters multiplied by 2 bytes per parameter. That is far beyond a single consumer GPU. At Q4_K_M, the same 70B model needs roughly 35 GB for weights before overhead, because each parameter takes about 0.5 bytes. The LocalAI.computer guide gives a simple rule of thumb: VRAM in gigabytes is approximately parameters in billions multiplied by 0.5 for Q4, by 1.0 for Q8, and by 2.0 for FP16 (LocalAI.computer LLM hardware guide).

A more conservative planning formula adds runtime and operating-system overhead: VRAM (GB) = parameters(B) × bytes_per_param × 1.2 + 2 GB OS overhead. The same source lists bytes per parameter as FP16 = 2.0, Q8 = 1.0, Q5_K_M = 0.625, and Q4_K_M = 0.5 (LLMHardware.io methodology summary). The 1.2 multiplier accounts for runtime overhead, and the 2 GB term covers operating-system overhead. This formula is a planning estimate, not a guarantee. Actual memory use depends on the runtime, the exact quantization file, context length, batch size, and other factors.

The memory budget is more than model weights

A common mistake is to calculate only the weight memory and then buy a GPU that barely fits the result. Inference also needs memory for the key-value cache, often called the KV cache. The KV cache stores attention information for the tokens in the current context so the model does not recompute everything from scratch on each token. Longer context means a larger KV cache. The LocalAI.computer guide estimates that 8K context adds about 1 GB per 10B parameters, 32K context adds about 4 GB, and 128K context can double VRAM needs (LocalAI.computer LLM hardware guide).

Runtime overhead is another part of the budget. The model file may be loaded into memory alongside temporary buffers, CUDA or ROCm context, tokenizer data, and framework overhead. The LLMHardware.io formula handles this with the 1.2 multiplier and the 2 GB OS overhead term. If you are using a GPU, some of that overhead lands in VRAM and some in system RAM. If you are using CPU inference, the weights and KV cache land in system RAM. In all cases, the total memory requirement is larger than the raw weight size.

How much memory do common model sizes need?

The following table combines the model-size mapping from LocalAI.computer with the bytes-per-parameter multipliers from LLMHardware.io. The Q4 column is the most relevant for consumer hardware. The Q8 and FP16 columns are derived from the same multipliers and are included to show how quickly memory requirements grow. Treat every number as an estimate that must be adjusted for context length and runtime overhead.

The Q4 figures for 7-8B, 13-14B, 32-34B, 70-72B, and 405B models come from the LocalAI.computer model-size mapping (LocalAI.computer LLM hardware guide). The Q8 and FP16 figures are calculated with the published bytes-per-parameter multipliers. The same guide notes that Llama 3.1 70B needs 43 GB at Q4 in its dataset and 71 GB at Q8, which illustrates why overhead and context can push estimates above the bare weight mapping. It also states that one RTX 4090 runs models up to the 32B class, and that a 70B model takes two 24 GB cards or a large-memory Mac.

Context length can change these numbers dramatically. A 32B model at Q4 may fit on a 24 GB card with a short context, but a 128K context can double VRAM needs. If you plan to feed long documents, chat histories, or codebases into the model, reserve memory for the KV cache before you buy hardware. The LocalAI.computer guide’s context estimates are a useful starting point: 8K context adds about 1 GB per 10B parameters, 32K adds about 4 GB, and 128K can double VRAM needs (LocalAI.computer LLM hardware guide).

Choosing an approach: GPU, CPU, Apple Silicon, or cloud

There is no universal best hardware choice. The right approach depends on the model size you need, the context length you use, your budget, your operating system, and whether your data can leave your machine. The following framework organizes the main options.

NVIDIA GPU with CUDA

NVIDIA dominates local LLM inference because of the CUDA ecosystem. Most runtimes, quantization formats, and optimization guides target CUDA first. The LocalAI.computer guide states that NVIDIA dominates for LLMs due to CUDA, while AMD is catching up with ROCm (LocalAI.computer LLM hardware guide). If you want the widest compatibility and the most community troubleshooting, an NVIDIA GPU is the safest starting point.

VRAM capacity matters more than raw compute for fitting larger models. A fast 12 GB GPU can be less useful than a slower 24 GB GPU if your target model does not fit in 12 GB. The LocalAI.computer guide describes the RTX 4070 Ti Super 16GB as the sweet spot for most users, with 32B models requiring a 24 GB tier because Qwen 2.5 32B needs about 20 GB at Q4 (LocalAI.computer LLM hardware guide). That is a useful decision boundary: 16 GB is comfortable for 13-14B models at Q4 and can handle some 32B models only with aggressive quantization or CPU offload, while 24 GB opens the 32B class at Q4.

AMD GPU with ROCm

AMD GPUs can offer more VRAM per dollar in some tiers. The LocalAI.computer guide lists the RX 7900 XTX 24GB as the best value 24 GB option in its high-end tier (LocalAI.computer LLM hardware guide). The tradeoff is software maturity. ROCm support varies by operating system, runtime, and model format. If you choose AMD, verify that your intended runtime supports your exact GPU and operating system before buying. Community reports are valuable here because support changes over time.

Apple Silicon and unified memory

Apple Silicon machines use unified memory, where the CPU and GPU share the same memory pool. This allows a Mac with a large memory configuration to load models that would not fit in a typical consumer GPU’s VRAM. The LocalAI.computer guide notes that a 70B model can run on a large-memory Mac when it does not fit on a single 24 GB card (LocalAI.computer LLM hardware guide). The tradeoff is that unified memory is shared with the operating system and applications, and memory bandwidth and GPU compute vary across Apple Silicon generations. Unified memory is not the same as dedicated VRAM, but it can be a practical path for larger models when you do not need maximum tokens per second.

CPU-only inference

CPU-only inference uses system RAM instead of VRAM. It is the most accessible path because it does not require a discrete GPU, but it is usually much slower for large models. CPU inference can be useful for small models, batch jobs, or testing. If you go this route, the planning formula still applies, but the memory target is system RAM rather than VRAM. The LocalAI.computer guide recommends 32 GB of system RAM as a minimum and 64 GB for 70B models (LocalAI.computer LLM hardware guide). Those recommendations are for a general local AI setup; CPU-only inference of larger models may need more.

Cloud or API fallback

If your target model does not fit your hardware, a hosted API can be the pragmatic choice. The tradeoff is data flow: local inference keeps prompts and outputs on your machine, while a cloud API sends them to a provider. For WordPress automation, a common pattern is to run the model on a separate GPU machine or local workstation and expose a small API to the WordPress site. The WordPress host rarely has a GPU, so the model server and the web server should be planned separately. This guide focuses on the model server’s memory requirements.

A worked example: will a 27B Q4 model fit?

Suppose you want to run a 27B parameter model at Q4_K_M. The LLMHardware.io methodology summary provides a worked check for Qwen3.8-27B at Q4_K_M: 27 × 0.5 × 1.2 + 2 = 18.2 GB. On the same date, Unsloth UD-Q4 files were 15.4-17.6 GB, and the Ollama tag qwen3.8:27b listed 18 GB (LLMHardware.io methodology summary, checked 2026-10-03).

That estimate tells you several things. First, a 16 GB GPU is not a safe fit for this model at Q4_K_M with normal context. The weights alone may be close to 16 GB, and the KV cache and runtime overhead will push past the limit. Second, a 24 GB GPU has room for the model and a reasonable context, though a very long context could still exhaust VRAM. Third, if you run the model on the CPU, you need at least 18.2 GB of system RAM for the model estimate, plus memory for the operating system and other applications. A 32 GB system RAM machine is a more realistic CPU-only target for this model.

To make the example concrete, here is a planning sequence you can reuse:

1. Identify the model size in billions of parameters. For this example, 27B.
2. Identify the quantization level. For this example, Q4_K_M, which uses about 0.5 bytes per parameter.
3. Multiply parameters by bytes per parameter: 27 × 0.5 = 13.5 GB of raw weight memory.
4. Apply the planning overhead: 13.5 × 1.2 + 2 = 18.2 GB.
5. Add context memory. If you need 32K context, the LocalAI.computer guide suggests adding roughly 4 GB for models in this range, so plan for about 22 GB or more.
6. Compare the result to your VRAM or system RAM. A 24 GB GPU is a better target than a 16 GB GPU for this model with 32K context.

This is a hypothetical planning example based on the published formula. It is not a claim that every 27B Q4 model uses exactly 18.2 GB in every runtime. The actual number will vary with the quantization file, the runtime, the context length, and the batch size. The value of the exercise is that it gives you a defensible starting point before you spend money or spend hours debugging an out-of-memory error.

Practical path: estimate, select, run, verify

The following path turns the concepts into an actionable workflow. It assumes you have a model in mind and a machine you are considering. You can follow it before buying hardware or before loading a new model on existing hardware.

Step 1: Define the workload

Write down the model family and parameter count, the quantization level you intend to use, the maximum context length you need, and whether the workload is interactive or batch. Interactive chat needs low latency; batch summarization can tolerate slower generation. A long-context document analysis workload needs more KV cache memory than a short chat. This step prevents you from optimizing for the wrong constraint.

Step 2: Calculate weight memory

Use the bytes-per-parameter multipliers: FP16 = 2.0, Q8 = 1.0, Q5_K_M = 0.625, Q4_K_M = 0.5 (LLMHardware.io methodology summary). Multiply the parameter count in billions by the multiplier. For a 13B model at Q4_K_M, that is 13 × 0.5 = 6.5 GB. For a 70B model at Q4_K_M, that is 70 × 0.5 = 35 GB. These are raw weight estimates before overhead.

Step 3: Add runtime and OS overhead

Apply the conservative planning formula: VRAM (GB) = parameters(B) × bytes_per_param × 1.2 + 2 GB OS overhead (LLMHardware.io methodology summary). For the 13B Q4 example, that is 13 × 0.5 × 1.2 + 2 = 9.8 GB. For the 70B Q4 example, that is 70 × 0.5 × 1.2 + 2 = 44 GB. The 70B result is close to the 43 GB figure reported by LocalAI.computer for Llama 3.1 70B at Q4 (LocalAI.computer LLM hardware guide).

Step 4: Add context and KV cache

Estimate context memory using the LocalAI.computer guide’s rules: 8K context adds about 1 GB per 10B parameters, 32K context adds about 4 GB, and 128K context can double VRAM needs (LocalAI.computer LLM hardware guide). These are rough rules, not precise formulas. For a 70B model at 32K context, the added memory can be substantial. If your workload needs very long context, consider a smaller model, a lower quantization, or a GPU with more VRAM.

Step 5: Compare against available memory

If you are using a GPU, compare the total estimate to VRAM. If the model does not fit, you have four main options: use a smaller model, use a more aggressive quantization, reduce context length, or offload some layers to the CPU. CPU offload uses system RAM and slows generation, but it can make a model runnable when it would otherwise fail to load. If you are using CPU-only inference, compare the total estimate to system RAM and leave headroom for the operating system and applications.

Step 6: Choose a runtime and load the model

Common local runtimes include llama.cpp, Ollama, LM Studio, and others. The exact commands differ, but the memory planning is the same. Start with a short context and a small prompt to confirm the model loads. Then increase context and prompt size gradually while monitoring memory. If the model loads but generation is extremely slow, check whether layers are being offloaded to the CPU and whether VRAM is fully utilized.

Step 7: Verify with observable checks

Success is not just that the model loads. Check that it produces coherent output for your task, that generation speed is acceptable, and that memory usage stays below the limit during long interactions. Useful checks include:

• The model loads without an out-of-memory error.
• The first response arrives in a reasonable time for your use case.
• Generation speed remains stable as the conversation grows.
• VRAM or RAM usage stays below the hardware limit with headroom.
• Output quality meets your task requirements at the chosen quantization level.

If any check fails, return to the estimate and adjust one variable at a time. Changing the model, quantization, context length, and offload settings all at once makes it hard to learn what caused the improvement or regression.

Evaluation: quality, speed, and responsible operation

Hardware planning is not complete until you evaluate the result. A model that fits but produces poor output for your task is not a successful deployment. A model that produces good output but takes minutes per response may also be unusable. Evaluate along three axes: quality, speed, and resource use.

Quality evaluation

Quantization reduces memory use by reducing precision, and it can affect output quality. The effect varies by model, quantization method, and task. A common approach is to compare Q4_K_M against Q5_K_M or Q8 on a small set of representative prompts. Use the same prompts, the same system instructions, and the same decoding settings. Score the outputs for correctness, coherence, and task completion. If Q4 quality is acceptable, keep the memory savings. If not, move up to a higher-precision quantization and check whether your hardware can still fit the model.

Do not assume that a higher quantization is always better for every task. Some tasks are robust to Q4, while others, such as precise code generation or structured data extraction, may benefit from Q5 or Q8. The right choice is the lowest memory configuration that meets your quality bar.

Speed evaluation

Measure time to first token and tokens per second. Use the same prompt, context length, model, quantization, and runtime for each comparison. Record the hardware, operating system, runtime version, and model file. Without a controlled setup, speed comparisons are not meaningful. This guide does not provide benchmark numbers because performance depends on the exact environment. Instead, use a simple repeatable test: run the same prompt three times, record the median time to first token and median tokens per second, and note whether any layers are offloaded to the CPU.

Resource and privacy evaluation

Monitor VRAM and system RAM during generation. If memory usage approaches the limit, reduce context length or move to a smaller model before the system starts swapping. Swapping to disk can make generation dramatically slower and may cause instability. On the privacy side, local inference keeps prompts and outputs on your machine, which is a major advantage for sensitive data. Cloud APIs send data to a provider, so review the provider’s data handling terms if privacy matters. For WordPress automation, keep credentials and API keys out of the model prompt and store them in the automation layer, not in the model server.

Licensing and commercial use

Open weights do not automatically mean open source or permission for commercial use. A model may be released under a license that restricts commercial use, requires attribution, or imposes other conditions. Check the model card and license before deploying a model in a business workflow. This is separate from hardware planning, but it is part of responsible operation.

Troubleshooting common local AI hardware problems

The following problems are common when running local models. Each entry gives a symptom, a likely cause, an action, and a way to verify improvement. These are general diagnostics, not confirmed fixes for every runtime.

Symptom: the model fails to load with an out-of-memory error

Likely cause: The total memory requirement, including weights, KV cache, and runtime overhead, exceeds available VRAM or RAM. A model that fits by raw weight size may still fail once context and overhead are included.

Action: Calculate the estimate with the planning formula. If the model is close to the limit, try a more aggressive quantization, reduce context length, use a smaller model, or enable CPU offload. If you are on a GPU, check whether other applications are using VRAM.

Verification: The model loads and completes a short prompt without an error. Monitor memory during the first few responses to confirm headroom.

Symptom: generation is very slow

Likely cause: Some or all layers are running on the CPU, the GPU is not being used, or the context is very long. CPU offload can make a model runnable but slow.

Action: Check the runtime logs for GPU offload settings. Increase GPU layers if VRAM allows. Reduce context length. Try a smaller model or a lower quantization that fits entirely in VRAM.

Verification: Measure tokens per second before and after the change. Confirm that GPU utilization increases and CPU offload decreases.

Symptom: output quality drops after quantization

Likely cause: The quantization level is too aggressive for the task. Q4 is a common balance, but some tasks need Q5_K_M or Q8.

Action: Compare outputs from the same prompts at Q4_K_M, Q5_K_M, and Q8. Choose the lowest precision that meets your quality bar. If higher precision does not fit, consider a smaller model at higher precision.

Verification: Use a small evaluation set and score correctness and coherence. Confirm that the higher-precision model fits in memory with your required context.

Symptom: system RAM is exhausted during CPU inference

Likely cause: The model weights and KV cache are larger than available system RAM, or other applications are consuming memory.

Action: Close unnecessary applications. Use a smaller model or lower quantization. Increase system RAM if possible. The LocalAI.computer guide recommends 32 GB as a minimum and 64 GB for 70B models in a general local AI setup (LocalAI.computer LLM hardware guide).

Verification: The model loads and runs without swapping. Monitor system RAM during generation to confirm headroom.

Symptom: the model works for short prompts but fails on long context

Likely cause: The KV cache grows with context length and exceeds available memory. The LocalAI.computer guide notes that longer context requires more VRAM and that 128K context can double VRAM needs (LocalAI.computer LLM hardware guide).

Action: Reduce the maximum context length. Use a smaller model or lower quantization. If long context is essential, choose hardware with more VRAM or use a runtime that supports efficient KV cache management.

Verification: Run a prompt near the new context limit and confirm that memory stays within bounds and the model completes the response.

Limits of memory estimates and when to ask for help

Memory estimates are planning tools, not guarantees. The LLMHardware.io formula includes a 1.2 multiplier and 2 GB OS overhead, but actual usage depends on the runtime, the quantization file, the context length, the batch size, and the operating system. The LocalAI.computer mapping gives ranges rather than exact numbers for the same reason. Two Q4_K_M files for the same model can differ in size because of how the quantization was produced. A model that fits in 24 GB on one runtime may fail on another.

When you are unsure, verify with a small test before committing to a hardware purchase. Load the model with a short context, measure memory, and then increase context gradually. If you are comparing GPUs, look for community reports from users running the same model and quantization on the same GPU. Be specific about the model, quantization, context length, runtime, and operating system when you ask for help. A question like ‘Will this GPU run this model?’ is hard to answer; a question like ‘Has anyone run Qwen3.8-27B at Q4_K_M with 32K context on a 24 GB card in Ollama?’ is much more useful.

Next steps and learning path

Use the following progression to move from beginner to advanced local AI hardware planning.

• Beginner: Start with a 7-8B model at Q4. The LocalAI.computer guide maps 7-8B Q4 models to 4-5 GB, so an 8 GB or 12 GB GPU can work. If you are CPU-only, plan for 16 GB or more system RAM.
• Intermediate: Move to 13-14B at Q4, which maps to 7-8 GB, or 32B at Q4, which needs a 24 GB GPU because Qwen 2.5 32B needs about 20 GB at Q4 (LocalAI.computer LLM hardware guide).
• Advanced: For 70B at Q4, plan for roughly 43 GB with overhead or 36-37 GB by bare mapping. This usually means two 24 GB cards or a large-memory Mac. For 405B at Q4, the mapping is 203 GB and multiple GPUs.

Your immediate next action should be to calculate the memory requirement for the exact model and quantization you want to run. Use the VRAM calculator at LLMHardware.io to apply the formula, then compare the result to your hardware. If the model does not fit, decide which variable you are willing to change: model size, quantization, context length, or hardware.

If you are building a WordPress automation workflow, keep the model server separate from the web server. Plan the model server’s VRAM and RAM with the same method described here, and expose only the API endpoint that WordPress needs. This keeps the web host simple and the model hardware focused on inference.

Finally, treat hardware planning as an iterative process. Start with a small model, measure real memory use and speed, and scale up only when you know your workload’s actual requirements. The community is most helpful when you share your goal, environment, model, quantization, context length, and observed results. That level of detail turns a vague hardware question into a reproducible experiment.