Out-of-memory (OOM) errors when loading a local AI model usually mean one of two things: the model weights do not fit the selected GPU or GPUs, or the runtime cannot allocate the memory it needs for KV cache, activations, communication buffers, and CUDA overhead after the weights are loaded. The fastest way to diagnose the problem is to capture the full backend log, identify whether the failure happens during weight loading or later, and then compare a calculated memory estimate against available VRAM. This guide is for practitioners running local models with runtimes such as NVIDIA NIM, vLLM, LocalAI, or GGUF-based backends.
What an OOM Error Actually Tells You
According to NVIDIA’s NIM troubleshooting guide, OOM errors are common startup failures when deploying LLM NIM containers. They occur when the model’s memory requirements exceed available GPU VRAM, often because of a mismatch between the selected model profile and the hardware, or because of misconfigured system parameters (NVIDIA NIM troubleshooting). During startup, NIM and the vLLM backend allocate GPU memory in stages. The largest single consumer is usually the model weights, but weights are not the whole story. GPU memory is also needed for KV cache, activations, communication buffers, and CUDA (same source).
That distinction matters because an OOM during weight loading points to a different fix than an OOM after the model has loaded. If the failure occurs while weights are being placed on the GPU, reduce weight memory by changing precision, tensor parallelism, or model size. If the failure occurs later, reduce runtime memory pressure by lowering context length, batch size, concurrency, or memory utilization targets.
First Pass: Capture the Real Backend Error
The generic HTTP 500 or container exit message is rarely enough. LocalAI’s runtime error documentation advises reading the server log rather than the HTTP body, because the useful text is almost always in the server log (LocalAI runtime errors). Run with DEBUG=true to see the full backend output (same source). For NIM and vLLM, inspect the container or process logs for the allocation phase and the exact tensor or buffer that failed.
LocalAI exposes several load-time error strings that are easy to confuse. The table below maps the literal log strings to likely causes and fixes, based on LocalAI’s documentation.
| Error string or symptom | Likely cause | Fix |
|---|---|---|
could not load model: ... | The selected backend started but rejected the model: bad path, corrupt or truncated GGUF, wrong architecture, or unsupported quantization. | Read the wrapped backend message. Re-download the model if it is truncated. Confirm the backend matches the model; for example, a GGUF model needs llama-cpp. Run with DEBUG=true to see the full backend output. |
could not load model (no success): ... | The backend replied to the load request but reported failure without a fatal error. | Same checks as above. The trailing message is the backend’s status text; check it for the concrete reason, such as out of memory, unsupported option, or missing file. |
could not load model - all backends returned error: ... | LocalAI tried every candidate backend for the model and each one failed. Usually the backend for this model type is not installed, or the model file is unusable. | Install the correct backend with local-ai backends install or from the Backends page. Confirm the model config’s backend: field names an installed backend. |
If the log contains out of memory or an allocation failure, continue to the memory estimate. If it contains a different error, such as an unsupported quantization or a missing file, fix that first. OOM is a hypothesis until the backend message confirms it.
Estimate Weight Memory Before Changing Settings
NVIDIA’s guide gives a simple heuristic for per-GPU weight memory:
weight_memory_per_gpu = total_parameters x bytes_per_parameter / TPHere TP is tensor parallelism, meaning the number of GPUs across which the model is split. The bytes-per-parameter value depends on precision (NVIDIA NIM troubleshooting):
| Precision | Bytes per parameter |
|---|---|
| BF16 | 2 |
| FP16 | 2 |
| FP8 | 1 |
| INT4 | 0.5 |
| NVFP4 | 0.5 |
For example, NVIDIA’s documentation calculates Llama 3.1 8B at BF16 with TP=1 as 8 billion x 2 bytes = 16 GB. That fits on a single 24 GB GPU such as an A10G or RTX 4090, with room for KV cache and overhead. Llama 3.3 70B at BF16 with TP=4 is 70 billion x 2 bytes / 4 GPUs = 35 GB per GPU, which fits on four A100-40GB or four A100-80GB GPUs with varying room for KV cache. The same 70B model at FP8 with TP=2 is 70 billion x 1 byte / 2 GPUs = 35 GB per GPU, so FP8 halves weight memory and allows two GPUs instead of four (same source).
These are weight estimates, not total VRAM requirements. A model that appears to fit by weight memory alone can still OOM when the runtime allocates KV cache, activations, communication buffers, and CUDA context. Leave headroom. On a 24 GB card, a 16 GB weight footprint leaves 8 GB for everything else; that may be enough for a short context and small batch, but not for long contexts or high concurrency.
Diagnostic Sequence for Loading Failures
Work from low-impact checks to changes that affect model quality or throughput. Change one variable at a time so you can attribute the result.
- Record the environment. Note GPU model and VRAM, number of GPUs, driver and CUDA versions, runtime and version, model name, parameter count, precision or quantization, context length, batch size, and tensor parallelism. Without these, an OOM report is not reproducible.
- Capture the full log. For LocalAI, set
DEBUG=trueand read the server log. For NIM or vLLM, capture the container or process log. Identify the exact allocation that failed. - Calculate weight memory. Use the formula above and the model’s parameter count and precision. Compare the per-GPU estimate with available VRAM. If weights alone exceed VRAM, the fix is precision, tensor parallelism, or a smaller model.
- Check tensor parallelism and GPU visibility. If TP is set higher than the number of visible GPUs, or if the model’s attention heads cannot be divided across the requested TP size, the load can fail. Confirm the runtime sees all intended GPUs.
- Reduce runtime memory pressure. If weights fit but the load still OOMs, lower context length, batch size, and concurrent requests. For vLLM-style runtimes, review the GPU memory utilization setting so the runtime does not reserve more than the device can provide. KV cache grows with context length and batch size.
- Try a lower-precision profile. Moving from BF16 or FP16 to FP8, INT4, or NVFP4 reduces bytes per parameter. NVIDIA’s table lists FP8 at 1 byte per parameter and INT4 or NVFP4 at 0.5 bytes per parameter. This can make a model fit, but it may affect output quality; verify on your own workload.
- Verify the fix under load. A model that loads successfully can still OOM when a real request arrives. Send representative prompts, monitor VRAM, and confirm the original error no longer appears.
Backend-Specific Checks
LocalAI
LocalAI’s documentation emphasizes that the wrapped backend message is the useful part of a load failure. For could not load model: ..., read the backend’s own message and re-download the model if it is truncated. Confirm the backend matches the model: a GGUF model needs llama-cpp. Run with DEBUG=true to see the full backend output. For could not load model - all backends returned error: ..., LocalAI tried every candidate backend and each failed; this usually means the correct backend is not installed or the model file is unusable. Install the correct backend with local-ai backends install or from the Backends page, and confirm the model config’s backend: field names an installed backend (LocalAI runtime errors).
NIM and vLLM
NVIDIA’s guide frames OOM as a mismatch between the selected model profile and the hardware, or a misconfigured system parameter. Check the profile’s precision and tensor parallelism against the actual GPU count and VRAM. Use the weight-memory formula to see whether the profile can fit before adjusting runtime flags. The guide also notes that memory is allocated in stages, so the log’s stage information helps separate weight-loading failures from later allocation failures (NVIDIA NIM troubleshooting).
GGUF and llama.cpp-style runtimes
For GGUF-based runtimes, the same principles apply: confirm the backend matches the model format, then reduce the number of layers offloaded to the GPU, lower context size, or reduce batch size. A corrupt or truncated GGUF can also fail during load and may produce a misleading memory-related message; re-download the file and check the backend’s wrapped error (LocalAI runtime errors).
When the Cause Is Not Memory
Not every load failure is an OOM. LocalAI’s error table lists bad paths, corrupt or truncated GGUF files, wrong architecture, unsupported quantization, missing files, and unsupported options as causes of load failures. If the backend message does not mention memory or allocation, treat OOM as unconfirmed. Fix the reported problem first, then retest. A model file that is valid but too large for the GPU will usually produce a clear allocation error; a model file that is truncated may fail for a different reason.
Limitations and Missing Information
An OOM diagnosis depends on information that is often absent from a user report: full logs, exact model and quantization, runtime version, context length, batch size, tensor parallelism, and GPU memory at the time of failure. Without those details, the cause cannot be determined with confidence. The steps above are a diagnostic framework, not a guarantee that a specific configuration will fit. Memory estimates are approximations; actual usage varies by runtime, driver, and workload.
Next Action
Start by capturing the full backend log with verbose logging enabled. Then calculate per-GPU weight memory and compare it with the GPU’s VRAM, leaving room for KV cache, activations, communication buffers, and CUDA overhead. Adjust one variable at a time: precision, tensor parallelism, context length, batch size, or concurrency. If you are stuck, share your GPU model, runtime and version, model and quantization, context length, batch size, and the exact backend error with the Ai-Forum.org community so others can help reproduce the problem.
FAQ
Does a 16 GB model always fit on a 24 GB GPU?
Not necessarily. NVIDIA's example calculates Llama 3.1 8B at BF16 as about 16 GB of weights and says it fits on a 24 GB GPU with room for KV cache and overhead. Longer context, larger batches, or higher concurrency can still consume the remaining memory.
Why does OOM happen after the model loads?
Weights are not the only GPU allocation. The runtime also needs memory for KV cache, activations, communication buffers, and CUDA. Those allocations can push a model over the limit even when the weights fit.
What does `could not load model - all backends returned error` mean in LocalAI?
LocalAI tried every candidate backend for the model and each one failed. This usually means the correct backend is not installed or the model file is unusable. Install the correct backend and confirm the model config's backend field names an installed backend.
How do I get the real error from LocalAI?
Read the server log rather than the HTTP body, because the useful text is almost always in the server log. Run with DEBUG=true to see the full backend output.
Can FP8 help a model fit?
Yes, FP8 uses 1 byte per parameter versus 2 bytes for BF16 or FP16, so it halves weight memory. NVIDIA's example shows a 70B model at FP8 with TP=2 using 35 GB per GPU, the same per-GPU weight memory as BF16 with TP=4.