AIForum.

How to Run Multiple Local AI Models at Once Without Running Out of Memory

On this page

THE SHORT ANSWER

Running several local models on one GPU is mostly a scheduling decision. If the models fit in VRAM together, load them together; if they do not, make one leave memory before the next arrives. Ollama swaps models automatically on demand, while LocalAI keeps them resident by default and gives you explicit eviction controls such as a cap on active backends.
Quick facts

Running several local AI models on one machine is usually a scheduling problem before it is a hardware problem. If the models fit in VRAM together, load them together; if they do not, arrange for one to leave memory before the next arrives. Ollama does this automatically, loading and unloading models on demand and evicting the previous model when you switch (Aimade Tools). LocalAI takes the opposite default — models stay resident once used — and exposes explicit controls such as a cap on active backends with least-recently-used eviction (LocalAI documentation). Choosing between those behaviours, and understanding what actually consumes memory, is most of the work.

This guide is written for a technically curious beginner or intermediate practitioner running local models on a single GPU, or on a GPU plus system RAM, who wants to serve more than one model without hitting out-of-memory failures.

Start With the Memory Budget, Not the Model List

Model file size is a convenient shorthand and a misleading one. The VRAM a running model occupies is the sum of at least three things: the weights, the key-value (KV) cache that grows with context length and the number of concurrent requests, and the runtime’s own overhead for compute graphs, buffers, and framework state. Quantization shrinks the weights — a 4-bit build is far smaller than its 16-bit equivalent — but it does not shrink the KV cache, which is driven by context length, layer count, and how many sequences you serve at once.

Two practical consequences follow. A model that fits comfortably at a short context may not fit at a long one. And a model that fits alone may not fit beside another model that is already resident. Any plan for running multiple models should therefore start from a measured footprint at the context length and concurrency you actually use, with headroom left over, rather than from the size of the downloaded file.

Five Ways to Share One GPU

The strategies below are the common ones, along with the tradeoffs the Aimade Tools guide associates with each. Treat the latency figures as typical values reported in that comparison, not as guarantees: they depend on the model, quantization, storage speed, and hardware.

StrategyVRAM usageLatencyBest for
Sequential swappingOne model at a time2–5 s swapSingle developer
Concurrent loadingSum of all modelsNoneMulti-user serving
Partial offloadingGPU + CPU splitVariableBudget setups
LoRA adaptersBase + tiny adaptersUnder 100 ms swapFine-tuned variants
Speculative decodingDraft + verify modelNoneSpeed optimization

The guide states the rule plainly: if your models fit in VRAM together, load them together; if they do not, swap them. That single decision eliminates most of the guesswork.

Sequential Swapping: The Default That Usually Works

Ollama handles model lifecycle automatically. When you switch models, the previous one is evicted from VRAM and the new one loads. The documented pattern looks like this:

# Model A loads into VRAM
ollama run qwen3.5:27b "Fix this bug"

# Model A is unloaded, Model B loads
ollama run codestral:22b "Complete this function"

The second prompt is illustrative; the point is the behaviour, not the exact wording. The cited comparison puts a swap at roughly 2–5 seconds, which is negligible for a single developer working interactively and expensive for a service where every request would pay that cost. The guide’s key insight is that most workflows are sequential anyway — you use one model, then switch to another — so simultaneous residency is often unnecessary.

Concurrent Loading: Pay for Everything at Once

When the sum of the models fits, keeping them loaded removes swap latency entirely and is the recommended approach for multi-user serving. A common pairing is a small, fast 7–9B model for tab completions alongside a larger 22–27B model for chat interactions. Other documented combinations include a code-specialised model for implementation plus a general model for architecture discussion, a cheap generator plus a stronger reviewer, and separate models for different programming or natural languages.

The catch is arithmetic: concurrent loading consumes the sum of all resident models, so the fit check must include KV cache growth and runtime overhead, not just weights. If the sum is close to the card’s capacity, a longer context or a second simultaneous request can push it over.

Partial Offloading: Splitting Between GPU and CPU

Partial offloading places part of a model on the GPU and the rest in system RAM. The cited comparison describes it as a budget-setup strategy with variable latency — useful when you want a second model available without a full unload and reload cycle, but slower than keeping everything on the GPU. It is a reasonable middle ground when one model is used heavily and another only occasionally.

LoRA Adapters: Many Behaviours, One Base Model

If your “multiple models” are really fine-tuned variants of the same base, LoRA adapters are the cheapest option in the table. The base model stays loaded and only a small adapter is swapped, which the guide puts at under 100 ms — an order of magnitude faster than a full model swap. This is the pattern to reach for when you want several specialised behaviours without paying for several full sets of weights. It does not help when the models are architecturally different.

Speculative Decoding Is Not a Memory Strategy

Speculative decoding appears in the comparison as a speed optimisation, and it is worth being explicit about why it does not belong in a memory-saving plan. It keeps a draft model and a verification model available at the same time, so it adds to the resident footprint rather than reducing it. Use it when you have spare VRAM and want lower latency; do not use it to make an oversized workload fit.

Server-Side Controls: What LocalAI Adds

Ollama’s swap-on-demand behaviour is convenient for a single user, but it is not the only sensible default. LocalAI keeps models loaded in memory once they are first used, which creates a specific failure mode: if you load a large model that consumes most of your VRAM, subsequent requests for other models may fail, and idle models stay resident with no automatic mechanism to unload them unless you do it manually through the web interface. The documentation references this as a recurring issue (issues #6068, #7269, and #5352).

Three mechanisms address it:

  1. Max active backends (LRU eviction). Limit how many models may be loaded at once. When the limit is reached and a new model is needed, the least recently used model is unloaded automatically. The limit can be set with a CLI flag or an environment variable.
  2. Concurrency groups. Per-model anti-affinity rules that prevent specific models from coexisting on the same node. This trades peak throughput for predictability: two models that never run together can never exhaust VRAM together.
  3. Watchdog mechanisms. Automatic unloading of idle or stuck models based on configurable timeouts.
# Allow up to 3 models loaded simultaneously
./local-ai --max-active-backends=3

# Using environment variables
LOCALAI_MAX_ACTIVE_BACKENDS=3 ./local-ai
MAX_ACTIVE_BACKENDS=3 ./local-ai

Setting the limit to 1 is equivalent to single-model operation with automatic eviction, which makes LocalAI behave more like Ollama’s swap-on-demand model.

A Practical Decision Procedure

  1. Measure each model’s real footprint at the context length and concurrency you use, including KV cache and runtime overhead. Watch GPU memory during a representative request rather than trusting the file size.
  2. Classify the workflow. If requests to different models are naturally sequential, swapping is the simplest correct answer. If several models must answer concurrently, you need concurrent residency.
  3. If sequential: rely on Ollama’s default load/unload behaviour, or set LocalAI’s --max-active-backends=1.
  4. If concurrent: check that the sum of footprints plus headroom fits. If it does not, reduce quantization, reduce context length, cap concurrency, or move one model to partial offloading.
  5. Set a ceiling. Use --max-active-backends to bound the number of resident models, and configure watchdog timeouts so idle models release memory.
  6. Verify the behaviour. Run model A, then model B, then model A again, and confirm which model was evicted and how long each switch took. That sequence is the reproducible check that tells you whether your configuration matches your assumption.

Diagnosing “Not Enough VRAM” Failures

The signature symptom is a request to load a new model failing because there is not enough available VRAM, even though that model fits when loaded alone. Work through the causes in order of cost:

  • Is the previous model still resident? This is the documented LocalAI default and the most common cause. Cap the active backends or unload the idle model before loading the next one.
  • Has the context grown? A model that fit at a short context may not fit at a long one, because the KV cache scales with context length and concurrent sequences.
  • Is another process holding memory? Desktop compositors, browsers, and other inference runtimes compete for the same VRAM pool.
  • Is the working set simply larger than the limit? If you routinely need more resident models than the card can hold, no configuration will fix it; reduce the models, reduce their precision, or move work to a larger machine.

Limitations and Tradeoffs Worth Knowing

Every strategy in the table buys something at a cost. Swapping adds 2–5 seconds per switch, which is invisible for one developer and unacceptable for a latency-sensitive service. Concurrent loading removes that latency but requires the full sum to fit, leaving less headroom for context growth. Partial offloading keeps a model available but runs part of it on slower hardware. LoRA adapters only help when the variants share a base model. LRU eviction and watchdog timeouts both free memory by making a future request pay a reload cost, and aggressive limits can cause thrashing if your working set genuinely exceeds the cap. Concurrency groups prevent out-of-memory failures by preventing coexistence, which is a deliberate reduction in peak capability.

The figures cited here come from a published comparison and from project documentation, not from a controlled benchmark on your hardware. Treat them as planning inputs and confirm the behaviour on your own machine before depending on it.

What to Do Next

Make the smallest change that matches your workflow: if you are a single user switching between models, confirm that your runtime is actually evicting the previous model; if you serve several users, set an explicit active-backend limit and a watchdog timeout rather than leaving residency unbounded. Then measure one swap and one concurrent load, and let those two numbers decide the rest.

If you have hit a specific out-of-memory error, a swap that took far longer than expected, or a configuration that finally made two models coexist reliably, share your environment and the exact error in the community — reproducible details are what make these threads useful to the next person.

FAQ

Why does loading a second model fail when each model fits on its own?

Because the first model is usually still resident. LocalAI keeps models in memory once they have been used, so a large resident model can leave too little VRAM for the next one. Cap the number of active backends, unload the idle model, or configure a watchdog timeout.

Does speculative decoding reduce VRAM use?

No. It keeps a draft model and a verification model available at the same time, so it adds to the resident footprint. It is listed as a speed optimisation, not a memory-saving strategy.

Is swapping models slower than keeping them loaded?

Yes. The cited comparison puts a sequential swap at roughly 2–5 seconds, while concurrent loading has no swap latency but requires the sum of all models to fit in VRAM together.

How many models can I keep loaded at once?

That depends on your measured memory budget, including KV cache and runtime overhead. LocalAI's --max-active-backends sets a hard ceiling and evicts the least recently used model when the limit is reached.

References

  1. https://www.aimadetools.com/blog/run-multiple-models-one-gpu/
  2. https://localai.io/docs/advanced/vram-management/

Related terms

VRAM
Video memory on the GPU. It holds model weights, the KV cache, and runtime buffers, and is typically more limited than system RAM.
KV cache
Memory that stores attention keys and values for processed tokens. It grows with context length and with the number of concurrent sequences, so it is not covered by weight quantization.
Quantization
Storing model weights at lower precision to reduce their size. It shrinks the weights but not the KV cache or runtime overhead.
LRU eviction
Least-recently-used eviction: when a limit on loaded models is reached, the model that has gone unused longest is unloaded to make room.
LoRA adapter
A small set of trained weights applied on top of a shared base model, allowing several fine-tuned behaviours without loading several full models.
Partial offloading
Splitting a model between GPU memory and system RAM, trading speed for the ability to run models that do not fit entirely in VRAM.
Speculative decoding
Using a small draft model to propose tokens that a larger model verifies. It improves speed but keeps both models resident, so it increases memory use.

Leave a Reply

Your email address will not be published. Required fields are marked *