AIForum.

How to Run Local LLMs on a Mac: Unified Memory, Metal, and MLX

On this page

THE SHORT ANSWER

Apple silicon makes local LLM inference practical because the CPU, GPU, and Neural Engine share one memory pool, so weights never have to be copied to a separate VRAM. Metal provides the compute path and MLX is the runtime built around it. This guide explains the stack, how to size a model to your machine, and how to run a first model.
Quick facts

Macs are a practical place to run local language models, and the reason is architectural rather than incidental. On Apple silicon, the CPU, GPU, and Neural Engine draw from one physical memory pool, so weights loaded once can be read directly by the GPU instead of being copied across a bus. Metal supplies the GPU compute path, and MLX — Apple’s open-source array framework — is the runtime built most directly around that combination.

This guide is aimed at technically curious beginners and intermediate practitioners. It explains why the Mac approach works, how to size a model against your machine, how the main runtimes differ, and how to get a first model generating tokens. It also covers the multi-Mac scaling path Apple demonstrated in October 2026, and the failure modes people hit most often.

Why unified memory changes the local-inference math

On a conventional desktop PC, the CPU has its own RAM and the GPU has its own video memory. A model’s weights must live in whichever pool is doing the work, and switching between them means copying data across a bus. On an M-series Mac — M1 through M5 — that split does not exist: the CPU, GPU, and Neural Engine share a single physical memory pool, so there is no copy step and the GPU can read model weights that already sit in the same memory the CPU loaded. Because LLM inference is largely about reading billions of weights from memory as fast as possible, removing that transfer is a structural advantage rather than a small optimization (AI/TLDR).

Two clarifications matter. First, unified memory is a bandwidth and capacity advantage, not a claim that a Mac out-computes a high-end discrete GPU on raw arithmetic. Second, “unified” does not mean unlimited: the pool is shared with the operating system and every other application, and macOS applies a wired-memory limit to how much the GPU may pin. A 64 GB Mac does not offer 64 GB of usable model space.

The three layers: memory, Metal, and MLX

It helps to separate the stack into three layers, because problems usually belong to one of them.

  • Unified memory determines what fits and how fast weights can be streamed. This is the hardware layer, fixed at purchase.
  • Metal is Apple’s low-level GPU API. Every serious Mac inference stack ultimately dispatches compute through it — MLX’s own kernels, llama.cpp’s Metal backend, and PyTorch’s MPS backend all end up there. If a runtime is not using Metal, it is running on the CPU and will be dramatically slower.
  • MLX is Apple’s open-source array framework for Apple silicon, designed around unified memory and accessed by most users through mlx-lm. Apple’s October 15, 2026 research post describes a new MLX release and demonstrates running a frontier-class model with nearly a trillion parameters on Mac Studio hardware (Apple Machine Learning Research).

Sizing a model to your Mac

The most common mistake is comparing a model’s parameter count to total system RAM and assuming it will fit. Weight size is not total memory use. Three things consume memory at inference time:

  1. Weights. A rough rule of thumb for 4-bit quantization is about 0.5 GB per billion parameters, so a 70B model needs roughly 35–40 GB before anything else.
  2. KV cache. This grows with context length and batch size. Its size scales with the number of layers, the number of key/value heads, head dimension, context length, batch size, and bytes per element. Long contexts can add many gigabytes on top of the weights.
  3. Runtime overhead. Activations, scratch buffers, and framework allocations.

The table below is an estimate using the 4-bit rule of thumb plus headroom. Treat it as a planning starting point, not a measurement.

Model size (4-bit)Approximate weightsPractical unified memory to target
~8B~4–5 GB16 GB
~16B~8–9 GB24 GB
~32B~17–18 GB32–48 GB
~70B~35–40 GB64 GB or more
~1T classhundreds of GBMultiple machines, or a very high-memory Mac Studio

Apple states that a single Mac Studio can be configured with up to 512 GB of unified memory. The near-trillion-parameter demonstration used four Mac Studios rather than one, which is the practical signal: very large models on Macs are a cluster story, not a laptop story.

Choosing a runtime

Four options cover most Mac users. They are not interchangeable, and none is universally best.

  • mlx-lm — a Python package layered on MLX. It handles Hugging Face MLX-format models, offers generation, chat, and a local HTTP server, and is usually the shortest path to Apple-silicon-native quantized models.
  • llama.cpp — a C/C++ inference engine with a Metal backend and GGUF quantizations. It has the widest model coverage and good control over how work is split between CPU and GPU.
  • Ollama — a packaging layer that trades some control for convenience: a model library, a simple CLI, and a REST API.
  • LM Studio — a graphical application with a model browser and a built-in local server, useful if you would rather not work in a terminal.

A reasonable decision rule: start with MLX if you want the most Apple-native path and are comfortable with Python; choose llama.cpp if you need a specific GGUF quantization or model that MLX does not cover; choose Ollama or LM Studio if convenience matters more than tuning.

A first run with MLX

The following is a suggested setup based on documented usage, not an editor-tested benchmark run. Flag names and defaults change between releases, so check --help for your installed version.

Prerequisites: an Apple silicon Mac, a reasonably current macOS, Python 3.10 or newer, and enough free disk space for the model weights.

Step 1 — install into a virtual environment.

python3 -m venv ~/.venvs/mlx
source ~/.venvs/mlx/bin/activate
pip install --upgrade pip
pip install mlx-lm

Step 2 — confirm the GPU is the default device.

python -c "import mlx.core as mx; print(mx.default_device())"

On Apple silicon this should report a GPU device. If it reports the CPU, the runtime is not reaching Metal and generation will be slow.

Step 3 — pick a quantized model from the MLX community on Hugging Face, sized against the table above. Prefer a 4-bit quantization for your first run.

Step 4 — generate text.

mlx_lm.generate --model mlx-community/<model-id> --prompt "Explain unified memory in two sentences." --max-tokens 200

The first tokens will take noticeably longer than later ones. That is expected: the prompt is processed in a single parallel pass (prefill), then tokens are decoded one at a time.

Step 5 — serve it locally.

mlx_lm.server --model mlx-community/<model-id> --port 8080

This exposes an HTTP endpoint in the OpenAI chat-completions format, which means existing client code and tools can point at http://localhost:8080 instead of a cloud provider. Verify with a single request before wiring it into anything else.

Scaling past one Mac

Apple’s October 15, 2026 post describes running Muse Spark Open, a frontier-class model with nearly a trillion parameters, on a cluster of four Mac Studios using the RDMA over Thunderbolt driver, reporting over 36 tokens per second at a context size of 8,192 tokens on M5 Ultra hardware. The post reports performance for three parallelization schemes, including pipeline parallelization (Apple Machine Learning Research).

These are vendor-reported figures measured on a specific configuration at a specific context length. Decode speed generally falls as context grows, and it depends on quantization, parallelism strategy, and interconnect. Treat the number as evidence that the approach scales, not as a prediction for your setup.

Troubleshooting the common failures

The process is killed or reports an out-of-memory error

Reduce the quantization level, shorten the context window, close other GPU-heavy applications, and check how much memory is already in use. Very long prompts and large batch sizes inflate the KV cache more than most people expect.

Generation is far slower than expected

Confirm the model is running on the GPU rather than the CPU. Then check whether the slowdown is in prefill or decode: a long prompt makes the first token slow, while a large model on limited memory bandwidth makes every subsequent token slow. Those are different problems with different fixes.

Output is incoherent or repeats itself

This is usually a prompt-format problem, not a model problem. Instruction-tuned models expect a specific chat template; feeding raw text to a model trained on a chat format produces degraded output. Use the template the model card documents, and check whether your runtime applies it automatically.

Performance degrades after several minutes

On laptops, sustained inference can trigger thermal throttling. Monitor the machine’s behavior over a long run rather than judging from a short burst.

Limitations worth accepting up front

  • The memory is shared. Everything the GPU pins is unavailable to the rest of the system. Running a large model while doing heavy work in other applications is a real constraint, not a theoretical one.
  • No CUDA ecosystem. Some kernels, libraries, and fine-tuning toolchains simply do not exist for Metal, or lag behind their CUDA equivalents.
  • Quantization costs quality. 4-bit weights are a tradeoff between memory and fidelity. Higher precision means more memory and slower generation.
  • Prefill is the weak spot. Macs are relatively stronger at token-by-token decode than at processing very long prompts, which matters for retrieval-heavy and long-document workloads.
  • Vendor benchmarks are not your workload. Measure with your own prompts, context lengths, and quantizations before committing to a hardware purchase.

What to do next

Pick one runtime, install it, and run a small 4-bit model end to end. Record tokens per second with prompts that resemble your actual use case, at the context length you actually need. That single measurement will tell you more about whether your Mac is sufficient than any parameter-count comparison. If you hit a wall, share your machine, macOS version, runtime version, model, quantization, and the exact error in the community — those five details are what make a problem reproducible for someone else.

FAQ

Can I run a 70B model on a 32 GB Mac?

Not comfortably. At 4-bit quantization the weights alone are roughly 35–40 GB, before the KV cache and runtime overhead are added. A 32 GB machine is better matched to models in the 16B–32B range at 4-bit.

Is MLX faster than llama.cpp on a Mac?

There is no universal answer. Both ultimately use Metal, and results depend on the model, quantization format, context length, and runtime version. Benchmark with your own prompts rather than relying on a general claim.

Does unified memory mean I can use all my RAM for the GPU?

No. The pool is shared with macOS and other applications, and the system applies a wired-memory limit to how much the GPU may pin. Plan for usable model space well below total installed memory.

Do I need an internet connection to run a local model?

Only to download the model weights. Once the files are on disk, generation runs entirely on the machine, which is the main privacy argument for local inference.

Why is the first token so much slower than the rest?

The prompt is processed in a single parallel pass called prefill, while output tokens are decoded one at a time. A long prompt therefore produces a long wait before the first token appears.

References

  1. https://machinelearning.apple.com/research/mlx-distributed-inference
  2. https://ai-tldr.dev/learn/local-open-models/running-models-locally/run-llms-on-mac/

Related terms

Unified memory
A single physical memory pool shared by the CPU, GPU, and Neural Engine on Apple silicon, removing the copy step between system RAM and dedicated video memory.
Metal
Apple's low-level GPU programming API. MLX kernels, llama.cpp's Metal backend, and PyTorch's MPS backend all dispatch compute through it.
MLX
Apple's open-source array framework designed for Apple silicon and unified memory; mlx-lm is the LLM-facing package built on it.
Quantization
Reducing the numeric precision of model weights — commonly to 4-bit — to cut memory use at some cost to output fidelity.
KV cache
Stored key and value tensors from earlier tokens that let a model attend to prior context. Its size grows with context length and batch size.
Prefill vs decode
Prefill processes the whole prompt in one parallel pass; decode generates output tokens one at a time. They have different performance characteristics.
GGUF
A model file format used by llama.cpp that bundles weights and metadata, including quantization settings.

Leave a Reply

Your email address will not be published. Required fields are marked *