AIForum.

What Is Unified Memory and Why Does It Matter for Local AI on Apple Silicon?

On this page

THE SHORT ANSWER

Unified memory is a single pool of physical DRAM shared by the CPU and GPU on the same Apple silicon chip. For local AI it removes the PCIe copy step and raises the capacity ceiling, but it is not unlimited memory and it does not guarantee faster tokens. Here is how the architecture works, how to estimate whether a model fits, and where the real limits are.
Quick facts

Unified memory is a single pool of physical DRAM on the same chip as the CPU and GPU, which both address directly. For local AI on Apple silicon, that produces two practical effects: model weights do not have to be copied across a PCIe bus before the GPU can read them, and the addressable pool is the machine’s whole memory rather than a separate, usually smaller, VRAM budget. It is the reason a Mac Studio can hold a frontier-scale model in one Metal buffer instead of splitting it across devices.

This explainer is for readers who already know roughly what a model file is and want to reason about whether a given machine will run a given model — and why memory, rather than raw compute, is usually the first wall they hit.

The architecture in plain terms

On a discrete GPU — an NVIDIA card in a desktop, for example — the CPU and the GPU sit next to separate pools of memory chips. The CPU has system RAM; the GPU has VRAM on the card. The two pools are wired together by a bus such as PCIe, and every time the CPU wants the GPU to see a weight tensor, the bytes must be copied across that bus. As one inference-engine reference puts it, a 16.76 GB model means shipping 16.76 GB over PCIe before the first token can even be thought about.

Apple silicon is built differently. The CPU cores and the GPU are on the same piece of silicon — a system-on-chip, or SoC — and both point at the same physical DRAM. There is no second pool and no bus to copy across. That single-pool design is what “unified memory” names. Apple’s own framing matches this: the company describes MLX as an open-source array framework that leverages the unified memory architecture of Apple silicon so researchers and developers can run inference and fine-tuning locally, in a private environment.

PropertyUnified memory (Apple silicon)Discrete GPU (typical)
Where weights liveThe same DRAM the CPU usesSeparate VRAM on the card
How the GPU gets themReads directly; no copy stepCopied across PCIe before use
Capacity ceilingThe machine’s total memoryThe card’s VRAM
BandwidthShared by CPU and GPUDedicated to the GPU

At the Metal level, this is why a single MTLBuffer can hold an entire model. The inference-engine chapter linked in the references describes exactly that case: one buffer holding a 16.76 GB model on a machine where the GPU already reads the same DRAM the CPU does, with set_buffer(slot, buffer, offset) binding it for a command encoder.

Why it changes local inference in practice

The copy step disappears

On a discrete setup, moving weights to the GPU is a real cost with a real failure mode: if the model does not fit in VRAM, the runtime must split it, offload layers to the CPU, or stream tensors over the bus during generation. Each of those strategies adds latency and complexity. With a shared pool, there is nothing to move. The weights are already where the GPU reads from, so the first forward pass can begin without a transfer phase.

That is a structural simplification, not a speed guarantee. It removes a bottleneck; it does not add compute throughput.

Capacity is usually the binding constraint, not compute

For local inference, the question “will this run?” is almost always a memory question first. A model that fits runs, even if slowly; a model that does not fit will not run at all without offloading tricks. Because unified memory is sized like system RAM rather than like a graphics card’s VRAM, Apple silicon machines can address model sizes that would be impractical on a comparably priced discrete card. Apple cites up to 512 GB of unified memory on a single Mac Studio.

It makes one-machine deployment plausible

There is an operational benefit beyond raw numbers. A single machine with one memory pool is easier to reason about, easier to secure, and easier to run offline than a multi-device setup with a transfer pipeline. That matters for anyone running private workloads where data should not leave the building.

What unified memory does not do

  • It is not unlimited memory. The operating system, your applications, the browser, and the GPU all draw from the same pool. A machine with 64 GB of unified memory does not have 64 GB available to a model.
  • macOS applies a wired-memory limit. The system caps how much memory can be locked for GPU work. The default depends on the machine and OS version and can be raised, but the exact figure should be checked against current Apple documentation rather than assumed.
  • Bandwidth is shared. CPU-side work — tokenization, data loading, preprocessing — competes with GPU reads for the same memory bandwidth.
  • It does not scale past one machine by itself. Unified memory is per-device. To go beyond one Mac’s memory, you need to distribute the model across machines, which is exactly what Apple’s RDMA over Thunderbolt work addresses.
  • It says nothing about compute. A discrete GPU with a large VRAM pool and high bandwidth can still outperform a Mac at the same model size. Unified memory raises capacity and removes a copy; it does not promise higher tokens per second.

Estimating whether a model will fit

Weight size is only the first term. A workable estimate for local inference is:

total ≈ weights + KV cache + runtime buffers + OS and app overhead

Weights scale with parameter count and the number of bytes per parameter your quantization uses. The KV cache scales with context length, and it is the term people most often forget. A common approximation is:

KV bytes ≈ 2 × layers × kv_heads × head_dim × context_tokens × bytes_per_element × batch_size

The leading 2 covers the key and value tensors. Grouped-query attention reduces kv_heads relative to the attention head count, which is why modern models often have a smaller KV cache than the raw layer count suggests.

As an arithmetic illustration — not a measurement of any real model — take a hypothetical configuration with 80 layers, 8 KV heads, head dimension 128, a float16 cache, and batch size 1. Per token: 2 × 80 × 8 × 128 × 2 bytes = 327,680 bytes, roughly 0.31 MiB. At 8,192 tokens that is about 2.5 GiB on top of the weights. Change any input and the number moves; measure rather than assume.

Checking real memory use on a Mac

Before trusting any estimate, look at what the machine is actually doing. These are standard macOS tools; confirm the exact output on your own OS version.

sysctl hw.memsize      # total physical memory, in bytes
vm_stat                # page-level memory statistics
memory_pressure        # current memory pressure summary

Activity Monitor’s Memory tab is the friendlier view: watch memory pressure and swap usage while a model is loaded and generating. Rising swap during generation is the clearest sign that the working set — weights plus KV cache plus everything else — exceeds what the system is comfortable holding.

A sourced example: MLX on Mac Studio

Apple’s October 15, 2026 post on local and distributed inference gives a concrete picture of what this architecture enables at the top end. Using the latest MLX release, Apple reports that a cluster of four Mac Studios generates over 36 tokens per second on Muse Spark Open, a model with nearly a trillion parameters, at a context size of 8,192 tokens. The post describes three parallelization schemes, including pipeline parallelization, and notes that scaling across multiple Macs uses an RDMA over Thunderbolt driver.

Two details are worth separating. The single-machine claim is about capacity: a Mac Studio can be configured with up to 512 GB of unified memory. The four-machine claim is about distribution: once a model exceeds one machine’s memory, the cluster is the answer, and the interconnect becomes part of the performance story. Both figures come from Apple and should be read as provider-reported results rather than independently reproduced benchmarks.

When unified memory is not the answer

Unified memory is a strong fit when you want to run a large model privately on one desk, when your models change often and you would rather not manage VRAM budgets, and when you value a single-machine deployment you can reason about end to end.

It is a weaker fit when your priority is maximum tokens per second at a fixed model size, when you already own a discrete GPU with enough VRAM, or when your workload is bursty enough that paying for cloud inference only when needed is cheaper than owning the hardware. None of those are failures of the architecture; they are different constraints.

For WordPress and automation work specifically, the relevant question is usually smaller: whether a modest local model can handle classification, summarization, or drafting without sending content to a third party. That decision depends far more on the model and the workflow than on whether the memory is unified.

Next step

Pick one model you actually want to run. Note its parameter count and quantization, estimate the weights, add a KV cache estimate for your target context length, and compare the total against your machine’s memory with headroom for the OS. Then load it and watch memory pressure. If it swaps, reduce context length or quantization before assuming the machine is too small.

If you have run into a specific wall — a model that loads but stalls, a context length that triggers swapping, or a cluster setup that underperforms — share your machine, model, quantization, context length, and observed memory use in the community so others can compare against a real configuration rather than a guess.

FAQ

Does unified memory make a Mac faster than a discrete GPU for local LLMs?

Not automatically. It removes the PCIe copy step and raises the capacity ceiling, but tokens per second still depends on compute throughput and memory bandwidth. A discrete GPU with enough VRAM and high bandwidth can be faster at the same model size.

Can the GPU use all of the Mac's RAM?

No. The operating system, applications, and GPU share the same pool, and macOS applies a wired-memory limit that caps how much can be locked for GPU work. Check the current default for your machine and OS version rather than assuming a fixed share.

Is unified memory different from regular RAM?

It is the same physical DRAM. What differs is who can address it: on Apple silicon the CPU and GPU both read the same pool directly, rather than the GPU having its own separate VRAM.

Why does context length matter so much for memory?

Because of the KV cache. It scales with the number of tokens in context and with batch size, so a model that fits comfortably at a short context can exceed available memory at a long one.

Do I need unified memory to run local models?

No. A discrete GPU with sufficient VRAM, a multi-GPU machine, or cloud inference are all viable. Unified memory is one way to solve the capacity and copy problem, not the only one.

References

  1. https://machinelearning.apple.com/research/mlx-distributed-inference
  2. https://highperformanceailab.com/muser-book/chapters/03-unified-memory-and-buffers.html

Related terms

VRAM
Dedicated memory on a discrete graphics card, separate from system RAM and reached over a bus such as PCIe.
PCIe
A bus that connects a discrete GPU to the CPU and system memory; weight tensors must be copied across it before the GPU can use them.
KV cache
The stored key and value tensors for previously processed tokens. It grows with context length and batch size and is a major part of memory use during generation.
Wired memory
Memory locked so it cannot be paged out. macOS caps how much can be wired for GPU work, which limits how much of the pool a model can practically occupy.
MLX
Apple's open-source array framework for Apple silicon, designed around the unified memory architecture and used for local inference and fine-tuning.
Quantization
Storing weights at lower precision (for example 8-bit or 4-bit) to reduce memory use, usually at some cost to output quality.

Leave a Reply

Your email address will not be published. Required fields are marked *