Oversubscribing Unified Memory on M5 Max: What Breaks When a Local LLM Runs Out of RAM (2026)

The pitch for Apple Silicon as a local inference box is unified memory. The CPU and GPU share one pool, so a 64GB machine can hold model weights that would need a specific class of discrete GPU to fit anywhere else. That is real, and it is why an M5 Max runs a 70B model at all. What the pitch leaves out is the failure mode. The same pool that lets you load a large model also feeds the KV cache, the OS, your editor, a browser with forty tabs, and whatever background sync your machine runs. When the sum of all of that crosses what the system can keep resident, macOS starts paging memory to disk, and a local LLM is close to the worst possible workload to page. This post is about where that line sits on an M5 Max, what crosses it in practice, and how decode speed behaves on each side of it.

The short version, from loading models of increasing size on a 64GB M5 Max and watching memory and throughput together, is that the danger is almost never the static weight footprint you planned around. It is the KV cache, which grows with context length and concurrency and is easy to forget when you size the machine. Weights are a fixed cost you pay once at load. The cache is a variable cost that climbs during a long session until it quietly tips you over the wired limit, at which point tokens per second does not sag, it collapses.

Where the Wired Limit Actually Sits

macOS does not let a process wire down all physical memory. A portion is reserved so the system stays responsive, and the GPU can only allocate against what is left. On a 64GB machine the practical ceiling for GPU resident allocation lands well under the full 64GB, and the exact figure depends on the OS version and what else is running. The number that matters is not the sticker capacity, it is the working headroom you have after the system takes its share.

You can raise the GPU allocation ceiling on Apple Silicon with a sysctl knob, iogpu.wired_limit_mb, which lets the GPU wire down more than the conservative default. Teams running large models locally set it higher to fit a model plus its cache. That helps, but it does not create memory. It reallocates the split between what the GPU can hold and what the rest of the system gets, and pushing it too high starves macOS itself, which produces the same paging behavior from the other direction. The knob moves the cliff, it does not remove it.

The honest way to size a machine is to add up three things: the quantized weight footprint, the peak KV cache at your longest expected context and highest concurrency, and a fixed reserve for the OS and your own applications. If that sum is under your working headroom, you are safe. If it is close, you are one long conversation away from the cliff.

The KV Cache Is the Variable That Bites

Weights are the number everyone plans around because they are easy to look up. A 70B model at 4-bit is roughly 40GB, a 32B model at 4-bit is roughly 18GB, and those figures do not change during a session. The KV cache does. It stores the key and value tensors for every token in the context so the model does not recompute attention over the whole history on each step, and its size scales with the number of layers, the hidden dimension, the context length, and how many sequences you serve at once.

Here is a compact way to see the growth. The cache footprint per token is proportional to two times the number of layers times the key and value dimension times the bytes per element, and you multiply that by the number of tokens in context and the number of concurrent sequences.

def kv_cache_gb(layers, kv_dim, tokens, seqs, bytes_per_elem=2):
    # 2 = one key tensor + one value tensor per layer
    per_token_bytes = 2 * layers * kv_dim * bytes_per_elem
    total = per_token_bytes * tokens * seqs
    return round(total / 1e9, 2)

# a 70B-class model, single stream, growing context
for ctx in (8000, 32000, 128000):
    print(ctx, "tokens ->", kv_cache_gb(80, 8192, ctx, 1), "GB")

The point of the snippet is not the exact constants, which vary by architecture, it is the shape. At a few thousand tokens the cache is a rounding error next to the weights. At long context it becomes a second model sitting in memory. A 70B model that loaded comfortably at 40GB of weights can add many gigabytes of cache once a conversation runs to a hundred thousand tokens, and that addition is what carries a machine from fine to swapping. If you serve more than one stream, multiply again. This is the same pressure that makes concurrent serving hard, which we worked through in the concurrent multi model serving analysis.

What the Cliff Looks Like

Below the wired limit, decode speed is steady and predictable. The model generates at whatever tokens per second its size and quantization allow, and adding context slows decode gently because attention has more history to read, not because anything is thrashing. This is the regime you want to live in.

Cross the limit and the character of the slowdown changes completely. Once the working set exceeds what can stay resident, the system pages least recently used memory out to the internal drive and reads it back when the model touches it. For inference that is close to a worst case access pattern, because generating each token walks the full set of weights, so the pages you evicted are exactly the pages you need again a fraction of a second later. Throughput does not drop by a manageable percentage. In our runs it fell by more than an order of magnitude, from a usable tens of tokens per second to a few tokens per second or worse, with long stalls where the interface simply hangs while the drive works. The machine is not broken, it is swapping, and it will stay in that state until the working set shrinks below the line again.

The asymmetry is the important part. Weight footprint gives you a smooth, predictable relationship between model size and speed. Memory oversubscription gives you a discontinuity. There is no gentle warning zone where things get modestly slower and tell you to back off. You are fast, and then you are in swap. That is why sizing by peak cache rather than by weights alone is the whole game.

Sizing So You Never Reach It

The first move is to budget the cache, not just the weights. Decide the longest context you will actually support and the highest concurrency you will actually run, compute the cache footprint at that point, add it to the weights and a reserve, and confirm the total fits your working headroom. If it does not, you have three levers before you buy a bigger machine.

Quantize the KV cache. Storing keys and values at 8-bit or 4-bit instead of 16-bit cuts the cache footprint by half or more, which directly buys back context length or concurrency, at a quality cost that is usually small and worth measuring on your own tasks. We compared the quality side of weight quantization in the Q4 versus Q8 versus FP16 analysis, and the cache follows the same logic.

Cap the context. A hard limit on tokens per session, with summarization or truncation of the oldest turns, keeps the cache bounded no matter how long a conversation runs. This is less elegant than infinite context but far better than a machine that hangs at token ninety thousand.

Offload deliberately rather than accidentally. Paging to disk under memory pressure is catastrophic because it is unplanned and touches hot weights. Explicit offload of cold cache pages to fast storage, managed by the serving layer rather than the OS pager, is a different and more controllable thing, which we covered in the KV cache NVMe offload piece. The difference between the two is the difference between choosing what leaves memory and letting the pager guess.

When This Applies to Your Stack

If you are running a local model behind a real product feature rather than a demo, unified memory sizing is not a spec sheet exercise, it is a reliability property. A build that passes every test on short prompts can fall into swap the first time a real user holds a long conversation, and the failure looks like a hang rather than an error, which makes it hard to diagnose. Size for peak KV cache, quantize the cache when you need the room, cap context deliberately, and keep a reserve for the rest of the machine.

If your team is putting local inference into a backend and wants the capacity model, the cache budget, and the fallback behavior designed so the machine degrades on purpose instead of by accident, Contra Collective builds AI infrastructure that holds up under real load rather than benchmark load. The weights are the easy number. The cache is the one that decides whether the thing stays up.

FAQ

How much RAM do I need to run a 70B model locally? For a 4-bit 70B model, the weights are roughly 40GB, but that is not the number to size on. Add the KV cache at your longest context and highest concurrency, which can add many gigabytes at long context, plus a reserve for macOS and your applications. On a 64GB M5 Max a 70B model runs well for short and medium contexts and gets tight at very long context or multiple concurrent streams.

What is the wired memory limit on Apple Silicon? macOS reserves a portion of physical memory for the system and lets the GPU wire down the rest. The practical ceiling on a 64GB machine is well under 64GB by default. The iogpu.wired_limit_mb sysctl can raise the GPU share, but pushing it too high starves the OS and triggers the same paging problem from the other side.

Why does my local LLM suddenly get extremely slow? Almost always memory oversubscription. As a conversation grows, the KV cache grows with it, and once the total working set exceeds what stays resident the system pages weights to disk and reads them back every token. That collapses throughput by an order of magnitude or more and shows up as stalls or hangs rather than a clean error.

Does quantizing the KV cache help? Yes, substantially. Storing keys and values at 8-bit or 4-bit instead of 16-bit roughly halves or quarters the cache footprint, which buys back context length or concurrency for a small, measurable quality cost. It is one of the most direct levers for staying under the wired limit at long context.