Running Two Models At Once On An M5 Max: Unified Memory Budgeting And Model Swap Latency (2026)

Almost every local inference backend starts with one model and stays honest right up until the first real feature request. Then someone wants a cheap classifier to decide whether an incoming message even needs the big model, or an embedding model resident for retrieval, or a small draft model to speed decoding, and suddenly the box that comfortably served one 70B model has to hold three things at once. On an M5 Max that is not a scheduling problem the way it would be on a discrete GPU. It is a memory budgeting problem, because unified memory is one pool shared by every resident model, every KV cache, and the operating system, and the pool does not grow when your ambitions do.

This post is about the decision that actually bites in production: when you keep more than one model hot, how much memory does each one really cost, how long does it take to swap between them if you do not keep them resident, and where is the crossover where paying the memory to keep two models loaded beats paying the latency to swap on demand. The hardware is a 128GB M5 Max, which is the configuration most small teams end up buying because it is the cheapest Apple Silicon part that can hold a quantized 70B model with room to work.

The Memory Pool Is The Whole Story

On a discrete GPU you think in terms of VRAM per card, and a second model either fits in the leftover VRAM or it does not. Unified memory changes the shape of the problem because the model weights, the KV cache, the activation buffers, and the OS all draw from the same 128GB. Apple reserves a slice for the system and, by default, caps how much any single process can wire down for the GPU. You can raise that cap, but you cannot raise the total, so every gigabyte you commit to a second resident model is a gigabyte you cannot spend on context length for the first.

Here is the practical budget for a 128GB M5 Max, measured as resident footprint under MLX with the model loaded and a modest context active.

Component	Quantization	Resident memory	Notes
Llama 3.3 70B	4 bit	40 GB	Primary generation model
Qwen3 8B	4 bit	5.2 GB	Router or triage model
BGE-M3 embeddings	8 bit	1.3 GB	Retrieval, always resident
KV cache, 70B, 32K context	fp16	10.5 GB	Grows linearly with context
KV cache, 70B, 128K context	fp16	42 GB	The number that breaks the budget
OS and wired overhead	n/a	12 GB	System reserve plus your app

Add the top of that table together at 32K context and you are near 69GB, which fits comfortably. Push the 70B model to 128K context and the KV cache alone wants 42GB, and now the three resident models plus the cache plus the OS clear 100GB, and you are one large retrieval batch away from the memory pressure that triggers compression and then swap. The lesson is blunt: on this machine, context length and model count are the same budget spent twice, and you cannot maximize both.

What A Model Swap Actually Costs

The alternative to keeping every model resident is to load on demand and evict when idle. The question is whether the swap is cheap enough to hide. It is not a single number, because loading a model on Apple Silicon has two very different costs depending on whether the weights are already in the file system cache.

Model	Cold load from disk	Warm load from file cache	Evict and free
Qwen3 8B, 4 bit	3.1 s	0.9 s	0.2 s
Llama 3.3 70B, 4 bit	14.8 s	4.2 s	0.6 s
Qwen3 235B MoE, 4 bit	41 s	11 s	1.4 s

Cold load is dominated by reading weights off the SSD, so it scales with file size and the internal NVMe bandwidth. Warm load is faster because the pages are already in the file system cache, but a warm load still has to wire the memory and initialize the Metal buffers, which is why even an 8B model does not load instantly. The eviction cost is small, which matters because it means the expensive direction is always loading, never unloading. If you want the full breakdown of where those seconds go, the local model cold start and load latency test takes the load path apart layer by layer.

The swap cost that hurts most is the one you did not plan for. If you evict the 70B model to make room for a burst of embedding work and then a chat request arrives, the user waits 4 seconds on a warm cache and 15 on a cold one before the first token. That is not a latency you can hide behind a spinner in an interactive product.

The Crossover: Resident Versus On Demand

The decision comes down to request mix and tolerance for tail latency. Keep a model resident and you pay its memory the entire time it sits idle. Swap it in on demand and you pay its load latency on the first request after every idle gap. The crossover is a function of how often the model is called and how much memory headroom you actually have.

A useful way to reason about it: if a model is called more than once per its own warm load time, keeping it resident is almost always right, because you would otherwise be reloading it constantly. The router and the embedding model in the table above are called on nearly every request, so they should never be swapped. The big generation model is the interesting case. If your traffic is bursty, with long quiet gaps between conversations, and you need those quiet gaps to run large retrieval or reranking batches that want the memory, then swapping the 70B model out during the quiet periods can be worth the occasional 4 second cold cache start. If your traffic is steady, keep it resident and cap the context length instead.

There is a third pattern that beats both for many workloads: keep a small model resident for the common path and only swap up to the large model when the router decides the request genuinely needs it. On a support workload where most messages are simple, an 8B model resident at all times plus an occasional 70B swap in produced a lower average latency than a 70B model held resident and starved of context, because the common case never paid the big model's cost at all. That pattern only works if your router is accurate, because every misroute that swaps the 70B model in unnecessarily pays the full load latency for nothing.

Serving Concurrency On Top Of All This

None of the above accounts for what happens when two requests hit the same resident model at once. A second concurrent request against the 70B model does not double the memory the way a second model would, but it does add another KV cache allocation, and those allocations are what actually run you out of the pool. Continuous batching helps you share the model's compute across requests without loading it twice, and it is the right default once you have more than a trickle of traffic, but it makes the memory math harder because concurrent contexts stack. The MLX continuous batching throughput measurements show how far you can push concurrent decode before the batch stops scaling, and the KV cache and context length analysis shows exactly how fast each concurrent context eats the budget you were trying to protect.

The uncomfortable synthesis is that on one M5 Max you are trading between three things that all draw from the same pool: how many models you keep hot, how long a context each one gets, and how many concurrent requests you serve. You get to maximize two. Pick which two based on your workload, and be explicit about the one you are giving up.

When This Applies To Your Stack

If you are building a local backend that needs a router, an embedding model, and a large generation model on a single Mac, budget the memory before you write the scheduler, because the scheduler cannot allocate memory that the context length already spent. Keep the always called small models resident, decide the large model's residency from your traffic shape, and treat context length as the variable you tune down when the budget gets tight rather than the constant you defend at all costs.

We help teams design local inference infrastructure that has to survive real traffic on real hardware, including the capacity planning that decides whether one Mac is enough or whether the workload wants a second box. If you are sizing a local deployment and want the memory budget and swap strategy worked out before you commit to hardware, that is the kind of AI infrastructure work we do.

FAQ

How much memory does a second model really add on an M5 Max? Its resident footprint plus its share of KV cache. A 4 bit 8B model is around 5GB resident, which is cheap, but the expensive part is that keeping it hot reduces the context headroom available to your large model, since both draw from the same unified pool.

Is it faster to keep a 70B model resident or swap it in on demand? Resident wins for steady traffic because you never pay the load latency. On demand only wins when the model sits idle long enough that the memory it frees can do useful work, and you can tolerate a warm load of a few seconds on the first request after each idle gap.

Can I run a 70B model at 128K context and a second model at the same time on 128GB? Not comfortably. The 70B KV cache at 128K context alone approaches 42GB, which combined with the weights, a second model, and OS overhead leaves almost no safety margin. Choose either the long context or the second resident model, not both.

Does continuous batching change the multi model math? Yes. Batching lets one resident model serve concurrent requests without a second copy, but each concurrent context adds a KV cache allocation, so throughput and memory pressure rise together. It reduces the cost of concurrency against one model, not the cost of holding many models.