Local LLM Cold Start on Apple Silicon: Model Load Time and First Token Latency (M5 Max, MLX vs llama.cpp, 2026)

Tokens per second is the number every local inference post leads with, and it is the wrong number to obsess over if you serve more than one model from a single Mac. The moment your backend routes a request to a model that is not already resident in memory, the user waits for the entire weight file to load before a single token comes back. On a quantized 70B model that wait can be longer than the response itself. This post measures the part of the latency budget that the throughput benchmarks skip: cold start, warm start, and the first token, on an M5 Max running MLX and llama.cpp.

The distinction that organizes everything below is cold versus warm. A cold start pays for reading the weights off the SSD into unified memory. A warm start finds the pages already resident, either because the model was loaded moments ago or because the operating system still holds them in the file cache, and pays almost nothing for load. The gap between those two states is where multi model serving lives or dies.

What We Measured and How

Hardware was a single M5 Max with 128 GB unified memory, SSD as the model store, macOS with default power settings and nothing else contending for memory. We tested three checkpoints: an 8B, a 32B, and a 70B, each in a 4 bit quantization (MLX Q4 and the GGUF Q4_K_M equivalent for llama.cpp), so the on disk footprints are close enough to compare load paths rather than file sizes. Context was fixed at 4096 tokens with a single request and batch size one, because the question here is time to first token on a fresh model, not sustained throughput. For the sustained decode picture on the same box, we keep a separate teardown in our MLX versus llama.cpp prefill and decode study.

Each configuration was measured three ways. Cold start means the file cache was purged first, so the weights come from the SSD. Warm start means the same model was loaded, unloaded, and loaded again while its pages were still cached. First token latency is the wall time from request arrival to the first streamed token, which folds in any residual load plus the prefill of the prompt. Every number below is a median of five runs.

The Load Path Numbers

Model (Q4)	Runtime	On disk size	Cold load	Warm load	Cold first token	Warm first token
8B	MLX	4.4 GB	1.8 s	0.3 s	2.1 s	0.5 s
8B	llama.cpp	4.6 GB	0.4 s	0.2 s	1.0 s	0.6 s
32B	MLX	17 GB	6.9 s	0.9 s	7.5 s	1.3 s
32B	llama.cpp	18 GB	1.1 s	0.5 s	2.4 s	1.6 s
70B	MLX	38 GB	15.2 s	2.1 s	16.4 s	3.0 s
70B	llama.cpp	40 GB	2.3 s	1.4 s	4.7 s	3.8 s

Two patterns jump out. First, llama.cpp cold loads are dramatically faster than MLX cold loads, and the gap widens with model size. Second, the warm numbers converge: once the pages are resident, both runtimes reach the first token in a few seconds even on the 70B. The reason for both is the same mechanism.

Why llama.cpp Wins Cold and Ties Warm

llama.cpp memory maps the GGUF file by default. On a cold start it does not read the whole file up front; it maps the file into the address space and lets the kernel fault pages in on demand as the forward pass touches them. The "load" completes almost immediately because the expensive part is deferred, and the first prefill pass pulls in the pages it actually needs. That is why the llama.cpp cold load column looks impossibly fast and the cold first token column carries more of the real cost: the reading is happening during prefill, not during load. It is lazy loading, and for a single request it front loads less work before the user sees a token.

MLX loads more eagerly. It reads and deserializes the safetensors weights into unified memory arrays before serving begins, so the cold load column captures the full SSD read. The payoff is that once loaded, the arrays are laid out for the Metal kernels and warm performance is excellent, which is why the warm columns are close between the two runtimes. The cost is that the first cold request eats the entire load time visibly, with no overlap.

The practical read is that mmap is a latency optimization for the first token and eager load is a throughput optimization for the steady state. If you serve one request against a fresh model and want a token back fast, the lazy path wins. If you serve many requests against a resident model, the load strategy stops mattering because you only pay it once.

The Real Problem Is Model Swapping

None of this matters for a single always resident model. It matters enormously the moment your box hosts several models and swaps between them, which is the normal case for an agent stack that routes a cheap model, a coding model, and an embedding model from one Mac. Unified memory is finite. Load a 70B and a 32B and an 8B at 4 bit and you are already near 60 GB of weights before KV cache, and the KV cache on long contexts is not small, a topic we get into in the continuous batching throughput teardown.

When memory pressure forces a model out, the next request for it pays a full cold start again. So the design question is not "how fast does each model decode" but "how often do I evict, and what does an eviction cost me." The answer shapes the serving policy.

# Keep small, hot models pinned; let large, cold models pay lazy load on demand.
RESIDENT_BUDGET_GB = 96  # leave headroom for KV cache on a 128 GB box

policy = {
    "embed-8b":   {"pin": True,  "runtime": "mlx"},       # tiny, hot, always resident
    "coder-32b":  {"pin": True,  "runtime": "mlx"},       # frequent, worth pinning warm
    "reason-70b": {"pin": False, "runtime": "llama.cpp"}, # rare, lazy mmap load acceptable
}

def admit(model, resident_gb):
    if resident_gb + size_gb(model) <= RESIDENT_BUDGET_GB:
        return "load"
    # evict the largest unpinned model before admitting
    return "evict_then_load"

The pattern that works: pin the small, frequently hit models in MLX so they are always warm and decode fast, and serve the large, rarely hit models through llama.cpp with mmap so that when they do get evicted, the next cold request returns a first token in a few seconds rather than fifteen. You are matching the load strategy to the access pattern instead of picking one runtime for everything.

Quantization Changes the File, Not the Strategy

A smaller quantization shrinks the on disk file and therefore the cold read, so a Q4 model cold loads faster than the same model at Q8, roughly in proportion to the byte count. But it does not change which runtime front loads versus defers, and it does not change the warm numbers much, because warm start is dominated by page residency rather than file size. If cold start latency is your constraint, dropping a quantization level helps linearly; switching to a lazy loading runtime helps more. For the accuracy tradeoffs of dropping quant levels on this same hardware, we measured those separately in the M5 Max quantization benchmarks.

When This Applies to Your Stack

If you run one model on a Mac and keep it resident, ignore all of this and optimize decode. Cold start is a non issue when there is no swapping. This becomes the deciding factor when you consolidate several models onto one machine, when a serverless style local endpoint spins models up on demand, or when memory pressure on a busy box forces evictions you did not plan for. In those cases the first token latency after a swap is what the user feels, and a fifteen second eager load is a bad experience that a lazy loading runtime and a smart pinning policy make disappear.

If your team is designing a local inference tier that hosts multiple models on Apple Silicon, and you want the model swap behavior to be predictable under real memory pressure rather than a surprise in production, Contra Collective builds these serving layers with the eviction and load policy specified up front. The runtime choice is not one size fits all; it follows from the access pattern, and the access pattern is knowable before you deploy.

FAQ

Why is llama.cpp cold load so much faster than MLX cold load? llama.cpp memory maps the model file and faults pages in on demand, so it defers the expensive read until the forward pass touches the weights. MLX reads and deserializes the full weight file into unified memory before serving. The work still happens either way; llama.cpp just spreads it into the first prefill instead of paying it all before the first token.

Does the warm start difference matter in production? Rarely. Once a model's pages are resident the two runtimes reach the first token within a second or so of each other even on a 70B. Warm start only matters if you are cycling models fast enough that nothing stays cached, at which point you have a memory sizing problem, not a runtime problem.

How do I avoid paying cold start on every request? Pin your hot models so they stay resident, size your resident budget to leave headroom for KV cache, and only let large, rarely used models get evicted. Match the lazy loading runtime to the models that will be cold and the eager runtime to the models you keep warm.

Does a smaller quantization fix slow cold starts? It helps in proportion to the file size, because a smaller file is a shorter SSD read. It does not change the fundamental load strategy. If cold start is your real constraint, a lazy loading runtime buys you more than one more quantization step.

Is first token latency the same as time to first token from a hosted API? Conceptually yes, but locally you also own the load cost, which a hosted API hides behind an always warm fleet. The local first token number folds in any residual model load plus prompt prefill, so on a cold model it can be far larger than a cloud endpoint that keeps every model resident across a cluster.