KV Cache and Context Length on Apple Silicon: The Memory Math for Local Inference (2026)

The model weights are not what kills your context length on Apple Silicon. The KV cache is. We have measured this across dozens of configurations on M4 Pro, M4 Max, and M5 Max hardware, and the same pattern shows up every time: engineers size unified memory for the model file, then watch their inference server OOM at 16K or 32K tokens because nobody did the cache math.

This is the explainer that should exist when someone types llama.cpp mlx backend, does llama.cpp support mlx, or kv cache apple silicon into a search bar. Here is what the memory math actually looks like, runtime by runtime, and what it means for your context length budget.

What the KV Cache Actually Costs

For a standard transformer with multi-head attention, the KV cache size per token is:

kv_per_token_bytes = 2 (K and V) * num_layers * num_kv_heads * head_dim * dtype_bytes

That 2 is K and V. The total cache for a sequence is that number multiplied by sequence length and batch size. For Llama 3.3 70B with 80 layers, 8 KV heads (grouped query attention), 128 head dim, at fp16:

2 * 80 * 8 * 128 * 2 = 327,680 bytes per token = 320 KB per token

At 32K context, that is 10 GB. At 128K context, 40 GB. The 70B weights at Q4_K_M are about 40 GB. So on a 64 GB M4 Max, you can fit the weights and 64K of fp16 KV cache, no headroom for activations or batch. At 128K context, you are over budget by 16 GB before the first token gets generated.

For Llama 3.3 8B with 32 layers, 8 KV heads, 128 head dim at fp16, the cost is 8 KB per token. 128K context is 1 GB. That is why the 8B class is forgiving and the 70B class is brutal.

Grouped query attention (GQA) is what makes any of this tractable. Llama 3 70B has 8 KV heads instead of 64 attention heads. That is an 8x reduction in cache size versus a non-GQA model of the same dimensions. If you are picking models for long context on Apple Silicon, GQA is non-negotiable.

llama.cpp: KV Quantization is the Lever

llama.cpp lets you quantize the KV cache independently of the weights. The flags are --cache-type-k and --cache-type-v. Options include f16 (default), q8_0, q5_0, q4_0, and on recent builds q4_1 and iq4_nl.

Practical impact on Llama 3.3 70B at 32K context:

Numbers are from our internal benchmark suite, Llama 3.3 70B Q4_K_M weights, 32K context, single user, M5 Max 40GB. Throughput hit from KV quantization is small (less than 10 percent at q4_0). The memory savings are large. For most workloads on memory-constrained Apple Silicon, q8_0 KV cache is the right default: half the memory, no measurable quality cost.

Note that some llama.cpp builds require flash attention (-fa) to be enabled before non-fp16 KV types take effect. Without it, the flag is silently ignored. Check the startup log.

MLX and mlx-lm: Different Memory Model

MLX uses Apple's unified memory and lazy evaluation. The KV cache lives in the same memory pool as everything else, but the allocator behaves differently from llama.cpp's pre-allocated buffer. You see less explicit OOM and more gradual slowdown as the system pages.

mlx-lm exposes KV quantization through the --kv-bits flag (added in mlx-lm 0.19). Options are 2, 4, 8 bit, with --kv-group-size controlling quantization granularity. The defaults are sane: 8 bit, group size 64. Quality cost is negligible at 8 bit, similar story to llama.cpp's q8_0.

The real difference: MLX does not pre-allocate the full context window's KV cache. It grows as the sequence grows. llama.cpp pre-allocates the entire n_ctx at startup. If you set n_ctx=131072 in llama.cpp, you pay for 40 GB of KV cache up front for a 70B model at fp16, whether you use it or not. MLX is lazier about this, which is friendlier on a shared development workstation but harder to reason about in production.

For a multi-user server, llama.cpp's pre-allocation is actually the right behavior: you want to know at boot whether you have the memory, not at request time.

Does llama.cpp Support MLX?

Short answer: no, not as a backend, and that is intentional.

This is the question that drives the llama.cpp mlx backend and does llama.cpp support mlx queries we see in search data. The architectural answer is that llama.cpp and MLX are different stacks. llama.cpp uses its own Metal kernels (via ggml-metal) for GPU acceleration on Apple Silicon. MLX is Apple's array framework with its own kernels and graph compiler. There is no in-process bridge between them.

What people usually mean when they ask this:

1. Can I run MLX-format models (.mlx, .npz) in llama.cpp? No. You need GGUF.
2. Can I run GGUF models in MLX? Not directly. You convert weights via mlx_lm.convert or use an MLX-native quantization.
3. Is MLX faster than llama.cpp's Metal backend? Sometimes, on prompt processing especially, and getting better with each MLX release. For pure decode throughput on quantized 70B models, llama.cpp's Metal kernels are still competitive and often ahead.

If you want both worlds in one stack: pick mlx-lm for the MLX-native models and integration with the Apple framework graph, or pick llama.cpp for the broadest model ecosystem and most mature KV quantization tooling. There is no productive way to make them share the same process.

Context Length Budget by Hardware

For a single-user local inference workstation, the practical context budgets at fp16 KV cache:

These assume single batch, room for activations, and OS overhead. Move to q8_0 KV cache and most rows roughly double their context budget. Move to q4_0 KV cache and you can roughly quadruple it, with the quality caveats above.

The 70B + long context combination is the breaking point on every M4 and most M5 hardware. If your workload requires both, you are either on M5 Ultra, on Mac Studio with 192 GB, or you are moving the workload off Apple Silicon to a real GPU server.

Practical Recommendations

For a small engineering team running local inference on Apple Silicon:

1. Default to q8_0 KV cache in llama.cpp, 8 bit --kv-bits in mlx-lm. The quality cost is invisible; the memory savings let you run longer context or larger models.
2. Do not size your unified memory for the weights file alone. Add at least 30 percent for KV cache, activations, and OS pressure. More if you need long context.
3. Pre-allocate context windows in llama.cpp to the actual size you will use, not the model's max. Setting n_ctx=131072 on a 70B costs you 40 GB at boot.
4. If you are running 70B class models, GQA is mandatory. Stay on Llama 3.x or Qwen 2.5+, avoid older non-GQA architectures regardless of how good the benchmarks look.
5. Use llama.cpp for serving (pre-allocated, predictable), use mlx-lm for experimentation and notebooks (lazy, friendlier on a shared machine).

When This Applies to Your Stack

If your team is running local LLMs on MacBooks or Mac Studios for development, evals, or low-latency on-device inference, this is the math that determines whether your context window plans are realistic. Most teams over-spec the weights, under-spec the KV cache, and discover the gap when a 32K prompt finally lands in QA. The KV cache budget should be sized first, then the weights chosen against the remaining memory.

If you are running anything beyond a single developer per machine, llama.cpp with explicit pre-allocation is the production-shaped path. MLX is the right call for experimentation. Mixing them is fine; sharing a process is not.

Contra Collective builds local and hybrid inference infrastructure for enterprise teams, including Apple Silicon clusters for development environments and hardened GPU deployments for production serving. If your team is hitting context-length walls on local hardware, the answer is usually a KV cache configuration change before it is a hardware purchase.

FAQ

Q: Does llama.cpp support MLX as a backend? No. llama.cpp uses its own Metal kernels via ggml-metal. MLX is a separate stack with its own kernels. There is no shared backend.

Q: Can llama.cpp run MLX-format models? No. llama.cpp requires GGUF format. You can convert Hugging Face weights to GGUF, but not from MLX-native formats directly. Going the other way, mlx_lm.convert produces MLX-format weights from Hugging Face checkpoints.

Q: Is KV cache quantization safe? At q8_0 (llama.cpp) or 8 bit (mlx-lm), quality impact is negligible across all workloads we tested. At q4_0 / 4 bit, you start seeing degradation on long reasoning chains and code generation. q5_0 is a reasonable middle ground.

Q: How do I check my actual KV cache memory usage in llama.cpp? Run with --verbose and check the startup log. The KV self size is reported in MiB. You can also compute it directly: 2 * n_layers * n_kv_heads * head_dim * n_ctx * dtype_bytes.

Q: Why does my 70B model OOM at 32K context on a 64 GB Mac? The weights at Q4_K_M are about 40 GB. fp16 KV cache at 32K is 10 GB. Activations, context, and OS overhead use 8 to 12 GB. You are at the edge. Switch to q8_0 KV cache and you free 5 GB.