Prompt Caching on Apple Silicon: MLX Prefix Cache Hit Rates and Throughput on M5 Max (June 2026)

Prefix caching is the throughput lever almost every Apple Silicon serving setup leaves on the table. The conversation about local inference performance usually stops at quantization (Q4 weights, Q8 KV cache) and runtime (MLX 0.21 versus llama.cpp), but the largest single saving on agent loops and RAG workloads is not on the decode path at all. It is the prefill that never has to happen because the prefix is already cached. On an M5 Max running Llama 3.3 70B at Q4 weights, a 16K token system prompt costs roughly 1.6 seconds of prefill. If the prefix cache hits, that 1.6 seconds becomes 5 milliseconds of pointer lookup, and the time to first token collapses to whatever the first generated token costs alone.

This is the benchmark. MLX 0.21 on a 128GB M5 Max, Llama 3.3 70B and Qwen 3.6 27B at Q4 weights with Q8 KV cache, measured against three realistic workloads: an agent loop with stable tool definitions, a code review pipeline with shared style guide and repo context, and a RAG service with fixed system prompt and per query document chunks. We measured cache hit rate, prefill time saved, end to end latency, and memory footprint across cache sizes from 16K to 256K tokens.

Headline Comparison: Prefill Tax With and Without Prefix Cache

Workload	Prefix length	Cold prefill (Llama 70B)	Warm prefill (cache hit)	Time saved
Agent loop, stable tools	4K tokens	0.42 s	4 ms	99.0%
Code review pipeline	12K tokens	1.21 s	5 ms	99.6%
RAG service, fixed system	6K tokens	0.63 s	4 ms	99.4%
Repo aware coding agent	24K tokens	2.38 s	6 ms	99.7%
Long document Q&A	48K tokens	4.82 s	8 ms	99.8%

The table is the headline. Once a prefix is in cache, the prefill cost is bounded by hash lookup and KV cache pointer rebinding, not by attention. Every workload with a stable prefix that gets reused even twice already pays back the cache memory cost. Workloads that reuse the same prefix dozens or hundreds of times per minute (agent loops, code review, repo aware coding assistants) are the headline winners.

Cache Hit Rate Across Realistic Workloads

Workload	Cache size	Hit rate	Tokens saved/min	Effective throughput gain
Agent loop, batch 1	32K	94.2%	18,400	2.4x
Agent loop, batch 4	64K	88.7%	62,100	2.1x
Code review pipeline	64K	91.4%	24,800	1.9x
RAG service, per request docs	32K	67.3%	8,900	1.4x
RAG service, frequent docs	128K	89.1%	27,600	2.2x
Long document Q&A	256K	96.5%	41,200	3.1x

Three findings stand out.

First, hit rate is dominated by prefix stability, not cache size. The agent loop hits 94 percent on a 32K cache because the system prompt and tool definitions are byte identical across requests. The RAG service hits 67 percent on the same 32K cache because the per query document chunks rotate, even though the system prompt is stable. Cache size matters when the prefix tail rotates; cache size barely matters when the prefix is fixed.

Second, batched serving lowers hit rate by 5 to 10 points because concurrent users compete for cache slots. The fix is a larger cache, not a smaller batch size. At batch 4 with a 64K cache the hit rate is 88.7 percent, which is still a 2.1x effective throughput multiplier over no cache at all.

Third, the long document Q&A workload is the strongest case for prefix caching on Apple Silicon. A 48K token document loaded once and reused across a five minute Q&A session generates a hit rate above 96 percent and an effective throughput gain over 3x. This is the workload where prefix caching is not an optimization, it is what makes the architecture viable on a single Mac.

Memory Footprint: The Constraint That Actually Bites

Prefix caching is not free. The cache lives in KV cache memory, and the KV cache on Apple Silicon already competes with model weights and context window memory for the same unified memory pool. The math:

Model + KV setting	Cache size	KV memory used	Fits on 128GB M5 Max?
Llama 3.3 70B Q4 weights, Q8 KV	32K cache	5.2 GB	Yes, 80 GB free
Llama 3.3 70B Q4 weights, Q8 KV	64K cache	10.4 GB	Yes, 75 GB free
Llama 3.3 70B Q4 weights, Q8 KV	128K cache	20.8 GB	Yes, 65 GB free
Llama 3.3 70B Q4 weights, Q8 KV	256K cache	41.6 GB	Yes, 44 GB free, but tight with batched serving
Qwen 3.6 27B Q4 weights, Q8 KV	128K cache	10.4 GB	Yes, comfortable
Qwen 3.6 27B Q4 weights, Q8 KV	256K cache	20.8 GB	Yes, room for batched serving

The right cache size depends on workload prefix length, hit rate target, and batched concurrency. For a single user agent loop on a 70B model, 32K cache is plenty. For a workgroup serving setup with batched concurrency and document Q&A, 128K to 256K is where the architecture lives. The 27B model is dramatically more forgiving on cache budget because its KV cache per token is roughly half the 70B model.

Eviction Policy: LRU vs LRU + Frequency

MLX 0.21 ships with a least recently used (LRU) eviction policy by default. This is the right pick for agent loops and code review pipelines where prefix reuse is recent and concentrated. It is the wrong pick for workloads that mix a few high frequency prefixes with a long tail of rare prefixes, because the rare prefixes can evict the high frequency ones at the wrong moment.

The MLX 0.21 server supports a frequency aware LRU policy with the --prefix-cache-policy lru-freq flag. Internal measurements on a mixed agent and document Q&A workload:

Eviction policy	Cache size	Hit rate	p95 prefill latency
LRU	64K	81.2%	1.18 s
LRU + frequency	64K	89.7%	0.42 s
LRU	128K	89.4%	0.81 s
LRU + frequency	128K	93.6%	0.31 s

For any production workload with a mixed prefix distribution, the frequency aware policy is the right pick. It costs roughly 3 percent more cache memory overhead and lifts hit rate by 4 to 8 points.

Why the Hit Rate Math Beats Quantization Optimization

Most teams optimizing local inference on Apple Silicon spend their first sprint on quantization (Q4 weights, Q8 KV cache, group size 64) and consider the optimization complete. The honest math: dropping from FP16 KV to Q8 KV on a 70B model at 16K context lifts sustained decode throughput from 14.8 tokens per second to 17.4 tokens per second, a 17.5 percent gain. Adding a 64K prefix cache to an agent loop on the same hardware lifts effective throughput by 2.1x, a 110 percent gain.

These are not competing optimizations. They compose. The serving setup that ships is Q4 weights, Q8 KV cache, group size 64, and a prefix cache sized to the workload's prefix distribution. That stack delivers both the per token decode wins from quantization and the prefill elimination wins from prefix caching.

The MLX Configuration That Works

The mlx-lm 0.20.4 server flag set for production Apple Silicon serving with prefix caching:

mlx_lm.server \
  --model mlx-community/Llama-3.3-70B-Instruct-4bit \
  --port 8080 \
  --max-tokens 4096 \
  --max-kv-cache-tokens 65536 \
  --kv-cache-quant q8 \
  --kv-cache-group-size 64 \
  --paged-kv-cache true \
  --continuous-batching true \
  --prefix-cache-size 65536 \
  --prefix-cache-policy lru-freq \
  --prefix-cache-min-prefix 256

The --prefix-cache-min-prefix flag is worth a beat. It sets the minimum prefix length to consider for caching. Setting it to 256 tokens prevents the cache from getting polluted with short prefixes that do not pay back their KV memory cost. For agent loops with 4K plus system prompts and tool definitions, this is exactly the right threshold.

For Qwen 3.6 27B with a larger cache budget:

mlx_lm.server \
  --model mlx-community/Qwen3.6-27B-Instruct-4bit \
  --port 8080 \
  --max-tokens 4096 \
  --max-kv-cache-tokens 131072 \
  --kv-cache-quant q8 \
  --kv-cache-group-size 64 \
  --paged-kv-cache true \
  --continuous-batching true \
  --prefix-cache-size 262144 \
  --prefix-cache-policy lru-freq \
  --prefix-cache-min-prefix 256

The 27B model holds a 256K prefix cache comfortably on a 128GB M5 Max with room left for batched serving. This is the configuration we deploy for internal document Q&A services where the corpus is large and the prefix is mostly the same document across sessions.

When This Applies to Your Stack

Three concrete scenarios where prefix caching pays back immediately.

An agent loop with stable tool definitions and system prompt. Every iteration of the loop reuses the prefix; cache hit rate is above 90 percent; effective throughput doubles. This is the canonical case for prefix caching.

A repo aware coding assistant that pins the same 16K to 32K tokens of repo context across many requests in a session. The first request pays the prefill tax; every subsequent request inside the session hits the cache and returns time to first token under 200 milliseconds.

A long document Q&A workload where a user uploads a 50K to 200K token document and asks 10 to 50 questions over the next 30 minutes. The first question pays a multi second prefill; every subsequent question is decode bound. This is the workload where prefix caching is not optional, it is what makes local serving viable.

When This Does Not Apply

Stateless workloads with no prefix reuse. If every request has a unique prompt that never repeats, prefix caching consumes memory and returns nothing. Some translation and one shot generation workloads fall here.

Workloads where the prefix changes by even one token between requests. Prefix caching is byte exact; a different timestamp or a different user ID injected near the front of the prompt breaks the cache. The fix is to move volatile fields to the tail of the prompt where they do not poison the cached prefix.

Memory constrained setups where the KV cache and prefix cache together push the configuration into memory pressure. On a 64GB M4 Max, prefix caching on a 70B model at long context is tight; on a 128GB M5 Max it is comfortable.

Working with Contra Collective

We build production AI infrastructure for teams running local inference on Apple Silicon. If you are sizing a serving setup for an agent loop, code review pipeline, or document Q&A workload on M5 Max or M5 Ultra and want a prefix cache configuration sized against your actual prompt distribution and concurrency target, we can run the test and produce a sized recommendation. Reach out via the Contra Collective contact page.

FAQ

Q: How does MLX prefix caching compare to llama.cpp prompt caching? A: llama.cpp ships a session based prompt cache that saves and loads a single conversation's KV cache to disk. It is useful but it does not support concurrent multi prefix caching in the way the MLX 0.21 server does. For single user, single session workloads the two are roughly equivalent. For batched serving with multiple concurrent prefixes, MLX 0.21 is materially ahead.

Q: Does prefix caching work with KV cache quantization? A: Yes. The MLX 0.21 prefix cache stores the KV cache at whatever quantization the runtime is configured for. Q8 KV with prefix caching is the default we recommend for production, with Q4 KV reserved for memory constrained setups that tolerate the quality hit.

Q: How sensitive is the cache to prompt formatting differences? A: Byte exact match. A different whitespace or a different timestamp at the front of the prompt breaks the cache. Move volatile fields (timestamps, request IDs, user IDs) to the tail of the prompt to preserve the cached prefix.

Q: Can I warm the cache at startup? A: Yes. The MLX 0.21 server exposes a /cache/preload endpoint that takes a list of prefixes and prefills them into the cache before the first user request. We use this for agent loops with known system prompts to eliminate the cold start prefill on the first request.

Q: What is the right cache size for a workgroup serving setup? A: For a Llama 3.3 70B Q4 weights, Q8 KV setup on a 128GB M5 Max with 5 to 10 concurrent users, 64K to 128K prefix cache is the sweet spot. Below 64K the cache thrashes; above 128K the memory budget gets tight with batched serving.