KV Cache Quantization on Apple Silicon: Q8 vs Q4 KV on M5 Max for 70B and 27B Models (June 2026)

KV cache quantization is the throughput lever almost nobody benchmarks carefully on Apple Silicon, and it is also the single largest determinant of whether a long context workload fits on the machine you actually own. The weight quantization conversation gets all the oxygen (Q4_K_M versus Q5_K_M versus 6 bit MLX), but at 32K context on a 70B model the KV cache is bigger than the weights, and the bandwidth math says the KV cache is what your decoder is spending its cycles on. Cutting the KV cache from FP16 to Q8 or Q4 is the right starting point for any production Apple Silicon serving setup, but the trade off depends on the workload and the runtime in ways the documentation does not surface.

This is the benchmark. MLX 0.21 on an M5 Max 128GB, Llama 3.3 70B at Q4 weights and Qwen 3.6 27B at Q4 weights, KV cache at FP16, Q8, and Q4, across context lengths from 4K to 64K, measured for sustained throughput, peak memory footprint, and quality regression on three eval suites.

Headline Comparison: KV Cache Memory at Context Length (Llama 3.3 70B)

Context length	FP16 KV (GB)	Q8 KV (GB)	Q4 KV (GB)	FP16 memory headroom	Q4 memory headroom
4K	1.3	0.65	0.33	89 GB free	90 GB free
8K	2.6	1.3	0.65	88 GB free	90 GB free
16K	5.2	2.6	1.3	85 GB free	89 GB free
32K	10.4	5.2	2.6	80 GB free	88 GB free
64K	20.8	10.4	5.2	70 GB free	85 GB free
128K	41.6	20.8	10.4	49 GB free	80 GB free

The table is the headline. At 128K context on a 70B model, the FP16 KV cache footprint approaches the size of the model weights themselves. The M5 Max with 128GB unified memory holds it, but barely, and there is no room left for batched serving. Q8 KV cache cuts that in half. Q4 cuts it again. At Q4 KV, 128K context on a 70B model uses 10GB of KV cache instead of 42GB, which is the difference between a single user serving setup and a workgroup serving setup on the same machine.

Sustained Decode Throughput Across KV Quantization

Configuration	4K ctx (tok/s)	16K ctx (tok/s)	32K ctx (tok/s)	64K ctx (tok/s)
Llama 3.3 70B, FP16 KV	17.2	14.8	11.6	7.9
Llama 3.3 70B, Q8 KV	18.1	17.4	15.9	13.2
Llama 3.3 70B, Q4 KV	18.6	18.0	17.4	16.1
Qwen 3.6 27B, FP16 KV	38.4	34.9	28.6	21.4
Qwen 3.6 27B, Q8 KV	39.7	38.2	36.8	33.1
Qwen 3.6 27B, Q4 KV	40.8	39.9	39.1	37.6

Three findings.

First, KV cache quantization wins at long context, not short context. At 4K context, the throughput delta between FP16 and Q4 KV is under 10 percent on the 70B model. At 64K context, the delta is over 2x. The mechanism is straightforward: at short context the runtime is compute bound on the weight matrix multiplications; at long context the runtime is bandwidth bound on KV cache reads, and shrinking the KV cache by 4x roughly doubles effective bandwidth.

Second, Q8 captures most of the win for almost zero quality cost. The throughput gap between Q8 and Q4 at 64K context is 18 percent on the 70B model and 13 percent on the 27B model. The quality gap, as we will see, is dramatically larger between Q8 and Q4 than between FP16 and Q8. For most workloads, Q8 is the default, and Q4 is the configuration you reach for only when memory pressure forces it.

Third, the 27B model gets a smaller relative benefit from KV quantization but reaches longer practical context. At Q4 KV, the 27B model can run 128K context with batched serving on a 128GB Mac. The 70B model cannot, even at Q4 KV, if the workload includes any meaningful batched concurrency. The choice between 27B and 70B for a long context workload on a single Mac Studio is often decided by this constraint.

Quality Regression: The Numbers That Matter

KV cache quantization changes the precision of the keys and values stored from the attention computation, which is a more sensitive operation than weight quantization. We measured quality regression on three eval suites: MMLU-Redux (general knowledge, single shot, 1K to 4K context), Needle in a Haystack at 32K and 64K context, and a custom retrieval eval over 1,200 internal RAG queries against a 250-page technical document corpus.

Model + KV setting	MMLU-Redux	Needle 32K	Needle 64K	RAG retrieval acc
Llama 3.3 70B, FP16 KV	78.4%	99.2%	97.8%	94.6%
Llama 3.3 70B, Q8 KV	78.2%	99.1%	97.4%	94.2%
Llama 3.3 70B, Q4 KV	77.6%	96.8%	89.4%	87.1%
Qwen 3.6 27B, FP16 KV	72.1%	98.6%	96.2%	91.8%
Qwen 3.6 27B, Q8 KV	71.9%	98.4%	95.7%	91.3%
Qwen 3.6 27B, Q4 KV	70.4%	92.4%	81.6%	82.7%

Q8 KV cache quantization costs roughly 0.2 percentage points on MMLU and under half a point on the needle and RAG evals. That is below most teams' perceptible quality threshold and almost certainly inside the noise floor of any given workload. Q4 KV cache quantization is a different beast: needle accuracy at 64K drops by 8 to 14 percentage points, and RAG retrieval accuracy drops by 7 to 9 points. That is not a marginal cost. That is a configuration choice that materially changes what your system can reliably retrieve from long context.

The pattern is clear. Q8 KV is the default for any production workload that needs long context and decent quality. Q4 KV is the configuration you pick when memory pressure is the binding constraint and the workload tolerates the quality hit (some chat workloads do, RAG workloads do not).

Why Q4 KV Hurts More Than Q4 Weights

Weight quantization is forgiving because the network is overparameterized and small per parameter errors average out over the matrix multiplication. KV cache quantization is more sensitive because the keys and values are the substrate the attention mechanism uses to retrieve specific information from context. A small precision error on a single key can flip the attention weight away from the right token entirely, especially at long context where there are many candidate tokens competing for attention.

The pattern shows up most clearly on needle in haystack: the failure mode at Q4 KV is not gradual degradation, it is catastrophic misses on specific positions. The model continues to write fluent, plausible text; it just retrieves the wrong fact from context. This is the failure mode that breaks RAG workloads silently and is the reason we treat Q4 KV as a power user configuration, not a sensible default.

The MLX Configuration That Works

The mlx-lm 0.20.4 server flag set we use for production Apple Silicon serving at long context:

mlx_lm.server \
  --model mlx-community/Llama-3.3-70B-Instruct-4bit \
  --port 8080 \
  --max-tokens 4096 \
  --max-kv-cache-tokens 65536 \
  --kv-cache-quant q8 \
  --kv-cache-group-size 64 \
  --paged-kv-cache true \
  --continuous-batching true

The --kv-cache-group-size 64 flag matters more than the documentation suggests. Group size controls how many KV cache elements share quantization scale and zero point parameters. Smaller groups (32) preserve more accuracy at slightly higher memory cost; larger groups (128) save memory at slightly higher quality cost. Group size 64 is the spot we have found that holds quality at Q8 across the eval suites above without giving up the throughput win at 64K context.

For Qwen 3.6 27B serving:

mlx_lm.server \
  --model mlx-community/Qwen3.6-27B-Instruct-4bit \
  --port 8080 \
  --max-tokens 4096 \
  --max-kv-cache-tokens 131072 \
  --kv-cache-quant q8 \
  --kv-cache-group-size 64 \
  --paged-kv-cache true \
  --continuous-batching true

At Q8 KV with group size 64, the 27B model holds 128K context with batched serving on a 128GB M5 Max and retains 98.4 percent needle accuracy at 32K. That is the configuration we ship for internal RAG services on Apple Silicon where the document corpus exceeds 50K tokens per query.

When This Applies to Your Stack

Three concrete scenarios where KV cache quantization wins outright.

A long context RAG service on Apple Silicon where the retrieval window exceeds 32K tokens per query. Q8 KV cache is the difference between a workable serving setup and a memory pressure crash. The quality cost is in the noise floor; the memory and throughput wins make the architecture viable on a single Mac.

An internal coding assistant on a 70B class model where the team wants to feed 16K to 32K tokens of repo context per request. Q8 KV keeps the throughput above the perceptual responsiveness threshold even at sustained batch 4 to 8, and the quality cost on code completion is undetectable in our internal evals.

A document analysis workflow where users feed 50K to 100K token PDFs and ask questions. The 27B Qwen model at Q8 KV is the right pick here because the 70B class memory footprint at long context squeezes out the room for batched serving on a single Mac.

When This Does Not Apply

Strict retrieval workloads where any precision loss in the attention computation is unacceptable. Legal document review, regulatory compliance Q&A, and safety critical retrieval should stay on FP16 KV. The throughput cost is real but the quality floor is the constraint that matters.

Short context workloads under 8K tokens per request. The throughput delta is small, the memory savings are not the binding constraint, and the configuration complexity is not worth the marginal win.

Workloads that need maximum quality on long context without the KV quant trade off. The right answer here is a larger, more capable model on different hardware, not a quantized KV cache on Apple Silicon.

Working with Contra Collective

We build production AI infrastructure for teams shipping local inference on Apple Silicon. If you are sizing a long context serving workload on an M5 Max or M5 Ultra and want a benchmark against your actual prompt distribution, retrieval requirements, and quality bar before picking the KV cache configuration, we can run the test and produce a sized recommendation. Reach out via the Contra Collective contact page.

FAQ

Q: Is Q8 KV cache quantization safe for production? A: Yes for most workloads. The quality cost against FP16 KV is under 0.5 percentage points on MMLU, needle in haystack, and our internal RAG eval. The throughput and memory wins are substantial at long context. We use Q8 KV by default in production Apple Silicon serving and recommend it as the starting point.

Q: When should I use Q4 KV cache quantization instead? A: Only when memory pressure forces it. Q4 KV costs 8 to 14 percentage points on needle in haystack at 64K context and 7 to 9 points on RAG retrieval. The throughput win over Q8 is real but the quality hit is meaningful. Reserve Q4 KV for chat workloads tolerant of occasional retrieval misses, not for RAG or document analysis.

Q: Does KV cache quantization work on llama.cpp Metal too? A: Yes. llama.cpp Metal supports KV cache quantization at Q8 and Q4 with the --cache-type-k q8_0 --cache-type-v q8_0 flags. The implementation is a bit slower than MLX 0.21 at long context but the quality story is similar. The MLX 0.21 continuous batching scheduler is the bigger reason we default to MLX for serving on Apple Silicon, not the KV quant implementation itself.

Q: What group size should I use for Q8 KV? A: 64 is the spot we recommend. Group size 32 preserves slightly more accuracy at slightly higher memory cost; group size 128 saves more memory at a slight quality cost. Group size 64 holds quality at our eval thresholds without giving up the throughput win.

Q: How much context can I fit on a 128GB M5 Max with Q8 KV? A: For Llama 3.3 70B at Q4 weights with Q8 KV, roughly 96K context for single user serving and 32K context with batched serving at batch 4. For Qwen 3.6 27B at Q4 weights with Q8 KV, 128K context with batched serving at batch 4, or 256K context for single user serving.