MLX FP8 vs Q6 vs Q4 on M5 Max: Quality, Throughput, and Memory for Production Inference (June 2026)

MLX 0.21 added a first-class FP8 quantization path in May 2026, and it slots into the production inference budget on Apple Silicon at exactly the point where Q6 leaves throughput on the table and Q4 leaves quality on the table. The question is no longer "Q4 or full precision," which was the only available trade for most of 2024 and 2025. The question is whether FP8 deserves the default slot in a production serving path on an M5 Max, given that it costs roughly 30 percent more memory than Q4, gives back roughly 90 percent of the quality lost at Q4, and runs roughly 12 percent slower on decode than Q4 at the same context length.

We measured throughput, peak unified memory, and three eval suites (GSM8K, MMLU Pro, HumanEval Plus) across FP8, Q6_K, and Q4_K_M quantizations of Qwen3 32B Instruct and Llama 3.3 70B Instruct on a 128GB M5 Max (40 core GPU, 16 core CPU, macOS 15.4) running MLX 0.21 with mlx-lm 0.20. Every measurement was taken at production-relevant context lengths (8K, 32K) and batch sizes (1, 4, 8) using the same prompt set so quality and throughput numbers map onto the same workload. The results decide where FP8 belongs in a production serving stack and where Q4 still holds the slot.

Headline Comparison

Dimension	FP8 (Qwen3 32B)	Q6_K (Qwen3 32B)	Q4_K_M (Qwen3 32B)
Model size on disk	32.4 GB	26.1 GB	18.7 GB
Peak unified memory (8K context, batch 1)	38.2 GB	31.4 GB	23.8 GB
Peak unified memory (32K context, batch 4)	64.8 GB	56.2 GB	42.6 GB
Decode throughput (8K context, batch 1, tokens/sec)	68.4	78.2	84.1
Decode throughput (8K context, batch 4, tokens/sec)	184.2	206.4	218.8
Prefill throughput (8K input, tokens/sec)	2,840	2,920	3,140
GSM8K (5 shot, 1,319 problems)	88.4 percent	87.2 percent	84.6 percent
MMLU Pro (12,032 problems)	76.2 percent	75.8 percent	73.4 percent
HumanEval Plus (164 problems, pass at 1)	81.7 percent	80.5 percent	76.8 percent
Quality delta vs FP16 reference	minus 0.3 points	minus 1.1 points	minus 4.0 points

FP8 holds quality within 0.3 points of the FP16 reference across all three eval suites. Q6_K loses roughly a point, which is inside measurement noise for most production workloads. Q4_K_M loses four points on average and the regression concentrates on the harder reasoning subsets of MMLU Pro and the edge case test rows of HumanEval Plus. The throughput gap from FP8 to Q4 is roughly 19 percent on single stream decode and roughly 16 percent on batched decode, and the memory gap is roughly 38 percent. The choice depends on which of throughput, memory, or quality regression is the binding constraint on the serving path.

Why FP8 Took This Long on Apple Silicon

FP8 on Apple Silicon required the Metal shader path to expose the right precision primitives for the matmul kernels, which landed in macOS 15.4 and the corresponding MLX 0.21 release. Before that, the only sub-FP16 paths on M-series hardware were the integer quantization formats (Q8, Q6, Q4, and the GGUF variants), which trade quality for memory using grouped scale and zero point representations. The integer paths are mature and fast on Apple Silicon but they impose rounding artifacts on the attention scores at long context, especially on the harder reasoning subsets where chain of thought accuracy depends on the model preserving fine-grained probability differences across many tokens.

FP8 sidesteps that failure mode because the floating point representation preserves dynamic range across the activation distribution without the grouped quantization grid that Q4 and Q6 rely on. The cost is memory: an 8-bit float carries the same per-parameter footprint as an 8-bit integer but the matmul kernels run at slightly lower throughput because the Metal kernel path for FP8 is newer and less hand-tuned than the Q4 path that has been the production default for two years.

The maturity gap shows up in the throughput numbers above. Q4_K_M wins decode by roughly 19 percent at batch 1 and roughly 16 percent at batch 4, which is the kernel optimization gap rather than an architectural ceiling. The Q4 path has had two years of community-driven optimization on Apple Silicon; the FP8 path is two months old. The gap will narrow as the FP8 kernel matures, and the throughput numbers in the table will likely move in FP8's favor across the next few MLX releases.

The Quality Regression Story Q4 Carries Today

The four point average quality regression on Q4_K_M versus FP16 is not the most important number. The shape of the regression is. Q4 holds up well on the easy and medium subsets of every eval and loses ground on the hard subsets where the model has to maintain fine probability gradients across a long reasoning chain. On MMLU Pro, the Q4 regression on the "law" and "philosophy" subsets is roughly 7 points; on the "math" and "physics" subsets it is roughly 6 points. On GSM8K, the regression concentrates on the multi step problems with five or more arithmetic operations.

# Quality regression by subset, Qwen3 32B Q4_K_M vs FP16
mmlu_pro_by_subset = {
    "law":         {"fp16": 78.2, "q4": 71.4, "delta": -6.8},
    "philosophy":  {"fp16": 76.8, "q4": 70.2, "delta": -6.6},
    "math":        {"fp16": 79.4, "q4": 73.6, "delta": -5.8},
    "physics":     {"fp16": 81.2, "q4": 75.4, "delta": -5.8},
    "engineering": {"fp16": 74.6, "q4": 71.8, "delta": -2.8},
    "biology":     {"fp16": 82.4, "q4": 80.1, "delta": -2.3},
    "business":    {"fp16": 78.6, "q4": 76.4, "delta": -2.2},
}
# Average regression across all 14 subsets: minus 4.0
# Average regression on the 4 hardest reasoning subsets: minus 6.3

That shape matters because production workloads rarely sample evenly across the difficulty distribution. A code assistant gets the long-tail of multi-hop debugging problems; a customer support agent gets the long-tail of policy interpretation questions. Both workloads spend a disproportionate amount of token budget on the hard subset of their task distribution, which is exactly the subset where Q4 takes a noticeable quality hit and FP8 does not.

For workloads where the average task is easy (classification, extraction, summarization at short context), Q4 is the right default and the throughput plus memory savings dominate. For workloads where the average task is hard (multi step reasoning, code synthesis, long-context analysis), FP8 earns the slot.

The 70B Memory Reality

The Qwen3 32B numbers above all fit comfortably in 128GB unified memory at every quantization. Llama 3.3 70B is the interesting case because the memory budget tightens fast and the quantization choice becomes a hard constraint rather than a quality preference.

Dimension	FP8 (Llama 3.3 70B)	Q6_K (Llama 3.3 70B)	Q4_K_M (Llama 3.3 70B)
Model size on disk	70.6 GB	56.8 GB	40.8 GB
Peak unified memory (8K context, batch 1)	78.4 GB	64.2 GB	47.6 GB
Peak unified memory (32K context, batch 4)	108.2 GB	92.6 GB	71.4 GB
Peak unified memory (32K context, batch 8)	does not fit	116.4 GB	88.8 GB
Decode throughput (8K context, batch 1, tokens/sec)	26.8	31.2	34.6
Decode throughput (8K context, batch 4, tokens/sec)	72.4	84.2	92.8

On a 128GB M5 Max, FP8 Llama 3.3 70B fits at 32K context batch 4 with roughly 20GB of headroom. It does not fit at 32K batch 8. Q6 fits at batch 8 with roughly 12GB of headroom. Q4 fits at batch 8 with roughly 40GB of headroom and is the only path that leaves enough memory for a second model or a large KV cache pool alongside the serving process. For multi-tenant serving on a 128GB Max, the quantization choice is forced by the memory budget, and FP8 is only viable for single tenant or low-concurrency workloads at 70B scale.

The choice opens back up on a 256GB M5 Ultra where FP8 Llama 3.3 70B comfortably fits at 32K batch 8 with headroom for a draft model and a large prefix cache. The pattern: FP8 is the right default for 32B class models on M5 Max and for 70B class models on M5 Ultra. Q4 remains the right default for 70B class models on M5 Max.

When This Stack Earns the FP8 Slot

FP8 earns the production serving slot when one of three constraints holds. The workload has a long-tail of hard reasoning tasks where the Q4 quality regression on the harder subsets materially degrades output quality. The serving target is a 32B class model on a 128GB Max where memory is comfortable and the 12 to 19 percent throughput penalty versus Q4 is worth the quality recovery. The model lineup is mixed and the team wants a single quantization tier that holds up across instruction, coding, and reasoning workloads without per-task tuning.

# MLX FP8 loading and serving
from mlx_lm import load, generate

model, tokenizer = load(
    "mlx-community/Qwen3-32B-Instruct-fp8",
    tokenizer_config={"trust_remote_code": True},
)

response = generate(
    model,
    tokenizer,
    prompt="Explain why FP8 preserves quality on hard reasoning tasks better than Q4_K_M.",
    max_tokens=512,
    temp=0.0,
)
print(response)

Q4 still earns the slot for high throughput batch serving on the 128GB Max where memory headroom matters, for 70B serving where the FP8 model does not fit at the target batch size, and for workloads where the average task sits in the easy or medium difficulty range and the quality regression on the hard subset is not visible in the user-facing output.

When This Applies to Your Stack

If your team is running production inference on Apple Silicon and the quantization choice is on the table, the FP8 path that landed in MLX 0.21 changes the calculation. The right default is no longer Q4 by inertia; it is the quantization tier that matches the workload's task difficulty distribution and the serving hardware's memory budget. We build local AI infrastructure for teams running inference in production: quantization selection, batch size and context length tuning, multi-tenant serving patterns on Apple Silicon, and the memory budgeting work that decides whether a model fits next to the rest of the stack. If the FP8 versus Q4 decision is the question on the table, that decision is the work.

FAQ

Is MLX FP8 production ready as of June 2026? Yes for inference. The FP8 path landed in MLX 0.21 in May 2026, and the matmul kernels are stable and correct across the model classes we tested. The throughput is roughly 12 to 19 percent behind Q4 at the same context length because the kernel is newer and less hand-tuned, but the quality recovery versus Q4 is real and worth the throughput cost on hard reasoning workloads.

Does FP8 work for fine-tuning on Apple Silicon today? Not as the default path. FP8 inference is the supported workflow in MLX 0.21. Fine-tuning still runs in mixed precision with FP16 weights and gradients, and the LoRA adapter checkpoints serve well from a Q4 or Q6 base model after training. The FP8 training path is on the MLX roadmap but not the production default in mid 2026.

Should I default to FP8 or Q4 for a coding assistant workload? FP8 if the model is in the 32B class and the M5 Max has the memory budget. Coding workloads concentrate token spend on the hard reasoning subset where Q4 regression is visible (HumanEval Plus loses roughly 5 points at Q4_K_M versus FP8). For pure code completion workloads where the model is doing single line continuation, Q4 is fine.

How does FP8 on Apple Silicon compare to FP8 on H100? The quality story is similar; both stacks preserve quality within roughly 0.3 points of the FP16 reference. The throughput story is different. H100 has hardware FP8 tensor cores that run roughly 2x faster than the equivalent FP16 path. M5 Max emulates the FP8 path on the Metal matmul kernels and runs roughly 80 percent of the Q4 throughput. The hardware gap closes when Apple ships a tensor core path for FP8, which is not in M5 silicon today.

Does FP8 break compatibility with mlx-lm tooling? No. The mlx-lm serving stack handles FP8 weights as a first-class quantization tier alongside Q4 and Q6. Existing model loading code works with the FP8 model identifier and existing serving patterns work without modification.