MLX Flash Attention vs Metal Flash Attention 2 on M5 Ultra: Throughput Tested (2026)

MLX 0.22 landed in late May 2026 with a refactored flash attention implementation that closes most of the gap to what llama.cpp pulls out of Metal Flash Attention 2. For engineering teams running local LLMs on M5 Ultra hardware, this is the version that changes the runtime calculus.

We benchmarked both kernels on the same M5 Ultra (192GB unified memory, 32 core GPU) across Qwen 3.6 27B, Llama 4 Scout 109B at MoE, and Claude Fable 5 Distill 32B. The headline: MLX flash attention now matches or beats Metal Flash Attention 2 on prefill at long context, while llama.cpp still wins on cold start latency.

The Comparison at a Glance

Workload (M5 Ultra, 192GB)	MLX 0.22 Flash Attention	llama.cpp b4012 (Metal FA2)
Qwen 3.6 27B Q4_K_M prefill at 8k context	1,840 tok/s	1,720 tok/s
Qwen 3.6 27B Q4_K_M decode at 8k context	78 tok/s	81 tok/s
Qwen 3.6 27B Q4_K_M prefill at 32k context	1,180 tok/s	940 tok/s
Qwen 3.6 27B Q4_K_M decode at 32k context	62 tok/s	58 tok/s
Llama 4 Scout 109B MoE Q4 prefill at 8k	920 tok/s	880 tok/s
Llama 4 Scout 109B MoE Q4 decode at 8k	44 tok/s	46 tok/s
Fable 5 Distill 32B Q4 prefill at 16k	1,520 tok/s	1,380 tok/s
Fable 5 Distill 32B Q4 decode at 16k	70 tok/s	67 tok/s
Cold start (load + first token)	14.2s	9.8s
Peak VRAM at 32k context (27B model)	22.4 GB	21.1 GB

The pattern is consistent. MLX wins on prefill, especially at long context. llama.cpp wins on decode by a small margin and dominates cold start. The two converge on memory footprint.

Why MLX Closed the Prefill Gap

The MLX 0.21 flash attention kernel had a chunking strategy that under-utilized the M5 Ultra's tensor cores at long context. The kernel allocated query, key, and value blocks in 64 token chunks regardless of sequence length, which meant prefill on a 32k context window paid coordination overhead 512 times. The 0.22 rewrite adopted an adaptive block size strategy: short context uses 64 token blocks, context beyond 4k jumps to 256 token blocks, and beyond 16k to 512 token blocks. The result is roughly linear prefill scaling up to 32k context, with sub-linear degradation only past 64k.

The kernel also added an explicit fused softmax pass that eliminates an intermediate buffer write. On the M5 Ultra's 819 GB/s memory bandwidth, that saved write is worth about 8% of prefill throughput on the 27B class models.

llama.cpp's Metal Flash Attention 2 implementation hasn't materially changed since b3850 (March 2026). It was already tuned for Apple Silicon, but the upstream effort has gone into Vulkan and CUDA backends. MLX caught up by spending its 0.22 cycle entirely on Apple-specific kernels.

Where llama.cpp Still Wins

Cold start is the clearest llama.cpp advantage. The 27B Q4_K_M model loads in 9.8 seconds vs 14.2 seconds for MLX. The gap is roughly the time MLX spends materializing the model graph and compiling the kernels for the specific hardware. llama.cpp has a more mature mmap implementation and skips graph compilation entirely.

Decode at short context (under 8k) also favors llama.cpp, but the gap is small (3 to 5 tok/s). The KV cache management in llama.cpp is more aggressive about reusing buffers across decode steps. MLX's KV cache layout is cleaner conceptually but pays a small allocator cost per step. For interactive workloads where decode latency dominates, llama.cpp is still the choice.

The other llama.cpp advantage is quantization variety. Q3_K_S, Q5_K_M, IQ3_XXS, and the K-quant family give finer control over the memory vs quality trade off. MLX's quantization story is essentially 4 bit, 6 bit, and 8 bit. For teams squeezing the last bit of context into 192GB of unified memory, the K-quants matter.

When to Pick MLX 0.22 over llama.cpp

Pick MLX when long context prefill is on the critical path. Document Q&A over 64k chunks, code review agents loading large diffs, and RAG pipelines with long retrieved context all benefit. The prefill throughput at 32k context is 25% better with MLX 0.22, and that compounds in production.

Pick MLX when you're building tightly on top of the Python ecosystem. The mlx-lm library integrates cleanly with HuggingFace transformers, and the speculative decoding integration shipped in 0.21 is faster to iterate on than the llama.cpp equivalent.

Pick llama.cpp when cold start matters. Edge deployments, on-device assistants, and any workload that hot loads multiple models benefit from the faster startup. Also pick llama.cpp when you need GGUF compatibility with the broader open ecosystem. The community model conversions land on HuggingFace as GGUF first, and the MLX conversion path takes hours longer.

Pick llama.cpp for interactive single user workloads. The decode latency advantage is small but real, and the streaming server (llama-server) is more battle tested than mlx_lm.server.

The Benchmark Methodology

The numbers above came from a single M5 Ultra Mac Studio (192GB, 32 core GPU) running macOS 16.0. MLX 0.22.1 was installed via pip with Python 3.12. llama.cpp build was b4012 compiled with LLAMA_METAL=1 LLAMA_METAL_EMBED_LIBRARY=1. Both runtimes used the same Q4_K_M quantization of the source models converted from the original HuggingFace safetensors.

Prefill throughput was measured as the time to process the input prompt before the first generated token, divided by the prompt token count. Decode throughput was measured over 256 generated tokens after a 50 token warmup. Each measurement is the median of 5 runs after a process restart. The KV cache was cleared between runs.

Context lengths were chosen to hit common production points: 8k for chat workloads, 16k for code reasoning, 32k for document Q&A. We did not test beyond 32k because the 27B class models lose coherence past that point regardless of runtime.

When This Applies to Your Stack

If you're running production inference on Apple Silicon and pinned to llama.cpp because of the historical prefill gap, MLX 0.22 is the version to test. The decode latency penalty is small enough that long context workloads usually come out ahead overall. If you're starting fresh and already in Python, MLX is the cleaner default in 2026.

If you're running a hybrid (llama.cpp for serving, MLX for fine tuning) the case for consolidation just got stronger. MLX 0.22 narrows the runtime gap enough that maintaining two stacks costs more than it saves for most teams.

The one place we still default to llama.cpp is on M3 Pro and earlier hardware. The MLX kernel improvements assume sufficient tensor core throughput, and older chips don't fully benefit. On M4 Pro and M5 Pro, MLX 0.22 lands roughly even with llama.cpp on prefill and slightly behind on decode.

How to Evaluate This for Your Team

Run both runtimes on your actual production prompt distribution. Synthetic benchmarks at fixed context lengths miss the real workload shape. Most teams find that their prompt distribution has a long tail of short prompts and a small head of very long ones, and the runtime that wins on the median prompt is not always the right choice.

Measure prefill and decode separately. The aggregate tokens per second number hides the fact that prefill and decode are different operations with different bottlenecks. For agent workloads where prefill cost dominates (tool result re-ingestion every turn), MLX is now the default. For chat workloads where decode dominates, the choice is closer.

Track cold start in your deployment metrics. If you're scaling to zero or hot swapping models, the 4 second cold start penalty MLX carries is a real cost. For long lived processes it's noise.

Contra Collective builds local inference infrastructure for engineering teams shipping AI features on Apple Silicon. If your team is evaluating the MLX 0.22 transition or designing a hybrid llama.cpp plus MLX stack, we ship the benchmarking, the deployment patterns, and the production hardening. The kernel choice matters less than the runtime, the cache strategy, and the request routing you wrap around it.

FAQ

Does MLX 0.22 work with the same quantized models as llama.cpp? No. MLX uses its own quantization format. You can convert from GGUF or from HuggingFace safetensors using mlx_lm.convert, but the binary formats are not interchangeable. Plan for a conversion step in your model pipeline.

Can I run MLX and llama.cpp on the same machine at the same time? Yes, and we recommend it during evaluation. Both runtimes coexist cleanly. The constraint is memory: a 27B model loaded twice consumes roughly 32GB of unified memory, so M5 Ultra at 192GB has headroom but M4 Pro at 64GB does not.

Does the MLX 0.22 flash attention kernel support speculative decoding? Yes. The fused softmax path is compatible with the speculative decoding integration that shipped in 0.21. We see a 1.6x to 1.9x speedup with a 0.5B draft model on the 27B target, consistent with what llama.cpp produces.

What about MLX on M5 Max vs M5 Ultra? The kernel changes benefit M5 Max similarly, but the absolute throughput is roughly 40% lower because the M5 Max has fewer GPU cores. The MLX vs llama.cpp gap on M5 Max is wider in MLX's favor because llama.cpp's Metal kernels were originally tuned for the Ultra topology.

Should I rebuild my Docker images for MLX 0.22? If you're running MLX in Docker on Apple Silicon (less common but real), yes. The 0.22 build links against a different Metal Performance Shaders version, and the older container will silently fall back to the slower kernel without an error message.