mlx-lm vs llama.cpp on M5 Max: Prefill, Decode, and Batch Scaling for 70B (2026)

Most Apple Silicon local inference benchmarks publish a single decode number for an 8B parameter model and stop. That works for a demo and falls apart the moment you try to put MLX or llama.cpp behind a real backend. The interesting question is what prefill latency, decode throughput, and batch size scaling actually look like on a 70B class model under the two runtimes most teams pick between in mid 2026: mlx-lm and llama.cpp with the Metal backend.

We ran the same Llama 3.3 70B Instruct weights on a M5 Max 16 inch MacBook Pro (64 GB unified memory, 16 inch chassis with the larger thermal envelope) under both runtimes. Quantization was matched as closely as the formats allow: 4 bit MLX (group size 64) for mlx-lm, Q4_K_M GGUF for llama.cpp. The workload was a mix of prefill heavy (8K and 16K input tokens, short generations) and decode heavy (short prompts, 1K and 2K generations) requests, run at batch sizes from 1 to 8. Numbers below are medians from a 200 request sample per configuration.

Headline Numbers

Workload	Batch	mlx-lm tok/s	llama.cpp tok/s	mlx-lm prefill (s)	llama.cpp prefill (s)	Winner
1K in / 1K out	1	22.8	21.4	1.9	2.1	mlx-lm (modest)
1K in / 1K out	4	41.6	38.2	2.4	2.7	mlx-lm
1K in / 1K out	8	52.1	44.6	3.1	3.6	mlx-lm
8K in / 512 out	1	18.4	19.1	9.6	7.4	llama.cpp (prefill)
8K in / 512 out	4	28.7	26.4	14.8	12.1	llama.cpp prefill, mlx-lm decode
16K in / 512 out	1	14.2	15.6	22.1	16.8	llama.cpp (clear)
16K in / 512 out	4	19.8	18.4	32.4	26.7	mixed

The pattern is not "one runtime wins." It is two runtimes optimized for different parts of the inference loop, and the right pick depends on whether your workload is prefill bound or decode bound.

Prefill: llama.cpp Wins When Inputs Are Long

The single most consistent result across our tests is that llama.cpp is faster at prefill on long contexts, and the gap widens as the input length grows. At 1K input tokens the two runtimes are within 10 percent. At 8K the gap is 20 to 30 percent. At 16K llama.cpp finishes prefill 24 percent faster on a single request and 18 percent faster in a batch of 4.

The reason is mostly attention kernel maturity. The llama.cpp Metal backend has been tuned aggressively for prefill since the introduction of Flash Attention in the GGML stack, and the kernel batches the query-key-value matmul across the sequence dimension efficiently. mlx-lm uses MLX's native attention path which has improved substantially through 2025 and 2026 but still trails on the long sequence case. If your workload is RAG over a 12K to 16K context window or any agent loop that stuffs tool call history into the prompt, llama.cpp will get you to first token faster.

This matters more than people think. If you are running an agent that submits a 12K token prompt every turn and generates 300 tokens of tool call output, prefill is 70 to 80 percent of the wall clock budget per turn. A 6 second prefill instead of an 8 second prefill is a 25 percent latency reduction on every turn.

Decode: mlx-lm Wins When Generations Are Long

The picture flips for decode. On the 1K in / 1K out workload at batch 8, mlx-lm runs 52.1 tokens per second compared to llama.cpp at 44.6. That is a 17 percent advantage and it grows with batch size up to the point where the M5 Max memory bandwidth saturates (around batch 8 on a 70B 4 bit model). At batch 1 the gap is small (22.8 vs 21.4 tok/s, roughly 6 percent) but the trend line is clear.

The reason is also kernel related. MLX's decode path uses a fused matmul-softmax-matmul kernel for the attention block that is well tuned for the small sequence length per token that decode produces. The kernel also keeps more of the working set in the AMX coprocessor's register file, which reduces the round trip cost to unified memory. For pure text generation where prefill is short and the model needs to emit a long completion, mlx-lm is the faster runtime.

Batch Scaling: The M5 Max Memory Bandwidth Wall

Both runtimes scale with batch size, but they hit the unified memory bandwidth wall at different points. The M5 Max ships with around 546 GB/s of memory bandwidth (per Apple's published specifications, validated against our memtest runs). A 4 bit quantized 70B model with KV cache is bandwidth bound during decode regardless of runtime.

In our tests, decode throughput scales nearly linearly from batch 1 to batch 4. Beyond that:

• mlx-lm keeps scaling to batch 8 with a 25 percent drop in per request throughput compared to batch 4 (52.1 / 8 = 6.5 tok/s/request vs 41.6 / 4 = 10.4 tok/s/request).
• llama.cpp flattens earlier. Batch 8 only delivers 44.6 tok/s total, barely above batch 6 (40.8 tok/s) in our supplementary runs.

The practical implication is that mlx-lm is the better choice if you can pack batches of 4 to 8 requests through the runtime, which is realistic for a chat backend with even moderate concurrency. llama.cpp is the better choice for single request latency on long context prefill.

Memory Footprint and KV Cache

A 70B model in 4 bit quantization sits at roughly 38 to 40 GB of weights. The KV cache adds 2 to 3 GB per concurrent sequence at 8K context, 4 to 6 GB at 16K context. On a 64 GB M5 Max that leaves you with:

• 4 concurrent 16K context sessions: 38 + (4 × 5.5) = 60 GB. Tight, will OOM on edge cases.
• 4 concurrent 8K context sessions: 38 + (4 × 3) = 50 GB. Comfortable.
• 8 concurrent 4K context sessions: 38 + (8 × 1.8) = 52.4 GB. Comfortable.

mlx-lm and llama.cpp both honor explicit --max-context or kv_cache_size flags and will reject requests that would overflow the configured cache. mlx-lm has slightly lower KV cache overhead in our tests (around 8 percent less memory for the same configuration), likely because of more aggressive cache layout. If you are pushing a 64 GB box to its limits, that 8 percent buys you another concurrent session.

Server Mode and Production Patterns

Both runtimes ship an OpenAI compatible HTTP server (mlx_lm.server and llama-server). The interfaces are functionally equivalent. Where they diverge in production:

Streaming consistency. llama-server has more battle tested streaming token output. We have seen mlx-lm server occasionally emit a final chunk without the data: [DONE] terminator under specific cancellation patterns. Workaround is a client side timeout.

Continuous batching. llama.cpp has continuous batching merged and stable. mlx-lm has it in 0.20+ but the implementation is newer. If you need predictable behavior under variable request arrival times right now, llama.cpp is the safer pick.

Cancellation. mlx-lm's request cancellation is cleaner. Cancelling a request mid decode frees the KV cache slot immediately. llama.cpp is more conservative and may hold the slot through the end of the batch step.

For a deeper dive on the server side specifically see our mlx-lm server vs llama-server comparison which goes into request routing, OpenAI API conformance, and operational characteristics.

When to Pick Which

If your workload looks like a chat assistant with average prompts under 4K tokens and long completions, pick mlx-lm. The decode advantage compounds across every response and the batch scaling is better.

If your workload looks like RAG over long contexts or an agent that submits 8K to 16K token prompts every turn, pick llama.cpp. The prefill advantage dominates the wall clock budget and the kernel maturity is real.

If your workload is mixed, run both in parallel behind a router. We have shipped this pattern for clients: a small classifier (or just request metadata) routes long input requests to llama.cpp and decode heavy requests to mlx-lm. Both runtimes pin to the same GPU device, so the only overhead is two warm processes (about 80 GB combined for a 70B model loaded into both, which means you need a 96 GB or 128 GB Mac Studio for that pattern).

What This Means for Your Stack

If you are evaluating local inference on Apple Silicon for an enterprise workload (on premises retail, healthcare, financial services where data residency matters), the runtime choice has direct business impact. A 25 percent prefill latency reduction on agent loops translates to fewer concurrent box hours billed when you scale to a Mac Studio cluster. A 17 percent decode advantage on chat workloads translates to lower steady state hardware count for the same SLA. These numbers compound, and the cluster scaling math we cover in our Apple Silicon sustained throughput analysis gets ugly fast when you pick the wrong runtime for your traffic shape.

The other axis worth thinking about is operator experience. mlx-lm is a Python first ecosystem. llama.cpp is C++ with Python and Node bindings. If your team is comfortable with native dependencies and wants the C++ performance ceiling, llama.cpp wins. If your team will live in Python and wants to extend the runtime (custom samplers, custom logit processors, structured output), mlx-lm is friendlier.

If you are building a serious local inference backend on Apple Silicon and want help benchmarking runtime trade offs against your actual traffic patterns, that is exactly the kind of AI infrastructure work Contra Collective does. We have shipped on premises inference clusters using both runtimes and the configuration decisions matter more than the marketing benchmarks suggest. See our llama.cpp vs MLX vs Ollama vs vLLM full comparison for the broader runtime landscape.

FAQ

Is mlx-lm always faster than llama.cpp on Apple Silicon? No. mlx-lm tends to win on decode throughput for short to medium context workloads. llama.cpp tends to win on prefill for long context inputs. The right runtime depends on your prompt and generation length distribution.

Can I run both on the same Mac? Yes. We have run both runtimes concurrently on a 128 GB Mac Studio with two copies of a 70B model loaded. The bottleneck becomes memory bandwidth contention, which is manageable if you stagger workloads.

What batch size should I configure on M5 Max for Llama 3.3 70B? For chat workloads, batch 4 to 8 with mlx-lm is the sweet spot. For long context RAG, batch 1 to 4 with llama.cpp keeps prefill latency predictable.

Does the M5 Ultra change the picture? Yes. The M5 Ultra's higher memory bandwidth shifts the bandwidth wall later, which favors mlx-lm even more at higher batch sizes. We will publish dedicated M5 Ultra numbers separately.

Are these results applicable to other 70B class models? Yes for similar architectures (Llama 3.x, Qwen 2.5 72B, DeepSeek V2 base models). Mixture of Experts models (Mixtral, DeepSeek V3) have different bandwidth characteristics and need separate testing.