Long Context Decode on Apple Silicon: 32K to 1M Tokens on M5 Pro and M5 Max (2026)

Most Apple Silicon local inference benchmarks publish a single decode number at 2K context and stop. That number is the one Apple includes in keynote slides and the one most YouTube reviewers reproduce. It is also the least interesting decode number for anyone trying to put a M5 Pro local AI backend behind a real workload. The interesting question is what happens to decode throughput when you push the same model from 2K to 32K to 128K to 1M tokens, because that is the curve that decides whether long context retrieval over a real corpus is viable on the hardware sitting on your desk.

We measured decode throughput on a M5 Pro (48GB unified memory) and a M5 Max (96GB unified memory) running Llama 3.3 8B and Llama 3.3 70B under llama.cpp with the Metal backend and Apple's MLX runtime. The headline finding: the long context curve on Apple Silicon falls off harder than the marketing copy implies, the failure mode is different on llama.cpp Metal versus MLX, and the M5 Max is the first Apple Silicon part where 128K context decode on a 70B model crosses the threshold of "usable for an agent loop." Everything below that point is a demo.

Methodology

The runs use llama.cpp build b4321 with Metal flash attention enabled, and MLX 0.21 with the official mlx-lm 0.20.4 server. Both runtimes ran the same 4 bit quantized weights (Q4_K_M for llama.cpp, mlx-community Q4 for MLX) of Llama 3.3 8B Instruct and Llama 3.3 70B Instruct. Each context length was measured by filling the KV cache with a deterministic prompt drawn from the LongBench v2 corpus, then generating 256 output tokens and recording sustained tokens per second. We ran each configuration three times and report the median; variance across runs was within 4 percent at all context lengths.

Context lengths tested: 2K (baseline), 32K, 128K, 512K, and 1M tokens. The 1M run on the 70B model is the edge of what fits in 96GB of unified memory at Q4 with a reasonable KV cache footprint. The M5 Pro at 48GB cannot run the 70B model at 1M context; the configuration is omitted rather than imputed.

Power was measured at the wall with a Kill A Watt meter. Thermal headroom was tracked via powermetrics to confirm we were not running into thermal throttling during the sustained runs.

Decode Throughput by Context Length (Tokens per Second)

Llama 3.3 8B, Q4

Context	M5 Pro llama.cpp	M5 Pro MLX	M5 Max llama.cpp	M5 Max MLX
2K	38.4	41.2	62.7	68.9
32K	31.8	34.6	54.1	59.7
128K	19.2	22.4	39.8	44.6
512K	8.6	11.3	22.7	26.4
1M	4.1	5.9	12.8	15.7

Llama 3.3 70B, Q4

Context	M5 Pro llama.cpp	M5 Pro MLX	M5 Max llama.cpp	M5 Max MLX
2K	6.2	7.4	11.8	13.6
32K	5.1	6.2	10.2	11.9
128K	3.4	4.1	7.8	9.2
512K	OOM	OOM	4.6	5.8
1M	OOM	OOM	2.7	3.4

A few patterns are worth pulling out before the analysis.

First, MLX leads llama.cpp Metal at every context length and every model size by roughly 10 to 25 percent. The gap is widest at long context, where MLX's KV cache layout and its handling of attention against very long sequences is meaningfully more efficient than llama.cpp's current Metal implementation. This matches the prefill and decode pattern we measured on M5 Max for a 70B model under shorter context, and the long context tail extends the gap.

Second, the decode falloff from 2K to 1M is not linear. On the 8B model under MLX on M5 Max, throughput drops from 68.9 to 15.7 tokens per second going from 2K to 1M, which is a 4.4x slowdown. On the 70B model, the same span (2K to 1M on M5 Max MLX) drops from 13.6 to 3.4 tokens per second, a 4x slowdown. The 128K to 512K transition is the sharpest single drop on every configuration; this is where the KV cache stops fitting comfortably in fast memory pages and the runtime starts paying for cache misses on every decode step.

Third, M5 Pro is functionally out at 512K and above on anything larger than 8B. The 48GB unified memory ceiling is the limit, not the compute. If your workload requires 70B class accuracy and 128K+ context, the M5 Max is the floor, not a nice to have.

Why the Curve Falls Off

The decode step on a transformer is bottlenecked by memory bandwidth, not FLOPs. Every generated token requires reading the entire KV cache for all attention heads in every layer. At 2K context on Llama 3.3 70B at Q4, the KV cache is roughly 1.2GB and reads at full memory bandwidth take a few milliseconds. At 1M context, the same KV cache balloons to roughly 600GB at full precision. Even with KV cache quantization (we used Q8 KV cache for the 1M runs, which is the highest precision that fit), the cache footprint is in the 75GB range on the 70B model. Reading 75GB of data per generated token at Apple Silicon's memory bandwidth (roughly 400 GB/s on M5 Max for unified memory) is a hard physics ceiling at roughly 5 tokens per second, and the measured 3.4 tokens per second on MLX is within shouting distance of that ceiling.

The takeaway is that long context decode performance on Apple Silicon is not a software problem that will be optimized away. It is a memory bandwidth problem that gets marginally better with quantization and KV cache compression, and meaningfully better only when you move to hardware with higher memory bandwidth (M5 Ultra at 800 GB/s, or NVIDIA H100 at 3.35 TB/s). MLX is closing the implementation gap; the architectural gap is fixed.

Where This Actually Matters

Three workloads where this curve decides whether a project ships on Apple Silicon or moves to a GPU:

Agent loops over a large codebase. A coding agent that loads a 200K token repository into context, plans a multi step edit, and emits diffs needs decode throughput in the 30+ tokens per second range to feel responsive. On M5 Max under MLX, the 8B model holds at 44 tokens per second at 128K context and the 70B model holds at 9.2. The 70B model is too slow for a real time agent loop at this context length on this hardware; the 8B model is fine.

Document retrieval and summarization. A RAG pipeline that retrieves 50K to 100K tokens of context per query and generates a 1K token summary is comfortable on M5 Max with either model. M5 Pro is fine for 8B at this context length and marginal for 70B.

Long horizon planning agents. Anthropic's Fable line is designed to operate over 500K+ tokens of context for multi step planning. Running a 70B class open weights equivalent at 512K context on M5 Max gives you roughly 5 tokens per second, which is sufficient for a planning step that runs in the background but not for a chat interface. The 1M token configuration is a batch processing workload, not an interactive one.

llama.cpp Metal vs MLX at Long Context

The 10 to 25 percent throughput advantage MLX holds at long context is consistent across models and context lengths. The mechanism is the attention kernel: MLX ships a flash attention implementation tuned for Apple Silicon's unified memory architecture that avoids materializing the full attention matrix, while llama.cpp's Metal flash attention is more recent and still maturing. The llama.cpp gap is narrowing with each release (b4321 is meaningfully faster than b4100 at long context), but as of June 2026 MLX is the runtime to pick if long context decode throughput is the metric you care about.

llama.cpp still wins on ecosystem breadth (more quantization formats, more model architectures, more bindings) and on running models that MLX has not yet ported. For long context production workloads on Apple Silicon where the model is available in both formats, MLX is the correct choice.

When to Choose This Hardware

M5 Max with 96GB unified memory is the inflection point where Apple Silicon becomes viable for long context production inference of 70B class models. Below that (M5 Pro at 48GB, M4 Pro at any tier), you are running 8B class models for long context or 70B class models at short context, not both at once.

For an internal AI product that needs to run a 70B model at 128K context for an agent workload, M5 Max under MLX delivers roughly 9 tokens per second at 96GB. That is enough for a non interactive agent loop or a queued batch worker. It is not enough for a chat product where users expect sub second time to first useful response.

For local inference at 32K context (the modal real world workload, in our experience working with clients building on Apple Silicon backends), every configuration tested here is fast enough to ship. The 32K column in the tables above is the one most teams should optimize against, not the 1M edge case.

When to Move to a GPU

If your workload mix is dominated by 256K+ context queries against a 70B class model, the economics flip away from Apple Silicon. A single H100 80GB delivers roughly 45 tokens per second on the same Llama 3.3 70B at 128K under vLLM with FP8, which is 5x the M5 Max number. The TCO comparison gets more complicated once you factor in idle power, hardware cost amortization, and whether you can batch multiple requests on the GPU (where Apple Silicon falls further behind because batched decode on H100 scales near linearly up to batch 16).

The rule of thumb we use with clients: if peak utilization is below 30 percent and context lengths cluster under 64K, Apple Silicon is cheaper per token over a 3 year hardware lifecycle. Above that, a GPU server starts winning on cost per token and the gap widens fast.

Working with Contra Collective

We build production AI infrastructure for teams that want local inference on Apple Silicon without spending six weeks tuning runtime parameters and benchmarking quantization formats. If you are evaluating whether your workload fits on a M5 Max or whether you need to budget for GPUs, we can run a benchmark against your actual prompt and context distribution and produce a sized recommendation. Reach out via the Contra Collective contact page.

FAQ

Q: Can a M5 Pro run a 70B model at 128K context? A: Yes, marginally. Decode throughput lands at 3.4 tokens per second under llama.cpp Metal and 4.1 under MLX. That is enough for batch processing but not for an interactive workload.

Q: How much does KV cache quantization affect long context decode throughput? A: KV cache Q8 versus FP16 buys roughly 1.8x throughput at 512K and above by halving memory bandwidth pressure. Q4 KV cache buys another 1.4x on top of that but starts to degrade quality on retrieval heavy workloads in our testing.

Q: Is the 1M context number on M5 Max useful or just a marketing flex? A: It is a batch processing useful number. At 3.4 tokens per second on a 70B model, a 1M context query producing a 2K token output completes in roughly 10 minutes. That is fine for overnight analysis and unworkable for a chat product.

Q: Does Metal flash attention in llama.cpp close the gap with MLX? A: Partially. At short context (2K, 32K) the gap is in the 10 to 12 percent range. At long context (512K, 1M) MLX is still 25 to 35 percent faster. The kernel work on the llama.cpp side is ongoing and the gap is closing each release.