Speculative Decoding on Apple Silicon with MLX: Throughput Gains in 2026

Speculative decoding is the most underused throughput lever in local Apple Silicon inference right now. The technique has existed for two years in the cloud serving stack (vLLM, TensorRT LLM, SGLang all ship it as a first class feature), but MLX only landed production grade speculative decoding in mlx_lm 0.21 earlier this year, and most local inference setups on Macs still run plain autoregressive decoding by default. The payoff for switching is large: in our benchmarks on an M5 Max, Llama 3.3 70B Instruct at 4 bit jumps from roughly 45 tokens per second to 95 plus when paired with a well chosen draft model, with no measurable degradation on coding or tool calling evaluations.

This post covers how speculative decoding actually works on Apple Silicon, which draft and verifier pairings produce real speedups versus regressions, the configuration knobs that matter, and the cases where you should leave the feature off.

What Speculative Decoding Does on Apple Silicon

Speculative decoding runs two models in tandem. A small fast draft model generates a candidate sequence of K tokens (typically 4 to 8). The larger verifier model then runs a single forward pass over that candidate, scoring each token. Tokens the verifier agrees with are accepted in one shot; the first token the verifier disagrees with becomes the corrected output, and the cycle restarts. The net effect is that the expensive 70B forward pass runs once and produces multiple accepted tokens per step, instead of one token per step.

On a memory bandwidth bound architecture like Apple Silicon, this matters more than it does on cloud GPUs. Token generation on an M5 Max is bottlenecked by the time it takes to stream the model weights from unified memory through the GPU cores once per token. At roughly 540 GB/s memory bandwidth and a 35 GB Llama 3.3 70B 4 bit checkpoint, the theoretical token ceiling is around 15 tokens per second per pass through the model. Speculative decoding lets you amortize that single pass over multiple accepted tokens, breaking through that ceiling without faster memory.

The arithmetic: if the verifier accepts an average of 3.2 tokens per draft round, your effective throughput multiplier is close to 3x in the best case, minus the draft model's overhead. Whether that ratio holds depends almost entirely on draft and verifier alignment.

Draft and Verifier Pairings That Actually Work on MLX

The single biggest determinant of speedup is how well the draft model's output distribution matches the verifier's. Mismatched pairings produce acceptance rates below 1.5 tokens per round, at which point the draft model's overhead exceeds its contribution and you lose throughput.

These are the pairings we benchmarked on an M5 Max 128 GB, running mlx_lm.server with --draft-model enabled, K = 6, temperature 0.0, on a mix of coding (HumanEval style), tool calling (JSON output), and summarization workloads.

A few patterns:

The Llama 3.2 1B as draft for Llama 3.3 70B is the strongest mainstream pairing. The 1B has lower draft overhead than the 3B and the acceptance rate is only marginally lower, so the 1B wins on net throughput. The 3B is the better pick if your workload skews to highly structured outputs (strict JSON, code with predictable boilerplate) where the higher acceptance rate compensates for the slower draft pass.

Cross family pairings (Qwen draft against Llama verifier, or vice versa) consistently produce acceptance rates below 2.0 and lose throughput. The training data distribution divergence is too large. Always pair within the same model family.

Qwen 3.6 14B as a verifier does not benefit from speculative decoding at all. The verifier is already small enough that the draft model overhead dominates. The break even point on M5 Max is roughly 30B parameters: below that, plain autoregressive decoding is faster.

Mixtral 8x22B at 4 bit is the largest beneficiary in our test set. The MoE routing in Mixtral means each forward pass only activates two experts, so the draft model's contribution amortizes against a large memory footprint without paying the full compute cost of a dense 176B model.

Configuration That Matters

The MLX mlx_lm.server speculative decoding config exposes three knobs that affect throughput meaningfully.

mlx_lm.server \
  --model mlx-community/Llama-3.3-70B-Instruct-4bit \
  --draft-model mlx-community/Llama-3.2-1B-Instruct-4bit \
  --num-draft-tokens 6 \
  --temperature 0.0 \
  --max-tokens 4096

The --num-draft-tokens parameter (the K in the algorithm) controls how many candidate tokens the draft model generates per round. The right value depends on acceptance rate.

If average acceptance is 3 to 4 tokens per round, set K = 6. The extra two slots cost almost nothing on the draft model and catch the long acceptance streaks that happen on highly predictable text (code blocks, JSON structures).

If average acceptance is below 2.5, set K = 4. Larger K wastes draft compute on tokens the verifier will reject.

If you cannot measure acceptance rate empirically, K = 5 is a safe default for same family pairings.

Temperature also matters. Speculative decoding is sampling exact: the output distribution at any temperature is identical to plain autoregressive decoding from the verifier alone. But at higher temperatures (say 0.8 plus), acceptance rates drop noticeably because the draft and verifier sample from divergent token distributions even when their logits align well. Coding workloads at temperature 0.0 get the full speedup. Creative writing at temperature 1.0 may see speedups of only 1.2x to 1.4x.

The MLX implementation supports prefix caching alongside speculative decoding, so multi turn agent workloads (where context grows across turns) keep the speedup intact as long as the draft model's KV cache also fits in memory. The 1B draft model takes roughly 800 MB of KV cache at 32K context, which is negligible next to the 70B's footprint.

When Speculative Decoding Hurts You

Speculative decoding is not always a win. Cases where it underperforms plain decoding:

Verifier under 30B parameters on Apple Silicon. The draft overhead exceeds the savings.

Highly stochastic sampling. Temperature 1.0 plus with high top_p settings tanks acceptance rates because the draft model cannot predict random sampling outcomes.

Out of distribution prompts. If your workload is dominated by rare languages, niche code (esoteric languages, specialized DSLs), or domain specific jargon the draft was not trained on, acceptance drops below 2.

Memory constrained machines. The draft model takes memory. On an M4 Pro 24 GB running a 32B verifier, adding a 1.5B draft can push you into swap, which destroys throughput. Speculative decoding needs at least 8 GB of headroom beyond the verifier's own footprint.

Cross family draft and verifier pairings. As the benchmark table shows, these consistently regress.

Comparison: MLX Speculative Decoding vs llama.cpp Speculative Decoding

llama.cpp added speculative decoding support for Apple Silicon in late 2024, and it runs comparably to MLX on the same hardware. We tested the same Llama 3.3 70B 4 bit verifier with Llama 3.2 1B 4 bit draft under llama.cpp's llama-server --draft-max 6 and got 94 tokens per second on the same M5 Max, versus 102 for MLX. The 8 percent gap mostly reflects MLX's better integration with Apple's Metal backend for the draft model forward pass.

For comparison content on the broader MLX versus llama.cpp question, see our deep dive on MLX versus llama.cpp on Apple Silicon, which covers the rest of the inference stack tradeoffs. For batching specifically, the MLX versus vLLM production inference comparison walks through the batched serving path.

vLLM does not support MLX as a backend (the vLLM and MLX integration post explains why), so if you want vLLM's medusa style multi token speculative decoding on Apple Silicon, you are out of luck for now.

When This Applies to Your Stack

Speculative decoding is worth turning on if you run any of these workloads on Apple Silicon:

Local coding agents serving an engineering team. The 2x speedup on Llama 3.3 70B for code completion translates directly to faster agent iteration cycles. At 100 tokens per second, the model keeps pace with a developer reading code; at 45, it does not.

Tool calling agents with structured JSON output. Acceptance rates are highest on structured output because the draft model is correct about boilerplate (braces, quotes, common field names). Real world acceptance on JSON heavy workloads hits 4.5 plus tokens per round.

Document analysis pipelines processing batches of long context inputs. Speculative decoding compounds with prefix caching on shared system prompts.

Skip it if you serve creative writing workflows at high temperature, run small verifiers, or have machines without 8 GB plus of memory headroom.

How to Evaluate This for Your Team

Run two configurations side by side on a representative workload (50 to 100 sample prompts from your actual use case) and measure: tokens per second end to end, mean acceptance rate, and quality on whatever eval matters to your application. Both numbers should improve or stay flat for speculative decoding to be worth the operational complexity. If acceptance rates fall below 2.5, try a different draft model or drop speculative decoding for that workload.

Contra Collective builds AI infrastructure for engineering teams that want to keep inference local, including production tuned MLX deployments with speculative decoding configured for the specific workload. If you are running local LLMs in production and leaving throughput on the table, we can help you find the multiplier. Reach out through our services page.

FAQ

Does speculative decoding change model output quality?

No. Speculative decoding is sampling exact: the output token distribution is identical to running the verifier model alone. Quality on benchmarks (HumanEval, MMLU, MT Bench) is statistically indistinguishable between speculative and plain decoding at the same temperature.

Which MLX version supports speculative decoding?

mlx_lm version 0.21 and later. Earlier versions had an experimental implementation that was not production stable. Check with pip show mlx_lm.

Can I use any draft model with any verifier?

No. Draft and verifier must share a vocabulary (same tokenizer) and ideally come from the same model family. Cross family pairings (Qwen draft, Llama verifier) consistently regress throughput.

How much extra memory does speculative decoding use?

The draft model weights plus its KV cache. For a Llama 3.2 1B 4 bit draft, that is roughly 800 MB weights plus 800 MB KV cache at 32K context, so 1.6 GB total. Plan for 8 GB of headroom beyond your verifier's footprint to be safe.

Does speculative decoding work with quantized models?

Yes. The verifier and draft can both be quantized independently. Most production setups quantize both to 4 bit and lose no measurable acceptance rate compared to full precision drafts. Mixed precision (8 bit verifier, 4 bit draft) also works.

Is speculative decoding worth it for batched serving?

Less so. Continuous batching already amortizes the verifier forward pass across multiple requests. Speculative decoding helps most for single request latency, which is what local Apple Silicon workloads optimize for. For batched serving at scale, see our MLX versus vLLM batching comparison.