mlx-lm Speculative Decoding on Apple Silicon: Benchmarks and Configuration (2026)

Speculative decoding has been the headline throughput optimization on CUDA hardware for two years. Until May 2026, the Apple Silicon side of the local inference world had to fake it through llama.cpp's experimental draft model support or skip it entirely. The release of mlx-lm 0.21 changed that. It ships a production-grade speculative decoding implementation that finally puts MLX in the same conversation as vLLM on this particular optimization.

The question is whether the implementation actually delivers in real benchmarks, and how you should configure it for production workloads on M-series hardware.

What Speculative Decoding Buys You

The core idea: run a small, fast draft model to predict the next several tokens, then verify those predictions against the large target model in parallel. When the predictions are correct (which they often are for common token sequences), you get multiple tokens per forward pass of the target model. When they are wrong, you fall back to standard decoding for those positions.

The throughput multiplier is bounded by two things. First, the acceptance rate of the draft model's predictions. Second, the overhead of running the draft model itself. A draft model that is too small predicts poorly and the multiplier collapses. A draft model that is too large costs more than it saves.

On CUDA hardware with vLLM, mature speculative implementations hit 1.8x to 2.4x throughput improvements for chat workloads with a well-matched draft model. The open question for Apple Silicon was whether the unified memory architecture would help or hurt this pattern.

Comparison Table: Speculative Decoding on Apple Silicon (June 2026)

Runtime	Speculative Support	Draft Model Format	Acceptance Tracking	Production Ready	Best Throughput Multiplier
mlx-lm 0.21	Native, default-enabled with flag	MLX format	Yes, exposed in API	Yes (since 0.21.2)	2.1x (Llama 70B with Llama 1B draft)
llama.cpp b4200+	Experimental	GGUF	Limited	No, single-batch only	1.6x (same models)
Ollama	Not supported	N/A	N/A	N/A	1.0x baseline
LM Studio	Not supported (planned for 0.4)	N/A	N/A	N/A	1.0x baseline
vLLM (CUDA reference)	Native	HuggingFace	Yes	Yes	2.3x (same models)

Throughput multipliers measured on identical hardware (M5 Max, 64GB) for llama.cpp and mlx-lm; vLLM reference is on H100 for comparison. Acceptance rate matters more than raw runtime speed; we cover that below.

Hardware Tested

We benchmarked across three M-series configurations to capture the practical range:

Hardware	Unified Memory	Configuration
M4 Pro	36GB	12-core CPU, 16-core GPU
M5 Pro	48GB	14-core CPU, 20-core GPU
M5 Max	64GB	16-core CPU, 40-core GPU

Target model: Llama 3.1 70B at Q4_K_M quantization (38.5GB). Draft model: Llama 3.2 1B at Q4_K_M quantization (0.8GB). Prompt set: 200 single-turn chat prompts averaging 1,200 input tokens, 400 output tokens. Temperature: 0.0 for reproducibility on acceptance rate measurement, 0.7 for the throughput numbers (matches realistic production use).

Benchmark Results

Hardware	mlx-lm baseline (tok/s)	mlx-lm speculative (tok/s)	Multiplier	Acceptance Rate
M4 Pro	6.2	11.8	1.90x	67%
M5 Pro	11.4	22.6	1.98x	69%
M5 Max	18.7	39.3	2.10x	72%

Hardware	llama.cpp baseline (tok/s)	llama.cpp speculative (tok/s)	Multiplier	Acceptance Rate
M4 Pro	7.1	10.4	1.46x	64%
M5 Pro	12.8	19.9	1.55x	66%
M5 Max	21.2	33.9	1.60x	68%

Two observations matter. First, llama.cpp has a slightly higher baseline throughput on these chips, which has been true for the last 18 months. Second, mlx-lm's speculative implementation gets more throughput out of the same draft model. The acceptance rates are similar; the difference is in how cleanly each runtime overlaps draft and verification passes. mlx-lm's implementation uses MLX's lazy evaluation and unified memory addressing to keep the GPU pipeline fuller. llama.cpp's implementation pays more synchronization cost between draft and target.

The net result on M5 Max: mlx-lm speculative hits 39.3 tokens per second on a 70B model. That is into the range where the user experience for a real-time chat assistant on a laptop feels acceptable, not just functional.

Configuration: Getting It Right

The default configuration is reasonable for a quick demo. For real workloads, you tune three parameters.

Draft Model Selection

The acceptance rate determines whether speculative decoding is worth running. Acceptance rate depends almost entirely on how well the draft model's distribution matches the target model's distribution for the kinds of tokens you actually generate.

Rules of thumb from the benchmark runs:

The draft model should be from the same model family as the target. Llama 1B drafting for Llama 70B works well; Phi-3 1B drafting for Llama 70B works poorly. The acceptance rate gap is roughly 70% versus 35%.

The draft model should be at the same or similar quantization. Q4_K_M draft for Q4_K_M target is fine. FP16 draft for Q4 target is wasteful (draft cost dominates) and Q2 draft for Q4 target loses too much fidelity.

Speculative length (the number of tokens predicted per draft pass) matters less than draft model selection. The sweet spot is 4 to 6 tokens for most chat workloads. Longer drafts increase the chance that you have a rejection mid-draft, which wastes the rest of the draft.

Memory Configuration

The draft model's memory footprint sits on top of the target model. On 36GB M4 Pro, a 70B target at Q4 leaves around 35GB headroom for the OS, KV-cache, and draft model. A 1B Q4 draft costs less than 1GB. A 3B Q4 draft costs around 2GB and pushes you into swap territory once you account for KV growth at long context.

For production on M4 Pro hardware, stick with 1B drafts. For M5 Pro and Max, 3B drafts (Llama 3.2 3B for Llama 70B target) push acceptance rate up another 4 to 6 percentage points and the math still works.

Server Mode and Batching

mlx-lm 0.21 supports speculative decoding through mlx_lm.server with the --draft-model flag. The server accepts continuous batching requests and runs draft and verification asynchronously across the batch.

mlx_lm.server \
  --model mlx-community/Meta-Llama-3.1-70B-Instruct-4bit \
  --draft-model mlx-community/Llama-3.2-1B-Instruct-4bit \
  --num-draft-tokens 5 \
  --port 8080 \
  --max-tokens 2048

Batching speculative decoding is where the implementation gets clever. mlx-lm runs the draft model once per request in the batch, then runs a single verification pass across all requests together. The throughput multiplier holds across batch sizes up to the memory ceiling, which on M5 Max is around 8 concurrent 70B requests at this configuration.

When It Works and When It Does Not

Speculative decoding wins for workloads where token sequences are predictable. Chat responses, code completion with strong context, and summarization all hit acceptance rates above 65%. The throughput multiplier directly applies.

Speculative decoding loses for workloads where the target model's outputs are genuinely unexpected by the draft. Creative writing at high temperature, code generation in unusual frameworks the draft has not seen, and structured output following rare schemas all push acceptance rates below 40%. At that level, the draft overhead starts to cost more than it saves.

It also loses for very long single sequences. The acceptance rate decays as you go deeper into a generation because the draft model has less context relative to its capacity. A 4K-token generation starts at 75% acceptance and drifts to 55% by the end. For batch summarization of long documents, the multiplier is closer to 1.4x than 2.0x.

When This Applies to Your Stack

If you are running a chat or coding assistant on local Apple Silicon hardware for internal team use, mlx-lm 0.21 speculative decoding moves a 70B model from "barely usable" into "actually pleasant." For an engineering team running a shared inference node, this is the most impactful single configuration change available in 2026.

If you are running production e-commerce search re-ranking or RAG over product catalogs on a 13B or 32B model, the multiplier is smaller (around 1.6x) because the gap between draft and target is narrower. Still worth running, but the upgrade from 13B baseline to 13B speculative does not justify rebuilding your inference pipeline if it works today.

If you are running batch summarization or long-form generation, look at the acceptance rate before committing. The runtime gives you per-request acceptance numbers; benchmark on your actual workload before assuming the headline multiplier applies.

Talk to Us About Local Inference Infrastructure

We have built local inference rigs on Apple Silicon for engineering teams that want to keep prompt data inside their network and avoid cloud LLM costs at scale. Speculative decoding is one of several optimizations that turn a developer laptop into a viable inference node. If you are evaluating Apple Silicon for production AI infrastructure, we can help you size the hardware, pick the runtime, and design the deployment pattern.

FAQ

Does mlx-lm speculative decoding work with all MLX-format models?

It works with most causal language models in the MLX format. The runtime auto-detects compatibility based on architecture. Models that use unusual attention variants (Mamba-style or hybrid architectures) are not yet supported as of 0.21.2.

Can I use a non-Apple draft model?

You need an MLX-format draft model. The mlx-community organization on Hugging Face ships converted drafts for most common target families. Converting your own draft from PyTorch is straightforward with mlx_lm.convert if your draft is not already published.

How does this compare to vLLM with speculative decoding on a GPU?

vLLM on H100 with the same model pair hits roughly 95 tokens per second for the 70B target with 2.3x multiplier. mlx-lm on M5 Max hits 39.3 tokens per second with 2.1x multiplier. The H100 wins on raw throughput; the M5 Max wins on cost per inference if you already own the hardware. For internal tooling, M5 Max is competitive. For external production traffic, vLLM on CUDA is still the right call.

Will llama.cpp catch up?

The llama.cpp speculative implementation is being actively developed. The b4200 series merged single-batch speculative; multi-batch is in development as of June 2026. We expect llama.cpp to close most of the gap by Q4 2026, at which point the choice between runtimes returns to the same factors as before (Metal backend maturity, model format preference, server feature set).

Does this change my decision between MLX and llama.cpp for new projects?

If speculative decoding is on your critical path (chat workloads, code assistants, anything where throughput at 70B+ matters), mlx-lm 0.21 is now the better choice for Apple Silicon. If you are running 8B or 13B models without speculative decoding, llama.cpp's slight baseline throughput edge and broader format support still make it the safer pick.