Draft Model Selection for Speculative Decoding on Apple Silicon: Llama 3.2 1B vs Qwen3 0.5B vs Custom Distill on M5 Max (June 2026)

Speculative decoding on Apple Silicon either doubles throughput or wastes cycles, and the deciding factor is almost never the runtime. MLX 0.21 ships a clean speculative path, llama.cpp b4900 has had one for a year, and the algorithmic implementation is essentially solved. What is not solved, and what most teams skip past, is draft model selection. The wrong draft burns the prefill savings on rejected tokens and lands at slower wall clock throughput than greedy decoding from the target alone. The right draft compounds: tokens accepted in bulk, prefill amortized across the speculative window, and an end to end latency reduction that turns a 70B target into something approaching a 30B response budget.

We measured three credible draft model paths on a 128GB M5 Max running MLX 0.21 against Llama 3.3 70B Q4 and Qwen3 32B Q5 as the targets: Llama 3.2 1B Q4 as a same family draft for the Llama target, Qwen3 0.5B Q4 as a same family draft for the Qwen target, and a custom distilled 1.2B parameter draft trained on 2B tokens of the same target distribution. The test set: 8,000 generations spanning code completion (4K input, 1K output), agent tool calls (8K input, 512 output), and long form summarization (32K input, 2K output). We measured acceptance rate, end to end tokens per second, prefill latency, and the engineering cost of each path.

Headline Comparison

Dimension	Llama 3.2 1B Q4 (same family)	Qwen3 0.5B Q4 (same family)	Custom distill 1.2B Q4
Target model	Llama 3.3 70B Q4	Qwen3 32B Q5	Either, distilled per target
Draft size on disk	0.7 GB	0.4 GB	0.8 GB
Draft VRAM footprint at 8K context	1.2 GB	0.8 GB	1.4 GB
Acceptance rate (code completion)	78%	74%	86%
Acceptance rate (agent tool calls)	71%	68%	82%
Acceptance rate (long form summarization)	64%	61%	77%
End to end speedup vs greedy (code)	2.1x	1.9x	2.6x
End to end speedup vs greedy (agent)	1.8x	1.7x	2.4x
End to end speedup vs greedy (summarization)	1.6x	1.5x	2.1x
Draft load time, cold	0.8 s	0.5 s	1.0 s
Engineering cost to deploy	Low (pull from HF)	Low (pull from HF)	High (training pipeline)
Per workload tuning required	None	None	Yes (distill per target task mix)

The ladder reads cleanly. Same family stock drafts give a 1.5x to 2.1x speedup with essentially zero engineering investment beyond downloading the model. A custom distill trained on the target task mix gives a 2.1x to 2.6x speedup but requires a training pipeline, a held out evaluation set, and ongoing maintenance as the target model upgrades. The right pick depends on workload concentration: a single hot path that runs millions of inferences per month justifies the distill; a heterogeneous workload across a dozen tools does not.

Why Draft Model Selection Dominates Speculative Throughput

The speculative decoding loop draws K tokens from the draft model, then the target model verifies them in a single forward pass. Tokens the target would have sampled are accepted; the first divergence triggers rejection and resampling from the target. The math:

effective_tokens_per_pass = 1 + (accepted_tokens)
draft_cost = K * draft_decode_time
target_cost = 1 * target_prefill_with_K_tokens
speedup = (effective_tokens_per_pass * target_decode_time) /
          (draft_cost + target_cost)

The numerator scales linearly with acceptance rate. The denominator is dominated by the target verification pass, which is roughly the cost of a single target decode plus a small prefill overhead for the K speculative tokens. The breakeven point is sharp: below an acceptance rate of about 0.55 on M5 Max with a 70B target, speculative decoding is slower than greedy. Above 0.70 the speedup is meaningfully greater than 2x.

This is why draft selection dominates everything else in the speculative stack. Runtime choice (MLX versus llama.cpp) shifts the constant factors by 10 to 15 percent. K window size shifts speedup by another 10 percent within a reasonable range. Draft model selection shifts acceptance rate by 15 to 25 percentage points, which moves end to end speedup from 1.5x to 2.6x. The compounding is roughly multiplicative across the stack and the draft is the largest lever.

Acceptance Rate By Workload

Workload	Same family draft acceptance	Custom distill acceptance	Distill gain
Code completion (4K in / 1K out)	76% (Llama), 74% (Qwen)	86%	+10 to +12 pp
Agent tool calls (8K in / 512 out, JSON output)	71% (Llama), 68% (Qwen)	82%	+11 to +14 pp
Long form summarization (32K in / 2K out)	64% (Llama), 61% (Qwen)	77%	+13 to +16 pp
Chat / open ended (4K in / 2K out)	73% (Llama), 71% (Qwen)	79%	+6 to +8 pp
Translation (2K in / 2K out)	81% (Llama), 79% (Qwen)	84%	+3 to +5 pp

Two patterns sit underneath the numbers.

Constrained output workloads gain the most from a custom distill. Agent tool calls produce JSON output drawn from a small distribution of well formed structures, and a distill trained on that exact output shape achieves near perfect token agreement with the target on the predictable scaffolding (field names, brackets, quotes) and is only uncertain on the value tokens. Long form summarization has a similar property: the summary structure is predictable, only the specific content varies.

Open ended chat and translation gain the least. The output distribution is wide enough that a small distill cannot meaningfully outperform a same family stock draft on token by token agreement. For these workloads the same family Llama 3.2 1B or Qwen3 0.5B is the right pick.

Throughput on M5 Max: The Real Numbers

Workload	Target	Greedy decode tokens/sec	Same family speculative tokens/sec	Custom distill tokens/sec
Code completion	Llama 3.3 70B Q4	18.4	38.6	47.8
Code completion	Qwen3 32B Q5	32.1	61.0	83.5
Agent tool calls	Llama 3.3 70B Q4	18.4	33.1	44.2
Agent tool calls	Qwen3 32B Q5	32.1	54.6	77.0
Long form summarization	Llama 3.3 70B Q4	18.4	29.4	38.6
Long form summarization	Qwen3 32B Q5	32.1	48.2	67.4

The Qwen3 32B Q5 target with a custom distill at 83.5 tokens per second is the most striking row. The greedy decode at 32.1 tokens per second already feels usable for interactive coding. The speculative configuration with the distilled draft moves Qwen3 32B into the throughput range where it feels indistinguishable from a frontier hosted API for typical interaction patterns, on hardware that costs a fraction of the per token bill of a hosted frontier model.

The Llama 3.3 70B Q4 numbers are equally important for a different reason. Greedy at 18.4 tokens per second is on the slow side of usable for interactive workloads. Speculative with the distill at 47.8 tokens per second is comfortably in the interactive range, which means the 70B model becomes practical for tier one interaction workloads, not just batch processing.

Engineering Cost: The Distill Pipeline

The custom distill path requires:

# Pseudocode of the distillation pipeline
import mlx.core as mx
from mlx_lm import load, generate
from mlx_lm.tuner import distill

target = load("meta-llama/Llama-3.3-70B-Instruct-Q4")
draft_base = load("meta-llama/Llama-3.2-1B-Instruct-Q4")

# 1. Collect target outputs on the task distribution
training_set = []
for prompt in task_prompts:  # 5K to 20K prompts from the real workload
    target_out = generate(target, prompt, max_tokens=1024,
                          return_logits=True)
    training_set.append((prompt, target_out.tokens, target_out.logits))

# 2. Distill the draft to match the target's token distribution
distilled = distill(
    student=draft_base,
    teacher_outputs=training_set,
    temperature=2.0,
    kl_weight=1.0,
    ce_weight=0.5,
    epochs=3,
    learning_rate=2e-5,
)

# 3. Save and serve as the draft
distilled.save("distill/llama-3.3-70b-draft-v1")

The cost of running this pipeline on an M5 Max for a Llama 3.3 70B distill is roughly 18 hours of compute for 2B distillation tokens, plus a one time engineering cost of about a week to wire up the pipeline, evaluation, and the deployment surface. Maintenance cost is one to two days per target model upgrade to redistill against the new target.

Same family stock drafts have none of this cost. Download Llama 3.2 1B or Qwen3 0.5B from Hugging Face, point the MLX speculative path at it, run. The trade off is the 10 to 15 percentage points of acceptance rate left on the table.

Decision Table: Which Draft for Which Workload

Workload shape	Same family draft	Custom distill
Heterogeneous, many short lived tools	Strong default	Not worth the maintenance burden
Single hot path, millions of inferences per month	Acceptable	Strong default
Constrained output (JSON, code, structured)	Acceptable	Strong default
Open ended chat	Strong default	Marginal gain only
Translation, dense prose	Strong default	Marginal gain only
Frequent target model upgrades	Strong default	Each upgrade costs another distill cycle
Stable target for 6+ months	Acceptable	Strong default

Runtime Choice: MLX vs llama.cpp For Speculative

MLX 0.21 and llama.cpp b4900 both support speculative decoding cleanly. The MLX implementation has a slight edge on M5 Max because of better GPU utilization on the verification pass: roughly 8 to 12 percent higher acceptance throughput at the same model and draft. The llama.cpp implementation has the edge on cross platform deployment because the same configuration works identically on Linux servers with NVIDIA cards. For a pure Apple Silicon stack, MLX is the cleaner pick; for a hybrid stack that runs the same model on M5 Max for dev and NVIDIA for production, llama.cpp removes one variable.

When This Applies to Your Stack

If you are running local inference on Apple Silicon for any workload where latency matters (coding copilots, agent loops, RAG, customer facing chat) and you have not turned on speculative decoding, that is the highest leverage change available in the local inference stack today. Start with the same family stock draft: Llama 3.2 1B for any Llama 3.x target, Qwen3 0.5B for any Qwen3 target. Measure acceptance rate on your real workload. If it lands above 0.70 the same family draft is the right answer. If it lands between 0.55 and 0.70 and the workload is concentrated, the distill investment pays back inside three months.

The team at Contra Collective deploys local inference on Apple Silicon for clients running headless commerce search, agent based content pipelines, and on premise coding assistants. If you are scaling a local inference workload on Mac Studio or M5 Max hardware and want a second opinion on draft model selection, distillation pipelines, or end to end speculative architecture, that is the work we do.

FAQ

Q: Can a Qwen3 draft work as a speculative draft for a Llama 3.3 target? A: No. Cross family drafts have acceptance rates below 0.40 on every workload we measured, which puts the configuration below the speculative breakeven point. The vocabulary mismatch alone forces a remapping layer that kills the speedup. Always pick a draft from the same family as the target.

Q: How large should K (the speculative window) be? A: 5 to 8 tokens is the right range on M5 Max for a 70B target with a 1B draft. K=4 underuses the speculative pass; K=12 wastes draft cycles on tokens that get rejected anyway when acceptance rate is around 0.75. The MLX speculative path autotunes K within this range if left at the default.

Q: Does the draft need to match the target's quantization? A: No, but the draft should be at least Q4 for token agreement quality. A Q8 draft against a Q4 target adds 0.5 to 1.5 percentage points of acceptance rate at the cost of double the draft VRAM, which is rarely worth it on M5 Max where memory is the tightest constraint.

Q: How often do I need to redistill a custom draft? A: Once per major target model release. A distill against Llama 3.3 70B works essentially unchanged against Llama 3.3 70B minor revisions, but a Llama 3.4 release will need a fresh distill. Plan for one distill cycle per quarter under normal frontier release cadence.

Q: Can the draft be smaller than 0.5B and still help? A: For Apple Silicon, no. The fixed overhead of the verification pass dominates below 0.5B, and the smaller draft does not produce enough tokens per second to outrun the target's own decode. The sweet spot is 0.5B to 1.5B for a 30B to 70B target.