DFlash on Apple Silicon: Block Diffusion Speculative Decoding for Local LLMs

Speculative decoding is the standard trick for making local generation faster: let a cheap drafter propose tokens, let the expensive target model verify them, and accept the ones that match so you get several tokens for the price of one target forward pass. DFlash keeps that contract but changes the drafter. Instead of a small autoregressive model that predicts one token at a time, DFlash uses a block diffusion drafter that predicts a whole block of sixteen tokens at once, then the target model verifies the block in a single pass. The distinction matters, and the numbers below show why.

Property	Classic draft model	DFlash block diffusion
Drafter output per step	one token, autoregressive	16 tokens, predicted in parallel
Verification	target checks tokens sequentially	target verifies the whole block in one forward pass
Acceptance	longest matching prefix	longest correct prefix, greedy argmax
Quality contract	lossless if verified	lossless: every emitted token is the target's greedy argmax
Drafter cost	small autoregressive model	roughly 1B parameter block diffusion companion

How block diffusion drafting differs from a draft model

A conventional draft model is just a smaller version of the same kind of model: it generates left to right, one token per step, and the target accepts the prefix that agrees with what it would have produced. The ceiling on that approach is that the drafter itself is autoregressive, so proposing sixteen tokens means sixteen sequential drafter steps before the target ever weighs in. The technique is solid and well understood, and it is the same family covered in the draft model speculative decoding walkthrough, but the drafter is a serial bottleneck.

DFlash replaces the serial drafter with a block diffusion model that predicts all sixteen positions in parallel from a single pass, then hands the block to the target for verification. The target runs one forward pass over the proposed block, compares each position against its own greedy argmax, keeps the longest correct prefix, and rolls back the rest. Because verification is greedy and exact, the output is lossless in a precise sense: every token that gets emitted is the token the target model would have produced on its own. You are not trading quality for speed, you are removing redundant target forward passes when the drafter guesses right. The catch is that the drafter, a roughly one billion parameter companion, has to be good enough that its blocks are accepted often, and acceptance is where the real world speedup lives.

The real M5 Max numbers, not the viral multiplier

The claim that spread fastest was a flat 4.6 times speedup on Apple Silicon. That number does not appear in the primary benchmark table. The dflash-mlx repository publishes results on an M5 Max with 64 GB, MLX 0.31.1, median of three runs, and the speedup varies by model size, quantization, and context length. Here are the numbers at 2048 tokens, the mid context case that most local work looks like.

Model	Baseline tok/s	DFlash tok/s	Speedup	Acceptance
Qwen3.5-4B	53.9	188.7	3.49x	87.7%
Qwen3.5-9B	30.7	113.0	3.65x	89.2%
Qwen3.5-27B-4bit	33.1	70.2	2.12x	89.6%
Qwen3.5-35B-A3B-4bit	141.4	255.0	1.81x	89.8%
Qwen3.6-27B-4bit	32.3	90.7	2.81x	84.6%

Three patterns decide how you should read this. First, smaller dense models get the largest multiples; the full table peaks near 4.37 times on the 9B at a 1024 token generation, which is the closest thing to the viral number and it is neither 4.6 nor universal. Second, the widely repeated "about 70 tokens per second on M5 Max" figure is the 27B-4bit row, and its speedup is roughly 2.1 times, not 4.6. Third, a Mixture of Experts model that was already fast at baseline gains the least in relative terms, because the 35B-A3B-4bit was already producing 141 tokens per second without speculation, so there was less idle target capacity to reclaim.

Context length erodes the speedup

The other number people quote without a context length is misleading, because acceptance and speedup fall as the generation gets longer on large quantized models. The dflash-mlx table for the 27B-4bit target makes this concrete.

27B-4bit generation length	Baseline	DFlash	Speedup
1024 tokens	33.6	79.0	2.37x
2048 tokens	33.1	70.2	2.12x
4096 tokens	31.5	55.7	1.77x
8192 tokens	33.9	45.3	1.34x

The lesson is to benchmark at the token counts you actually generate. A speculative decoding win measured on a 1024 token reply overstates what you will see on an agent that produces long, structured outputs, because a memory bandwidth bound quantized model spends more of its time in the regime where the multiplier compresses. Meta's Muse Glimmer, which uses the same DFlash family for its optional drafter, reports about 1.8 times on the same M5 Max on its own workload, which is a useful sanity check that large models on Apple Silicon tend to land in the 1.5 to 3 times band rather than a flat 4x wall.

Standing it up behind an OpenAI compatible endpoint

The practical value is that dflash-mlx ships a server, so you can point any OpenAI compatible client at it and get the throughput without rewriting your agent. Installation and serving are one command each, and for registered Qwen3.5, Qwen3.6, and Gemma4 targets the matching draft model resolves automatically.

pip install dflash-mlx

dflash serve \
  --model mlx-community/Qwen3.6-27B-4bit \
  --draft z-lab/Qwen3.6-27B-DFlash \
  --port 8000

Point your client at the local endpoint and read live throughput and acceptance while a request is in flight:

curl http://127.0.0.1:8000/metrics

A few flags earn their place. Use --fastpath-max-tokens 64 to skip speculation on very short replies where the drafter overhead is not worth it and target only decoding is faster. Watch acceptance on your own prompts, because incoherent or highly structured outputs can collapse draft acceptance and turn the speculative surplus negative; when that happens you want the target only fallback, not a drafter fighting the target on every block. This is the same production discipline that governs any MLX serving decision on batching and throughput: measure the workload you actually run, not the one that looks best in a table.

Honest limitations before you build on it

DFlash on MLX is an early project with real edges. Architecture coverage is narrow: it is tuned for the Qwen3.5 and Qwen3.6 hybrid attention families and Gemma4, and other targets either fall back to a slower path or fail without a compatible drafter. There is no CUDA path, so this is an Apple Silicon and MLX story only; NVIDIA builders stay on their existing stacks. The published numbers are M5 Max 64 GB specifically, and older Apple GPUs use different kernels for verification, so you cannot extrapolate the absolutes to an M4 Max or an M2 Max without running the benchmark yourself. And the drafter costs memory, roughly a one billion parameter companion plus cache, which is real headroom to find on a 32 GB machine already running a 27B quant. None of this makes it a bad bet, it makes it a technique to benchmark on your exact model and context rather than adopt on a viral multiplier. The choice of quantization also interacts with all of this, which is why the quantization format you serve belongs in the same measurement.

When this applies to your stack

Reach for DFlash on MLX when you run Qwen3.5, Qwen3.6, or Gemma4 targets locally on Apple Silicon and decode time is your bottleneck: agent loops, long tool call chains, or interactive sessions where tokens per second is what the user feels. Expect a genuine 2 to 3.5 times on mid sized dense models at moderate context, less on large quants and long generations, and less again on MoE targets that were already fast. Do not size a workflow on the 4.6 times headline; size it on a dflash benchmark run against the exact model, quantization, and token length you will ship, and keep a target only fallback for the outputs where acceptance drops.

Standing up a local inference tier that actually holds its throughput under real agent traffic, choosing between speculative decoding, quantization, and batching for a given model and Mac, and wiring it behind a stable OpenAI compatible endpoint is the AI infrastructure work Contra Collective takes on when a team wants local models to serve production load rather than a laptop demo. The technique is real. The gain is the one you measure on your own tokens.

FAQ

How is DFlash different from a normal draft model? A normal draft model generates tokens one at a time, autoregressively, so proposing a block still takes many sequential drafter steps. DFlash uses a block diffusion drafter that predicts sixteen tokens in parallel from one pass, and the target verifies the whole block in a single forward pass. Both are lossless when verification is exact, but the diffusion drafter removes the serial bottleneck in drafting.

Is DFlash actually lossless? Yes, in the precise sense that every emitted token is the target model's greedy argmax at verification time. The drafter only proposes; the target decides, and it keeps the longest correct prefix. Minor divergence from pure autoregressive decoding is possible from framework dispatch edge cases, but no unverified token is emitted.

Where does the "70 tokens per second on M5 Max" number come from? That is the Qwen3.5-27B-4bit row at 2048 tokens, which reaches 70.2 tokens per second, roughly a 2.1 times speedup over its 33.1 baseline. It is not the 4B or 9B peak, which go much higher, and it is not a 4.6 times gain. Speedup depends on model size, quantization, and context length.

Will it speed up my coding agent? It speeds up any long decode, which includes agent loops and tool call chains, but structured tool output can lower draft acceptance and shrink the gain. Test your model's tool call format and keep a target only fallback for the outputs where acceptance drops, so a fighting drafter does not slow you down.

Does it run on my M4 Max or M2 Max? The published benchmarks are M5 Max 64 GB only, and older Apple GPUs use different verification kernels, so the absolute numbers will differ. Run the included benchmark on your own machine rather than extrapolating from the M5 rows.

Benchmark figures reflect the dflash-mlx repository and its published M5 Max table as of August 2026, measured at specific quantizations and context lengths; acceptance rates and throughput change with releases, models, and prompts, so run the benchmark on your own hardware and workload before sizing a production tier. Meta Glimmer figures are vendor reported on a different runtime.