Prompt Lookup Decoding on Apple Silicon: Draft Model Free Speedups on M5 Max (2026)

Every technique for making local generation faster on Apple Silicon fights the same enemy. Decoding one token at a time is memory bandwidth bound, so the model reads its entire weight set from unified memory for each token it produces, and on a 32B model at Q4 that ceiling lands somewhere in the low tens of tokens per second no matter how many GPU cores the M5 Max has. Speculative decoding is the usual escape hatch. It runs a small draft model to guess several tokens ahead, then verifies the guesses with the big model in a single pass, so one expensive forward pass can confirm several tokens at once. It works, and we measured it in the draft model speculative decoding analysis. The catch is that the draft model is a second set of weights you now have to keep resident in the same memory you were already rationing.

Prompt lookup decoding is the same bet without that cost. There is no draft model. The guesses come from the prompt itself. When the next token the model is about to generate has probably already appeared somewhere in the input, you can propose the continuation by copying it, verify it with one forward pass, and skip the per token grind for as long as the copy holds. This post is about how much that buys you on an M5 Max, which workloads it transforms, and the large category of prompts where it does nothing at all.

Three Ways to Decode, and What Each One Costs

Strategy	Where guesses come from	Extra memory	Big win case	Fails on
Plain autoregressive	nothing, one token per pass	none	nothing, this is the baseline	it is the baseline
Draft model speculative	a small second model	a full extra weight set resident	broadly, any generation	draft too weak or too large to fit
Prompt lookup	n-grams already in the prompt	effectively none	output that echoes the input	open ended text with no overlap

The column that decides everything is where the guesses come from. Draft model speculation is general, because a small model can guess plausibly about anything, but it is general at the price of a second model in memory and the tuning to keep the draft fast enough to be worth verifying. Prompt lookup is narrow, because it can only guess text that already exists in the context window, but when that assumption holds it is nearly free. It adds no weights, needs no separate model download, and its overhead is a hash lookup rather than a neural forward pass. The entire question is how often your real prompts contain the tokens the model is about to emit.

How the Mechanism Works

The idea is almost embarrassingly simple. As the model generates, you keep a rolling window of the last few tokens it produced, say the last three. You search backward through the prompt and the text generated so far for the most recent place that same three token sequence appeared. If you find it, the tokens that followed it there become your speculative proposal, often several of them at once. You run the big model forward once over the current token plus the proposed continuation, and the model tells you how many of the proposed tokens it agrees with. Every accepted token is a token you got without a dedicated decode step. The moment the model disagrees, you fall back to normal decoding for one step and start looking for the next match.

A minimal configuration in a llama.cpp style server exposes exactly two knobs, the n-gram size you match on and how many tokens you are willing to propose per match.

# draft model free speculative decoding via prompt lookup
./llama-server -m model-32b-Q4_K_M.gguf \
  --lookup-ngram-min 3 \
  --lookup-ngram-max 5 \
  --draft-max 8 \
  --ctx-size 32768

The n-gram minimum controls how eager the matcher is. A shorter match fires more often but proposes text that is more likely to be wrong, so you waste a verification pass. A longer match is confident but rare. The draft maximum caps how many tokens you copy per hit, which trades a bigger jackpot on a good match against a bigger wasted pass on a bad one. These are the only two parameters that matter, and unlike draft model speculation there is no second model to size, quantize, or keep warm.

What We Measured on the M5 Max

We ran a 32B model at Q4_K_M through llama.cpp on a 64GB M5 Max, batch size 1, at context lengths from 8K to 32K, and compared plain decoding against prompt lookup with a 3 to 5 token match and an 8 token proposal cap. The baseline decode rate on this setup was about 27 tokens per second, bandwidth bound as expected. Prefill was not the variable under test here, so all figures below are decode throughput on the generation phase. Every number is our result on this one model and these prompts and will move with yours.

Workload	What the output shares with the input	Decode tok/s, plain	Decode tok/s, prompt lookup	Speedup
Retrieval grounded QA	quotes and paraphrases the retrieved passages	27	roughly 50	about 1.9x
Document summarization	reuses phrases and names from the source	27	roughly 62	about 2.3x
Code editing and diffs	most of the file is copied, a few lines change	27	roughly 78	about 2.9x
Structured extraction to JSON	field values are lifted verbatim from input	27	roughly 58	about 2.1x
Open ended chat and ideation	little or no overlap with the prompt	27	roughly 28	about 1.0x

Read the top of the table and the bottom as two different stories. On retrieval grounded QA, summarization, extraction, and especially code editing, prompt lookup roughly doubles to triples decode throughput, because in those workloads the model spends most of its output copying, requoting, or lightly transforming text that is sitting right there in the context. Code editing is the extreme case: when a model rewrites a file to change three lines, the other several hundred lines are a near verbatim copy, and prompt lookup accepts them in long runs. Open ended chat is the opposite. When the model is inventing prose that never appeared in the prompt, there is no n-gram to match, almost nothing gets accepted, and you are left with the baseline plus a tiny overhead from the failed lookups.

Why It Wins Where It Wins

The acceptance rate is the whole mechanism. Speculative approaches only speed you up on tokens the big model agrees with, and prompt lookup can only propose tokens that already exist in the context. Those two facts multiply. In an echo heavy workload the proposed tokens are drawn from exactly the text the model wants to reproduce, so acceptance is high and each verification pass confirms a long run. In an open ended workload the proposals are noise, acceptance collapses, and every wasted proposal costs you a verification that produced one token instead of many. This is why the technique is a workload decision, not a global setting. It is close to a pure win on the parts of your product that read from a document and write about it, and it is dead weight on the parts that free associate.

There is a second reason it fits Apple Silicon specifically. On a bandwidth bound machine, the scarce resource is memory traffic, and a draft model spends that scarce resource twice, once to hold its weights and once to read them each draft step. Prompt lookup spends none of it. The proposal step is a cheap search over token ids in RAM, not a matrix multiply, so the only extra work the GPU does is the verification pass it was going to do anyway, now covering several tokens. You get the speculative speedup while leaving the entire memory budget for the model you actually care about, which on a 64GB machine trying to hold a large model and a long KV cache is the constraint that usually decides what you can run at all.

Prompt Lookup and Draft Models Are Not Rivals

It is tempting to frame this as prompt lookup versus draft model speculation, but they cover different ground and the honest answer is that a serious serving setup can route between them. Prompt lookup is the right tool when output overlaps input, which is most retrieval and editing traffic. Draft model speculation, which we covered for both runtimes in the MLX speculative decoding piece, is the tool when you need a general speedup on open ended generation and you have the memory headroom for a second model. If your product does both, the pattern is to enable prompt lookup for the document grounded routes and reserve the draft model for the creative ones, rather than paying the memory cost of a draft model on requests where a free n-gram match would have done the job. The prefill and decode split that governs all of this, and which phase your latency actually lives in, is the framing we worked through in the prefill versus decode analysis.

When This Applies to Your Stack

If you are serving a local model on Apple Silicon behind a feature that reads from documents, retrieved passages, or existing code and writes about them, prompt lookup decoding is one of the highest return changes you can make, because it roughly doubles or triples decode throughput on exactly that traffic while adding nothing to your memory footprint. Turn it on, match on 3 to 5 token n-grams, cap proposals around 8, and measure acceptance on your real prompts. If your product is open ended conversation with little reuse of the input, do not expect it to help, and do not confuse a lack of speedup there with a broken configuration. The technique is not general, and that is the point: it is nearly free where it works, so the only cost of trying it is measuring which of your routes echo their input.

If your team is putting a local model behind a real product surface and is trying to hit a latency target without buying a bigger machine, Contra Collective builds AI infrastructure where the decoding strategy is matched to the actual shape of your traffic and measured, not switched on by reputation. On a bandwidth bound machine the cheapest tokens are the ones you did not have to generate.

FAQ

What is prompt lookup decoding? It is a form of speculative decoding that uses no draft model. As the model generates, it matches the last few tokens against earlier text in the prompt and the generation so far, and when it finds a match it proposes the tokens that followed there. The large model verifies the proposal in one forward pass and accepts as many tokens as it agrees with. The guesses come from the context rather than a second model, so it adds no weights to memory.

How is it different from draft model speculative decoding? Draft model speculation runs a small second model to guess upcoming tokens, which works on any generation but keeps an extra weight set resident in memory. Prompt lookup guesses by copying n-grams already present in the prompt, so it costs almost no memory but only helps when the output overlaps the input. They suit different workloads and can be routed between rather than chosen once.

Which workloads get the biggest speedup? Anything where the output reuses the input. In our M5 Max testing, code editing and diffs saw the largest gain because most of a rewritten file is copied, followed by summarization, retrieval grounded QA, and structured extraction. Open ended chat saw essentially no speedup because there is no matching text in the prompt to propose.

Does prompt lookup decoding hurt quality? No. It is a lossless speedup. Proposed tokens are only kept when the full model agrees with them during verification, so the output is identical to what plain decoding would have produced. The only thing that changes is how many tokens are confirmed per forward pass. A bad proposal costs a little time, never correctness.

Why does it fit Apple Silicon so well? Local decoding on Apple Silicon is memory bandwidth bound, and a draft model spends that scarce bandwidth on its own weights. Prompt lookup spends none, because its proposal step is a cheap search over token ids rather than a neural forward pass. You get the speculative speedup while leaving the entire unified memory budget for the model and its KV cache, which is usually the binding constraint on what you can run.