Long Context Prefill Latency on M5 Max: How Prompt Processing Scales From 8K to 128K Tokens (2026)

Time to first token is the latency users actually feel, and on a local Apple Silicon deployment it is almost entirely a prefill number. Prefill is the forward pass over the whole prompt that happens before the model generates anything, and it is a different kind of work than decode. Decode produces one token at a time and is limited by how fast the machine can stream the model weights out of unified memory. Prefill processes the entire prompt in parallel and is limited by raw compute, the floating point throughput of the GPU cores. Because these two phases bottleneck on different resources, a machine can be excellent at one and merely adequate at the other, and the M5 Max is exactly that machine: generous memory bandwidth that keeps decode fast, and a compute budget that makes long prompts expensive to ingest. This post is about the second half of that story, which is the half most people do not measure until a long context deployment feels slow and they cannot say why.

Why Prefill and Decode Scale Differently

The reason the two phases diverge is the shape of the work. During decode, each new token attends to every previous token, but the model only does one token of new computation per step, so the cost per token is roughly flat and the limiter is memory bandwidth as the weights and the growing key value cache are read each step. During prefill, the model computes attention across the whole prompt at once, and attention has a term that grows with the square of the sequence length, because every token attends to every other token. At short context that quadratic term is small relative to the linear cost of the feed forward layers, so prefill time looks roughly linear in prompt length. At long context the quadratic term takes over, and prefill time bends upward faster than the token count grows. That bend is the entire long context latency problem in one sentence.

There is a second cost riding alongside it. The key value cache, which stores the attention state for every token so decode does not recompute it, grows linearly with context length and consumes both memory capacity and, during the prefill write, memory bandwidth. We covered the capacity side of that in the KV cache and context length writeup; the point here is that at 128K tokens the cache is large enough that building it is itself a measurable slice of the prefill time, not a free side effect.

The Numbers, Measured on an M5 Max

These are directional results from a 128GB M5 Max running a 30B class model at 4-bit quantization in MLX, batch size one, warm model already resident in memory so cold load time is excluded. They are framed as observations to show the shape of the curve, not as universal constants, because the exact figures shift with model size, quantization, and runner. The pattern is what transfers.

Prompt tokens	Prefill throughput (tok/s)	Time to first token (s)	KV cache (GB)
8K	640	13	2.0
16K	610	27	4.0
32K	560	57	8.0
64K	470	136	16.0
128K	350	366	32.0

Read the throughput column and the time column together. If prefill were purely linear, throughput would hold flat and time to first token would simply double as the prompt doubled. Instead throughput falls as context grows, from 640 tokens per second at 8K down to 350 at 128K, and that decline is the quadratic attention term making each additional token more expensive to process than the last. The consequence is in the time column: going from 8K to 128K is a 16x increase in prompt size but roughly a 28x increase in wait, because you are paying more per token at the top of the range. A user who waits 13 seconds at 8K waits over six minutes at 128K, and no amount of decode speed fixes that, because decode has not started yet.

What Actually Moves the Number

The first and largest lever is not processing the prompt at all. If a large part of the context is stable across requests, a system prompt, a fixed instruction block, a document that many queries hit, prompt caching lets the runner keep that prefix's key value state and skip re-running prefill over it. On a stable 100K token context with a short changing suffix, this turns a six minute wait into the cost of prefilling only the new tokens. This is the single biggest win available for long context local inference, and it is why any serious deployment with a repeated prefix should treat prompt caching as a requirement rather than an optimization. It pairs naturally with keeping the model warm so you are not paying load time either, which we broke down in the cold start and model load latency post.

The second lever is chunked prefill. Rather than ingesting the whole prompt in one enormous forward pass, the runner splits it into chunks and processes them in sequence, which caps the peak memory the prefill needs and keeps a long prompt from spiking unified memory hard enough to trigger paging. Chunked prefill does not make the total compute smaller, the quadratic term is still there, but it makes long context prefill predictable and prevents the memory pressure failure mode where the machine starts swapping and throughput collapses far below what the compute alone would predict. Chunk size is a tuning knob: too small adds per chunk overhead, too large brings back the memory spike.

The third lever is choosing the model against the job. A 30B model at long context is paying the quadratic cost with more parameters per layer than an 8B or 14B model, so if the task does not genuinely need the larger model's capability, a smaller model processes the same long prompt several times faster because both the linear and quadratic costs shrink with the model. The comparison worth running is not which model scores highest on a benchmark but which model clears your quality bar at the context length you actually run, because a smaller model that fits the task turns a multi minute prefill into a manageable one. The runner matters here too, and the MLX and llama.cpp prefill behavior is not identical; we measured that split directly in the MLX versus llama.cpp prefill and decode comparison.

The Pattern to Design Around

The mistake teams make is designing for decode speed and discovering the prefill wall in production. If your workload feeds the model long prompts, RAG contexts that stuff many retrieved chunks, agent loops whose history grows every turn, document analysis over full files, then time to first token is your real latency budget, and it is dominated by a cost that grows faster than the token count. The defenses are structural: cache every stable prefix so you never re-prefill it, use chunked prefill to keep long prompts from triggering memory paging, trim the context to what the task needs rather than what fits, and size the model to the job so you are not paying quadratic attention on parameters you do not need. Decode will keep streaming fast on an M5 Max long after the prompt gets large; prefill is the phase that quietly decides whether long context is usable.

When This Applies to Your Stack

If you are building a local inference feature on Apple Silicon and the prompts are short, prefill is a rounding error and you can ignore all of this. The moment the context crosses into the tens of thousands of tokens, prefill becomes the dominant latency and its growth is superlinear, so a design that felt instant at 8K can feel broken at 64K. Measure time to first token at the context length you actually run, not at a convenient small one, and treat prompt caching as the first thing you build rather than the last. The order of operations that keeps long context local inference responsive is: keep the model warm, cache the stable prefix, chunk the prefill, and only then worry about decode.

If your team is putting local models behind a real product and the long context latency is not adding up, the fix is usually in the prefill path and the caching layer rather than the model choice, and it is the kind of measurement and architecture work we do at Contra Collective when we help teams move local inference from a demo that works on one short prompt to a system that holds up on the long ones users actually send.

FAQ

Why is prefill slower than decode at long context on Apple Silicon? They bottleneck on different resources. Decode is limited by memory bandwidth, which the M5 Max has in abundance, so it stays fast well past 100K tokens. Prefill is limited by compute, and attention adds a cost that grows with the square of the prompt length, so processing a long prompt stresses the exact resource the machine is less generous with, and time to first token climbs faster than the token count.

What is time to first token actually measuring? It is the wall clock time from sending the prompt to seeing the first generated token, and on a local deployment it is almost entirely prefill: the model has to read the whole prompt before it can produce anything. That is why a long prompt can sit for minutes with nothing on screen even on a machine that streams tokens quickly once it starts.

Does prompt caching remove the prefill cost? It removes it for the cached prefix. If a large stable block of context repeats across requests, the runner keeps its attention state and only prefills the new tokens, which can turn a multi minute wait into a fraction of a second plus the cost of the short changing part. It does nothing for a prompt that is different every time, which is why context that changes wholesale stays expensive.

Will a smaller model fix long context latency? Often, yes. Both the linear and the quadratic parts of prefill shrink with model size, so a smaller model that still clears your quality bar processes the same long prompt several times faster. The right question is which model is good enough at your real context length, not which model scores highest in the abstract.

Is chunked prefill faster than a single pass? Not in total compute; the work is the same. What it does is cap peak memory during prefill so a long prompt does not spike unified memory hard enough to trigger paging, which prevents the failure mode where throughput collapses well below what the compute predicts. It trades a little per chunk overhead for predictable long context behavior.