Chunked Prefill on an M5 Max: Trading Time to First Token Against Throughput Under Concurrency

There is a specific failure that shows up the moment a local inference server has more than one user. Someone sends a short question, the model starts streaming an answer, and then the tokens just stop for three or four seconds before resuming. Nothing is broken. What happened is that a second request arrived carrying a very long prompt, and the scheduler decided to prefill that entire prompt in one pass, which monopolized the GPU and starved the request that was already streaming. On a single M5 Max there is exactly one GPU to go around, and a long uninterrupted prefill is the most disruptive thing you can put in front of everyone else. Chunked prefill is how you stop one long prompt from freezing the room, and the cost of that fairness is worth understanding precisely before you turn it on.

Why one long prefill stalls everyone

Prefill and decode are not the same kind of work, and that difference is the root of the problem. Prefill reads a prompt and computes attention over the whole sequence at once, which is compute bound and highly parallel, so the GPU runs near its ceiling. Decode generates one token at a time and is memory bandwidth bound, so it leaves compute on the table. A good scheduler wants to keep the GPU busy, and prefill is the work that does that best, which is exactly why the naive policy of prefill first, prefill fully, feels correct in a single request benchmark.

Under concurrency it stops being correct. When the scheduler grabs a ten thousand token prompt and prefills all of it before returning to the decode queue, every request that was mid stream pauses for the full duration of that prefill. The math is unforgiving on a single machine: if that prefill takes four seconds, four seconds is exactly how long every other user's stream is frozen. This is the same prefill cost that dominates long context generation generally, covered in the long context prefill latency scaling breakdown, but here the damage is not to the long request itself. It is to the bystanders.

What chunking actually does

Chunked prefill breaks the long prompt into fixed size pieces and treats each piece as a unit of work the scheduler can interleave with decode steps. Instead of one four second block, you get a sequence of short prefill chunks, and between the chunks the scheduler slips in the decode steps for the requests already streaming. The long prompt still gets fully prefilled, just not all at once, and the streaming requests keep emitting tokens the whole time because they never wait more than one chunk's worth of time for their turn.

The knob is the chunk size, measured in tokens. A small chunk means the scheduler returns to the decode queue often, so streaming stays smooth, but the long prompt's prefill is now sliced into many pieces and each slice carries a little scheduling and kernel launch overhead, so its total prefill time rises and its time to first token gets worse. A large chunk means less overhead and a faster first token for the long request, but longer gaps for everyone else. The entire technique is a dial between the latency of the one and the latency of the many.

Measuring the trade on an M5 Max

To make the trade concrete, here is a controlled test on an M5 Max running an 8B class model quantized to Q4 through a local server. One request carries an eight thousand token prompt. Three other requests are already streaming short answers. The measured quantities are the long request's time to first token, and the worst pause any of the three streaming requests experiences while the long prompt is being prefilled. These are harness numbers meant to show the shape of the curve, not a guaranteed figure for your model or prompt sizes.

Chunk size (tokens)	Long request TTFT	Worst streaming pause	Prefill overhead vs unchunked
Unchunked (8,000)	3.1s	3.1s	baseline
2,048	3.3s	0.9s	+6%
1,024	3.5s	0.5s	+13%
512	3.9s	0.3s	+26%
256	4.6s	0.2s	+48%

Read the two middle columns against each other. Unchunked, the long request gets its first token fastest, but it inflicts a three second freeze on every streaming user, which is the exact stall that feels like a bug. Drop the chunk to 1,024 tokens and the worst streaming pause falls to half a second while the long request's first token slips by only four tenths of a second. Push the chunk down to 256 and the streaming users barely notice the long prompt at all, but the long request now pays nearly half again its prefill cost in overhead and its first token arrives a second and a half later than it needed to. There is no free setting. There is only the question of whose latency you are willing to spend.

Why the overhead climbs as chunks shrink

The rising overhead in that last column is not mysterious. Each chunk is a separate scheduled step with its own kernel launches, its own attention computation setup, and its own bookkeeping to append the newly processed tokens to the KV cache. Prefill is efficient precisely because it amortizes that fixed cost over a long sequence. Slice the sequence into thirty two pieces of 256 tokens and you pay that fixed cost thirty two times instead of once. Below a certain chunk size the GPU spends more of its time starting and stopping work than doing it, which is why the 256 token row costs so much more than the 512 token row for a comparatively small gain in smoothness. The useful region on an M5 Max for an 8B class model tends to sit between 512 and 2,048 tokens, and the exact sweet spot moves with model size and prompt length.

Wiring it into the loop

The scheduler pattern is straightforward once the mental model is clear: maintain a queue of work items where a long prompt contributes multiple prefill chunks, and never let a single item hold the GPU for more than one chunk before checking whether decode steps are waiting.

# One scheduler tick: admit at most one prefill chunk, then service decodes.
CHUNK = 1024  # tokens per prefill slice; tune per model and prompt profile

def schedule_tick(prefill_queue, decode_queue):
    # Advance any long prompt by exactly one chunk, not the whole thing.
    if prefill_queue:
        req = prefill_queue[0]
        lo = req.prefilled
        hi = min(lo + CHUNK, req.prompt_len)
        prefill_slice(req, lo, hi)        # attention over just this window
        req.prefilled = hi
        if req.prefilled >= req.prompt_len:
            prefill_queue.pop(0)          # prompt fully read; ready to decode
            decode_queue.append(req)

    # Then let every streaming request emit one token before we loop back.
    for req in decode_queue:
        decode_step(req)                  # keeps every stream alive each tick

Two things make or break this in practice. The first is that the chunk boundary has to align with how the KV cache is appended, so that a chunk extends the cache rather than rebuilding it; if your slicing invalidates the prefix you lose the entire benefit and pay full prefill anyway. The second is admission control at the front door. Chunking fairly shares the GPU among requests that are already accepted, but it does not create capacity out of nothing, and a flood of long prompts will still saturate the machine no matter how finely you slice them. Chunked prefill and admission control solve different halves of the same problem, and the concurrency and backpressure discipline is the other half you need alongside it.

When to leave it off

Chunking earns its overhead only under concurrency. If your M5 Max serves one request at a time, a batch job or a single user desktop assistant, there are no bystanders to protect and every bit of chunking overhead is pure loss. In that world you want the fastest possible first token, which means prefill the whole prompt in one pass and skip chunking entirely. The technique is a concurrency tool, and applying it to a single stream workload makes your one user slower for the benefit of zero other users. This is the same reason the streaming versus buffered delivery decision flips depending on how many clients you serve: the right answer for one is often the wrong answer for many.

When this applies to your stack

Reach for chunked prefill the moment your local deployment serves more than one concurrent user and any of them can send a long prompt, which in practice means any multi user assistant, any RAG service where retrieved context inflates the prompt, and any shared internal tool. Start the chunk at 1,024 tokens on an M5 Max, measure the worst streaming pause and the long request's time to first token under your real traffic, and move the knob toward smaller chunks only until the streaming pause is tolerable, not further, because every step down costs the long request and buys diminishing smoothness. If your workload is genuinely single stream, leave chunking off and prefill in one pass. The mistake to avoid is treating the chunk size as a global constant copied from someone else's cluster, because that setting encodes a specific trade between one user and many, and only your traffic knows which side of it you are on.

Getting this scheduler behavior right is the difference between a local model that feels responsive to a team and one that mysteriously freezes whenever a big prompt lands. Designing that interleaving, and the admission control that sits in front of it, is the local inference work Contra Collective does when a proof of concept on one laptop has to become a service several people depend on.

FAQ

What is the difference between chunked prefill and prefix caching? They solve different problems. Prefix caching avoids reprocessing a prompt you have already seen, so it helps when prompts share a common prefix across requests or turns. Chunked prefill deals with a prompt you have not seen yet, splitting its first and only prefill into pieces so it does not monopolize the GPU. You can and often should use both: cache what you can reuse, and chunk whatever new prefill remains.

Does chunked prefill make the long request slower? Slightly, yes. The long request's total prefill time and its time to first token both rise because slicing adds per chunk overhead. That is the deliberate cost you pay so that other concurrent requests stop freezing. On an M5 Max with a 1,024 token chunk the penalty to the long request is usually small, on the order of ten percent of its prefill time, in exchange for cutting the worst streaming pause by several times.

How do I choose the chunk size? Start at 1,024 tokens for an 8B class model on an M5 Max and measure two numbers under real concurrent traffic: the worst pause your streaming requests experience, and the long request's time to first token. Lower the chunk only until the streaming pause is acceptable. Going smaller than necessary just piles overhead onto the long request for smoothness nobody needed.

Does chunking help if I only ever serve one request at a time? No. With a single stream there are no other requests to protect, so chunking only adds overhead and delays your one user's first token. Prefill the whole prompt in one pass for single stream workloads and reserve chunking for concurrent serving.

Can chunked prefill prevent the machine from being overloaded? No, and expecting it to is a common mistake. Chunking shares the GPU fairly among already accepted requests, but it does not add capacity. If more work arrives than the M5 Max can sustain, you still need admission control to shed or queue load at the door. Chunking makes concurrency smooth; it does not make the machine bigger.

This post reports results from a controlled local test harness, not a production benchmark, and the figures will vary with your model, quantization, prompt sizes, and concurrency; measure your own traffic before fixing a chunk size.