Inter-Token Latency and Tail Jitter for Local LLM Serving on an M5 Max

Tokens per second is the number everyone quotes and the number nobody feels. It is a whole generation divided by a wall clock, an average that flattens the one thing a reader actually perceives: the gap between each token and the next. That gap is inter-token latency, and on a local M5 Max it is not a flat line. It has a floor, a mean that sits above the floor, and a tail that occasionally jumps by an order of magnitude. A model that averages fifty tokens per second but freezes for a third of a second twice per paragraph reads worse than a steady thirty, because a person tracks the stall and forgets the average. If a human waits on your local model, inter-token latency and its tail are the metrics that decide whether it feels fast, and this is how to measure and control them.

What the average hides

Time to first token and inter-token latency are two different clocks, and conflating them is the first mistake. Time to first token covers loading, prefill of the prompt, and the first decode step, and it is dominated by prompt length and prefill kernel efficiency. Inter-token latency is everything after: the steady state gap between token N and token N plus one during decode. The two respond to different levers, so you have to measure them separately or you will tune the wrong one.

The second mistake is reporting only the mean. Decode on a local machine is a shared resource competing with the operating system, other processes, memory management, and sometimes a second model. That competition does not raise the mean much. It fattens the tail. So the honest summary of a stream is a small distribution, not one number.

Metric	What it measures	Why it matters
TTFT	Prompt in to first token out	Perceived responsiveness on submit
ITL mean	Average gap during decode	The baseline reading speed
ITL p50	Median gap	The typical experience
ITL p99	Worst one percent of gaps	The stalls a user remembers
Max ITL	The single worst gap	The freeze that breaks trust

The pair that predicts felt quality is p50 and p99. A tight ratio between them means a smooth stream. A wide ratio means a stream that looks fast on a spec sheet and stutters in front of a person.

Measuring it without fooling yourself

The correct measurement is a timestamp captured the instant each token is emitted from the decode loop, not when it arrives at a client across a socket. Network and serialization add their own jitter and you want to isolate the model first, then add transport back deliberately. In an MLX generation loop you record a monotonic clock reading inside the token iterator, take first differences of that series, and drop the first interval because it contains prefill and belongs to TTFT.

import time
from mlx_lm import load, stream_generate

model, tokenizer = load("mlx-community/Qwen3-8B-4bit")
stamps = []
for resp in stream_generate(model, tokenizer, prompt, max_tokens=512):
    stamps.append(time.perf_counter())

gaps = [(stamps[i]-stamps[i-1]) * 1000 for i in range(1, len(stamps))]  # ms
gaps.sort()
n = len(gaps)
print(f"count {n}")
print(f"mean  {sum(gaps)/n:.1f} ms")
print(f"p50   {gaps[n//2]:.1f} ms")
print(f"p99   {gaps[int(n*0.99)]:.1f} ms")
print(f"max   {gaps[-1]:.1f} ms")

Three methodology rules keep this honest. Warm up first, because the first generation after load pays for lazy kernel compilation and graph construction that will never recur, and folding that into your steady state numbers overstates the tail by a lot. Run long generations, because a tail percentile computed over forty tokens is noise; you want several hundred intervals minimum before p99 means anything. And fix the conditions you are not studying: same prompt shape, same batch, same power state, same thermal starting point, so the only thing moving is the variable you are testing. Benchmark methodology on Apple Silicon lives and dies on warmup and sample size.

Where the jitter comes from

Steady state decode on an M5 Max has a hard floor set by memory bandwidth. Each token requires streaming the model weights through the matrix units once, so the fastest possible gap is roughly the model size in bytes divided by usable bandwidth, and no setting beats that physics. A four bit quantized eight billion parameter model is a few gigabytes per token of weight traffic, which is why it decodes fast, and a four bit seventy billion parameter model is much heavier per token, which is why it decodes slowly. That floor is not the jitter. The jitter is everything that pushes individual gaps above the floor.

The largest tail contributor is memory pressure. Unified memory is shared between the model, the KV cache, every other application, and the window server. When resident memory approaches the ceiling, the system starts reclaiming pages, and a decode step that has to wait on the memory system stalls visibly. A long context makes this worse because the KV cache grows with every token, so a stream that was smooth at token one hundred can start stuttering at token three thousand purely because the cache crossed a pressure threshold. If your p99 degrades as the generation gets longer, suspect the cache before anything else.

The second contributor is kernel and graph churn. Runtimes select or compile a kernel for the shape they see, and if a shape changes mid stream, or lazy evaluation triggers a recompile, one token pays a large one time cost that lands in your tail. Keeping shapes stable and disabling autotuning during serving removes a class of these spikes.

The third is contention. Background compilation, a Spotlight index, a Time Machine snapshot, or a second model sharing the machine all steal bandwidth and scheduler time. On a laptop, add thermal state: once the package heats up and clocks drop, the whole distribution shifts, and sustained decode over minutes drifts slower than the first ten seconds suggested.

Flattening the tail

The goal is a tight p50 to p99 ratio, and the levers trade against throughput, so choose deliberately.

Leave headroom in unified memory. The single most effective move is to size the model and maximum context so that peak resident memory stays comfortably below the ceiling, ideally under about three quarters of it, so the system never enters reclaim during a stream. Dropping from a four bit seventy billion parameter model to a four bit thirty something billion parameter model can turn a spiky p99 into a flat one, and the smaller model is often the better product decision precisely because it feels steady.

Cap and manage the KV cache. If your workload has long conversations, a bounded cache with a rotating or evicted window holds memory flat and stops the late stream degradation, at the cost of forgetting the oldest tokens. Quantizing the KV cache to eight bit halves its footprint and buys you more stable context length on the same machine. This connects directly to how far you can push context on Apple Silicon before decode latency starts scaling badly.

Serialize when latency is the product. Continuous batching raises aggregate throughput by decoding several requests together, but it couples their fates: a heavy request in the batch stretches the gap for every request in it, which shows up as tail jitter for interactive users. If the machine serves one human at a time and steadiness matters more than aggregate tokens, run batch size one and accept lower peak throughput for a much tighter tail.

Pin the environment. Fix the runtime version, keep prompt and batch shapes stable, disable autotuning during serving, and quiet the machine: no background indexing, no second heavy model, and on a laptop a thermal profile that does not let clocks sag mid session. None of these raise the mean. All of them shave the tail.

When this applies to your stack

Inter-token latency is the metric to instrument the moment a local model sits behind anything a person reads in real time: a chat surface, an inline assistant, a streaming summarizer. For pure batch jobs, where nobody watches the stream and only total wall time matters, ignore the tail and optimize aggregate throughput instead, because there the average genuinely is the number that counts. The two regimes want opposite settings, and the mistake we see most often is a team tuning an interactive product for throughput and shipping a stream that benchmarks well and feels broken.

If you are putting local inference on Apple Silicon into a real backend and want the felt latency to match the spec sheet, this is exactly the kind of measurement and tuning work our AI infrastructure practice does with teams: instrument the distribution, find the tail source, and pick the throughput and latency tradeoff that fits the product rather than the benchmark.

FAQ

Is inter-token latency just the inverse of tokens per second? Only for the mean, and only when the stream is perfectly smooth. Tokens per second is one over the average gap. It tells you nothing about the tail, and the tail is what a reader notices.

What p99 should I target for a chat interface? As a rule of thumb, keep p99 within roughly two to three times p50. A p50 near twenty to thirty milliseconds with a p99 under sixty feels smooth. When p99 runs five or ten times p50, the stream stutters even though the average looks fine.

Why does my stream start smooth and get choppy later? Almost always the KV cache growing with context until unified memory enters reclaim. Bound the cache, quantize it, or shorten the maximum context, and the late stream stutter usually disappears.

Does continuous batching help or hurt inter-token latency? It helps aggregate throughput and hurts per request tail, because requests in a batch share fate. For a single interactive user, batch size one gives a tighter tail. For many concurrent users, batching is worth the jitter.