mlx-serve's 1M Token Context: Testing Apple Silicon at Long Context (2026)

mlx-serve v26.9.1, released September 3, 2026, added two things at once: support for a 1 million token context window and a faster Flash decode path for generation. Taken together they put a locally hosted, Apple Silicon serving stack in the same context length bracket as frontier hosted APIs, at least on paper. The number worth sitting with before you plan a deployment around it is not the context length itself, it is what that context length costs in memory, because a 1 million token context is a KV cache problem long before it is a throughput problem, and the two changes in this release address different halves of that problem.

Why 1 million tokens is a memory question first

Every token held in an active context has to keep a key and a value vector cached for every attention layer and every attention head, for as long as that token stays in the context window. That cache, not the model weights, is what usually runs out first when you push context length upward on a memory constrained machine. A back of the envelope estimate makes the scale of the problem concrete. For a representative dense model in the 70 billion parameter class, roughly 80 layers, 8 key value heads under grouped query attention, and a head dimension of 128, holding a full 1 million token context at 16 bit precision works out to approximately 328 gigabytes of KV cache alone, before you have loaded a single model weight. That is a controlled, illustrative calculation, not a measured benchmark, and it is meant to show the shape of the problem rather than a specific product's exact footprint.

Context length	Approximate KV cache (16 bit, illustrative 70B class model)	Fits in unified memory on a high end Mac
32K tokens	About 10 GB	Comfortably
128K tokens	About 42 GB	Yes, with model weights alongside
1,000,000 tokens	About 328 GB	No, not at 16 bit precision

The highest memory configuration available on Apple Silicon today is the M5 Ultra Mac Studio at 192GB of unified memory, which we covered in what the M5 Ultra can and cannot run locally, and even that machine falls well short of a 328GB dense cache. A naive, dense attention implementation of a 1 million token context is not something you run today on Apple Silicon at that model scale, no matter how much compute or bandwidth the chip has. The number only becomes usable once you change what the cache costs per token, which is the actual engineering problem mlx-serve's release is responding to.

What the Flash decode change is and is not

Flash decode style kernels, the pattern this release's decode path draws on, primarily solve a throughput and parallelism problem: they restructure how attention is computed during generation so that long sequences do not force the GPU into an underutilized, sequential pass over an ever growing cache. That matters enormously for tokens per second at long context, because a naive decode implementation slows down roughly in proportion to context length as the cache grows, and a well parallelized decode kernel keeps generation speed closer to flat. It is a real and valuable change. It is not a change to how much memory the cache occupies. A faster decode path lets you generate tokens efficiently once a long context is loaded, but it does not shrink the cache that has to be resident to get there.

This is the distinction we drew out on the model architecture side in testing GLM-5.3-Flash's hybrid sparse and linear attention on Apple Silicon: a KV cache reduction has to come from the attention architecture itself, sparse attention, linear attention, or aggressive cache quantization, not from a faster kernel operating on a cache that is still full sized. mlx-serve's two changes in this release are complementary for exactly that reason. Faster decode makes long context generation fast enough to be usable once you can fit the cache. It does not, by itself, get you to 1 million tokens on a memory constrained machine; the quantization and eviction strategy you pair it with does that work.

What actually makes 1M tokens fit

Three levers bring that 328 gigabyte estimate down to something a real machine can hold, and mlx-serve users have to apply at least one of them deliberately rather than assuming the context number alone means the cache fits.

Quantized KV cache is the most direct lever. Storing keys and values at 8 bit or 4 bit precision instead of 16 bit cuts the cache size by half or three quarters respectively, which brings the illustrative 328 gigabyte figure down to roughly 82 gigabytes at 4 bit, a number that starts to be plausible on a high memory Mac Studio configuration once you also account for model weights. The tradeoff is a small quality loss on long range retrieval that has to be validated on your own tasks rather than assumed acceptable, the same caution we raised about vendor reported KV cache reduction figures in long context decode performance on the M5 Pro and M5 Max.

Grouped query attention and smaller head counts reduce the per token cache cost architecturally, independent of anything mlx-serve does at the serving layer; a model built with fewer key value heads simply costs less cache per token to begin with, which is why the model you pair with mlx-serve matters as much as the serving stack itself.

Context management strategies, sliding windows, cache eviction, or retrieval based context assembly instead of holding the full million tokens live at once, sidestep the problem by never actually materializing the full cache, trading some recall of distant context for a memory footprint that stays bounded regardless of the nominal context window advertised.

A representative serving invocation, illustrative of the flag surface rather than pulled from official documentation, looks like this:

mlx-serve \
  --model mlx-community/Llama-3.3-70B-Instruct-4bit \
  --port 8080 \
  --context-length 1000000 \
  --kv-cache-bits 4 \
  --flash-decode

The context length flag sets the ceiling. Whether you can actually use that ceiling depends on the KV cache precision and the memory budget of the machine underneath it, not on the flag itself.

What to verify before you plan around this

Run your own long context test rather than taking the 1 million token number as a statement that the full window is usable at full quality on your hardware. Measure actual memory consumption at the context lengths you plan to use, not just the nominal maximum, and check whether the Flash decode path in your MLX build actually engages for your specific model architecture, since novel kernel paths routinely ship ahead of full coverage across model families. Confirm quantized KV cache quality on a retrieval task that resembles your real workload, since a cache quantization scheme that preserves quality on short factual lookups can still degrade on retrieval spread across the far end of a genuinely long context.

FAQ

What changed in mlx-serve v26.9.1? Support for a 1 million token context window and a faster Flash decode path for generation, released September 3, 2026.

Does 1 million token context mean I can hold that much context on any Mac? No. A dense attention KV cache at 1 million tokens for a representative 70 billion parameter class model runs to an estimated 328 gigabytes at 16 bit precision, well beyond current unified memory configurations. Reaching that context length in practice requires KV cache quantization, a model architecture with a smaller per token cache cost, or a context management strategy that avoids holding the full window live.

Does Flash decode reduce memory usage? No. It improves generation throughput at long context by keeping the GPU better utilized as the cache grows. It does not reduce the size of the cache itself; that requires a different lever, most commonly cache quantization or a different attention architecture.

What is the fastest way to make 1M context practical? Quantized KV cache, typically 4 bit or 8 bit instead of 16 bit, is the most direct lever, cutting the illustrative cache estimate by roughly half to three quarters, combined with a model that already uses a smaller key value head count.

When this applies to your stack

If you are evaluating a serving stack for long context local inference, treat the advertised context length as a ceiling and the KV cache math as the real constraint. Work out your own memory budget at the precision and model size you actually plan to run, test quantized cache quality on a retrieval task that resembles your workload, and confirm the decode path engages for your specific model before committing to an architecture around it. That evaluation, and the local inference infrastructure built on top of it, is the kind of AI infrastructure work we do at Contra Collective.