Persisting the MLX Prompt Cache to Disk: Skipping Prefill on Repeated Local Inference (2026)

Prefix caching in memory is a solved problem and most people have already turned it on. Hold a stable system prompt in the KV cache, reuse it across requests, and the prefill happens once. The gap that is still open is what happens at the process boundary. An in memory cache is tied to the lifetime of the running server, so a restart, a crash, a machine sleep, or a scheduled redeploy wipes it, and the first request afterward pays the entire prefill again. MLX lets you close that gap by writing the prompt cache to disk and loading it back, so the cache survives the process that created it. This post is only about that persistence layer, because it is the part that turns prefix caching from a per session optimization into a property of the deployment.

Why a Restart Is Expensive and an In Memory Cache Does Not Help

Prefill is compute bound. The model has to run attention over every token of the prompt before it can generate the first new one, and that cost scales with the length of the prefix. For a short chat prompt it is invisible. For the workloads where local inference earns its keep, a long repository context, a large system prompt with many tool definitions, a full document loaded for question answering, the prefix is tens of thousands of tokens and the prefill lands in the multi second range on an M5 Max.

The in memory prefix cache removes that cost for the second request and every one after it, which is exactly why it works. What it does not touch is the first request of a new process. When the server starts, the cache is empty. The prefix has to be prefilled from scratch even though its contents are identical to what the last process already computed and then threw away on exit. If your service is long lived and rarely restarts, you eat this once and forget it. If it restarts frequently, and plenty of real setups do, then you are paying a fixed prefill tax on every boot for a prefix that has not changed in weeks.

Three deployment shapes make this bite. A local model behind a desktop app that starts and stops with the app. A batch job that spins up, processes a queue against a fixed context, and exits. A service on a machine that sleeps, or that you redeploy on a schedule as part of a content or code pipeline. In all three the process boundary is crossed often, and in all three an in memory cache is empty exactly when you want it warm.

The Cache Is Serializable

MLX treats the prompt cache as data you can write out and read back, not just a runtime object. The mlx_lm package exposes helpers to build a cache, save it, and load it, and there is a command line path that precomputes a cache for a prompt and writes it to a file.

# Precompute the KV cache for a fixed prefix and write it to disk.
mlx_lm.cache_prompt \
  --model mlx-community/Qwen3-30B-Instruct-4bit \
  --prompt "$(cat system_prompt.txt)" \
  --prompt-cache-file prefix.safetensors

# Reuse it on generation. The prefix is not prefilled again;
# it is loaded from the file, so only the new tokens cost prefill.
mlx_lm.generate \
  --model mlx-community/Qwen3-30B-Instruct-4bit \
  --prompt-cache-file prefix.safetensors \
  --prompt "Given the above, summarize the failure modes."

The Python surface is the same idea when you are serving rather than shelling out. You build a cache, prefill it once against the fixed prefix, and persist it, then load it on the next process and continue generation from that state.

from mlx_lm import load, generate
from mlx_lm.models.cache import make_prompt_cache, save_prompt_cache, load_prompt_cache

model, tokenizer = load("mlx-community/Qwen3-30B-Instruct-4bit")

# First process: prime the cache on the stable prefix, then persist it.
cache = make_prompt_cache(model)
generate(model, tokenizer, prompt=SYSTEM_PREFIX, prompt_cache=cache, max_tokens=1)
save_prompt_cache("prefix.safetensors", cache)

# Later process, fresh start: load the cache instead of prefilling.
cache = load_prompt_cache("prefix.safetensors")
out = generate(model, tokenizer, prompt=USER_QUERY, prompt_cache=cache)

The file is a safetensors serialization of the KV cache state plus a small manifest describing the model it belongs to and the tokens it covers. That manifest matters, and the next section is about why.

What the Load Actually Costs

Persisting to disk trades a compute bound prefill for an input bound file read, and that trade is only a win when the read is cheaper than the prefill it replaces. The read cost scales with the size of the KV cache on disk, which is set by the prefix length, the number of layers and heads in the model, and the KV cache precision. A cache saved with an 8 bit KV cache is roughly half the bytes of one saved at 16 bit, so quantizing the KV cache does double duty here: it shrinks resident memory at run time and shrinks the file you have to read at start time. We went deep on that precision choice in the KV cache quantization writeup, and the same Q8 versus Q4 decision governs how heavy your saved cache is.

Frame the numbers as observations from a single 128GB M5 Max, not a guarantee. A long prefix in the tens of thousands of tokens costs multiple seconds to prefill cold. Loading the equivalent saved cache from a local SSD is bounded by sequential read throughput and deserialization, and for a cache in the low hundreds of megabytes that lands well under a second. The larger the prefix, the better the ratio, because prefill grows with attention cost while the file read grows only with the raw byte count. Below some crossover the saved cache is not worth it, because a short prefix prefills fast enough that the file open, read, and validation overhead is comparable to just doing the prefill. The persistence trick is a long prefix optimization, and on short prompts it can lose.

Situation	Cost driver	Roughly scales with	When it wins
Cold prefill, no cache	Attention over prefix	Prefix length, compute	Never the goal here
In memory cache hit	Pointer rebind	Constant	Same process, later requests
Load saved cache from disk	Sequential read plus deserialize	KV cache bytes	New process, long prefix
Save cache to disk	Serialize plus write	KV cache bytes	Amortized over many loads

The Constraints That Make a Saved Cache Silently Wrong

A persisted cache is a snapshot of internal state, and it is only valid under the exact conditions that produced it. Ignore these and you do not get an error, you get a plausible and subtly corrupted generation, which is worse.

The model must match. A KV cache is specific to a set of weights. Load a cache built for one model, or even the same model at a different quantization, and the keys and values no longer correspond to the attention the current weights expect. The manifest carries a model key precisely so a mismatched load can be refused rather than run, and you should treat that check as mandatory rather than optional.

The prefix must be a true prefix. A saved cache represents a specific token sequence. Your new prompt has to begin with exactly those tokens and append after them. If you change even the system prompt wording, the cache no longer describes your prefix and continuing from it is meaningless. This is why persistence pairs naturally with a fixed, versioned prefix: when the prefix changes, you regenerate the cache file and bump its version, rather than editing the prompt and hoping.

Tokenization has to be identical. The same text with a different tokenizer or a different chat template produces a different token sequence, and therefore a different cache. Pin the tokenizer and the template alongside the weights.

Rolling context needs trimming. For a chat that grows, MLX can trim the cache so it tracks a moving window rather than growing without bound, but a naive saved cache captures whatever state existed at save time. If you persist a long lived conversation, decide explicitly what you are snapshotting: a stable system prefix that you will always reuse, not a transient turn that will be irrelevant next session.

Serving Across Restarts Without Reprecomputing

The cleanest production shape is to make the cache file part of the deploy artifact. Precompute it in the build step for the fixed prefix, ship the safetensors file next to the model, and have the server load it on startup. The first request after a deploy then comes out at warm speed because the server booted with the cache already populated, and you never precompute at runtime on the hot path.

MLX server has grown a directory based version of this, where cached prompts are restored on startup for the matching model and the in memory set is written back on exit, so the persistence happens around the process lifecycle instead of by hand. Whether you use the built in directory or manage a single file yourself, the principle is the same: the expensive prefill happens once, offline, and every process afterward pays only the read. If you are choosing between MLX and llama.cpp for the serving layer underneath this, the runtime trade offs are in the mlx-lm server versus llama-server comparison, and the in memory hit that persistence extends is covered in the prefix caching writeup.

When This Applies To Your Stack

Persist the prompt cache to disk when three things are true: the prefix is long, tens of thousands of tokens rather than hundreds; the prefix is stable, a fixed system prompt or a versioned document or repository context rather than something that changes per request; and the process boundary is crossed often, because you deploy on a schedule, run short lived batch jobs, or ship a desktop app that starts and stops. Under those conditions the saved cache turns a multi second cold start into a sub second read and removes the single worst latency spike a local service has, the first request after a boot.

Skip it when the prefix is short, when it changes every request so no snapshot is reusable, or when the process is long lived and restarts are rare enough that eating one prefill on boot is genuinely free. In those cases the file management is overhead with no payback.

If you are standing up local inference behind a real product and the cold start on your long prefix is the thing making it feel slow, that is the AI infrastructure work we do at Contra Collective. The model choice is usually the easy part; the latency budget lives in prefill, memory, and exactly this kind of persistence detail that decides whether a local service feels instant or sluggish on every boot.

FAQ

Is a persisted prompt cache the same as in memory prefix caching? No. In memory prefix caching removes prefill for later requests within the same running process. Persisting to disk extends that across the process boundary, so a fresh process after a restart, crash, or deploy loads the cache from a file instead of prefilling the prefix again.

How much faster is loading a saved cache than a cold prefill? It depends on prefix length. The saved cache read scales with the KV cache byte size, while cold prefill scales with attention over the prefix, so the longer the prefix the better the ratio. On short prefixes the file overhead can erase the benefit; this is a long prefix optimization.

Can I use a cache saved from a different model or quantization? No. The cache is tied to specific weights and precision. The file carries a model key for exactly this reason, and a mismatched load should be refused rather than run, because continuing from a wrong cache produces corrupted output with no error.

Does quantizing the KV cache help the disk version too? Yes. An 8 bit KV cache is roughly half the bytes of a 16 bit one, which shrinks both the resident memory at run time and the file you read at startup, improving the load time as well.

What is the safest way to ship this in production? Precompute the cache file for the fixed prefix in your build step, ship it alongside the model, and load it on server startup so the first request after a deploy is already warm. Version the cache with the prefix so a prompt change regenerates the file rather than silently invalidating it.