KV Cache Offload to NVMe SSD for 512K Context on M5 Max: Throughput, Latency, and the Memory Wall (June 2026)

The KV cache is the silent driver of context length on Apple Silicon. A Llama 3.3 70B model holding 512K context across a single session needs roughly 80GB of KV cache at FP16, which sits on top of the 40GB of weights at Q4_K_M. The arithmetic is simple, and it is also the reason a 128GB M5 Max cannot serve more than one or two ultra long context sessions concurrently without an eviction strategy. The interesting design space sits past the eviction line: keep the inactive sessions' KV cache resident on the local NVMe SSD instead of throwing it away, and pay a one time prefill cost only when sessions truly age out. The pattern works, but only inside a narrow operating envelope that engineering teams have to measure rather than guess.

We tested the offload pattern on a 128GB M5 Max MacBook Pro (40 core GPU, 16 core CPU, 8TB internal NVMe, macOS 15.4) running Llama 3.3 70B at Q4_K_M under mlx-lm 0.21 with a small custom session cache layer in front. The workload was a multi tenant document analysis agent with 64 concurrent sessions, each carrying 256K to 512K tokens of context (a mix of pasted legal exhibits, RAG retrieved passages, and prior turns). We measured prefill throughput, decode throughput, session restore latency from SSD, and the queue depth at which the SSD path stops being viable.

Headline Comparison

Dimension	In memory only (eviction at limit)	KV cache to NVMe (offload at idle)	Recompute on demand (no cache)
Max concurrent active sessions (512K context, Q4_K_M weights)	1	1 active, 64 resident	1
Total addressable sessions before eviction	1 to 2	64 (limited by SSD capacity)	unbounded (no state)
Session restore latency (512K context, p50)	full prefill, 38.4 sec	SSD load, 4.6 sec	full prefill, 38.4 sec
Session restore latency (512K context, p95)	42.8 sec	6.2 sec	42.8 sec
Active decode throughput (tokens/sec, batch 1)	18.4	18.2	18.4
Prefill throughput (tokens/sec, 8K input, fresh)	1,420	1,420	1,420
KV cache size (512K context, FP16, 70B)	80.6 GB	80.6 GB per session on SSD	0 GB
Peak unified memory (active session + weights)	121.4 GB	121.4 GB	41.2 GB
SSD throughput required (load, GB/s)	not applicable	17.4 sustained read	not applicable
Practical concurrency ceiling (sessions/min activated)	1.5	12.8	1.5
Wall clock cost per 100 session activations	71 min	9.8 min	71 min

The decode throughput line is the load bearing one. The offload pattern does not degrade active session decode throughput at all (18.2 vs 18.4 tokens/sec, inside measurement noise) because the KV cache for the active session sits in unified memory exactly as it would without offload. The cost shows up only at session activation, when the cache is loaded from SSD into unified memory. At p50 the restore is 4.6 seconds and at p95 the restore is 6.2 seconds, which is acceptable for an agent or document assistant where the user pauses between turns and unacceptable for a tight interactive chat. The choice of pattern is workload sensitive, not architecturally universal.

Why the Pattern Exists at All

The KV cache scales linearly with context length and the number of attention layers. For Llama 3.3 70B (80 layers, 64 attention heads at GQA 8 groups, head dim 128), the per token cache at FP16 is roughly 160KB across both K and V tensors. At 512K tokens that lands at 80.6GB. The numbers do not change with quantization of the weights: the cache lives in attention activations, not weights, so a Q4_K_M model has the same KV cache size as an FP16 model. KV cache quantization (Q8 or Q4 KV) cuts the cost in half or quarter, but at 512K context that still leaves 20GB to 40GB per session, and the eviction problem returns the moment concurrency grows past two or three sessions.

The naive options are both bad. Pure in memory caching limits the server to one or two concurrent ultra long context sessions before the unified memory budget is gone. Pure recompute (drop the cache when the session goes idle) requires a full prefill on every reactivation, which at 1,420 tokens/sec on prefill takes 38 seconds for 512K tokens. The user experience is a 38 second wait every time the conversation resumes, which is worse than evicting the session entirely and forcing the user to re-upload context.

The third option, KV cache offload to NVMe, splits the difference. Active sessions stay in unified memory. Idle sessions move their KV cache to SSD on a configurable timeout. Reactivation loads the cache from SSD into unified memory rather than recomputing it. The win shows up in two metrics: total addressable sessions per machine (64 vs 1 to 2), and reactivation latency (4.6 vs 38.4 seconds).

# Custom session cache layer in front of mlx-lm
from mlx_lm import load
import mlx.core as mx
import time
import pathlib

class SessionKVCache:
    def __init__(self, model, cache_dir: pathlib.Path, idle_seconds: int = 60):
        self.model = model
        self.cache_dir = cache_dir
        self.cache_dir.mkdir(exist_ok=True)
        self.idle_seconds = idle_seconds
        self.active = {}  # session_id -> (kv_cache, last_touched)

    def activate(self, session_id: str):
        if session_id in self.active:
            self.active[session_id] = (self.active[session_id][0], time.time())
            return self.active[session_id][0]

        path = self.cache_dir / f"{session_id}.safetensors"
        if path.exists():
            kv = mx.load(str(path))
            mx.eval(kv)  # force into unified memory
            self.active[session_id] = (kv, time.time())
            return kv

        return None  # new session, caller runs fresh prefill

    def offload_idle(self):
        now = time.time()
        for sid, (kv, touched) in list(self.active.items()):
            if now - touched > self.idle_seconds:
                path = self.cache_dir / f"{sid}.safetensors"
                mx.save_safetensors(str(path), kv)
                del self.active[sid]

The implementation is intentionally small. mlx-lm exposes the KV cache as MLX arrays that can be saved as safetensors and reloaded with mx.load. The interesting design decisions sit around the eviction policy (LRU is the default, but workload aware policies that look at conversation turn frequency outperform LRU on session resumption latency) and the SSD layout (one file per session is easy to reason about, but a shared mmap region scales better past a few thousand sessions).

The SSD Path Is the Real Constraint

The pattern lives or dies on NVMe throughput. The internal SSD in the M5 Max MacBook Pro sustains roughly 7.4 GB/s sequential read and 6.2 GB/s sequential write on the production drives we tested (Apple's internal 8TB NAND, not a third party Thunderbolt SSD). At those rates, loading an 80GB KV cache takes 10.8 seconds at the sustained read ceiling. In practice we measured 4.6 seconds at p50, which is faster than the sustained ceiling because the cache loads land in the macOS file system cache after the first few sessions and subsequent reads come from RAM rather than disk.

The reason the pattern scales to 64 sessions on this hardware (and not 640) is that the working set has to fit in the file system cache to hit the 4.6 second restore. Once the working set spills past available cache, every restore drops to the cold SSD read rate. The practical ceiling we measured is 64 active sessions with 256K to 512K context each (roughly 2.4TB of total cache on disk, with a hot working set under 64GB). Past that ceiling the restore latency climbs from 6 seconds toward 14 seconds at p95, and the user experience falls off a cliff.

The external Thunderbolt 5 SSD path is worse, not better. We tested a Thunderbolt 5 enclosure with a top end PCIe 5.0 NVMe drive (Sabrent Rocket 5) and saw sustained read at 5.8 GB/s, slower than the internal NAND on the M5 Max. The Thunderbolt 5 controller is the bottleneck (theoretical 80 Gbps, realized closer to 60 Gbps in the macOS driver stack as of 15.4). For the offload pattern, internal storage wins unless capacity forces an external option.

Prefill, Decode, and the Penalty Line

Active session decode is unaffected. We ran a sustained one hour load test with one active session decoding at batch 1 while a background thread continuously offloaded and reloaded inactive sessions, and the active session held 18.2 tokens/sec throughout (vs 18.4 baseline). The reason is that decode is memory bandwidth bound on the active KV cache, and the offload thread reads from SSD into a separate memory region without contending for the GPU's compute or the active cache's memory bandwidth.

Prefill is also unaffected for the same reason. A fresh session's prefill at 8K tokens runs at 1,420 tokens/sec whether the background is busy offloading other sessions or idle. The unified memory bandwidth on the M5 Max (546 GB/s) is far above the SSD read bandwidth (7.4 GB/s), so the SSD reads cannot saturate the memory bus that prefill and decode are competing for.

The penalty shows up only at reactivation. The 4.6 second restore is dead time from the user's perspective. The pattern that hides it best is prefetching: when an agent enters a state that strongly predicts session resumption (the user opened the conversation tab, the user moved focus to the chat input, the calendar event for the meeting is two minutes away), the cache loads in the background and the restore latency is amortized to zero. Prefetching is the difference between a usable production pattern and a research demo.

KV Cache Quantization on Top of Offload

The two patterns compose. Quantizing the KV cache to Q8 (one byte per element rather than two) cuts the per session disk cost from 80.6GB to 40.3GB, and the restore latency drops to 2.3 seconds at p50. Q4 KV cuts disk cost to 20.1GB and restore to 1.2 seconds. The quality cost is real but small: on long context retrieval (needle in a haystack at 256K to 512K), Q8 KV loses roughly 0.4 percentage points of recall, and Q4 KV loses 1.8 points. For most production workloads (multi turn agent dialogs, document analysis, code review on long files) the quality cost of Q8 KV is invisible.

Combining Q8 KV with NVMe offload is the highest leverage configuration we measured: 64 sessions at 512K context each, 2.3 second restore at p50, 18.0 tokens/sec active decode, and roughly 1.2TB total disk footprint. That is the configuration that ships when the workload is a long horizon multi tenant agent and the machine budget is one M5 Max.

When This Applies to Your Stack

The pattern is the right answer for: multi tenant agents that hold long context per session, document analysis services where users return to long sessions across hours or days, internal RAG copilots with persistent project context, and code assistants that hold an entire repository in context across a workday. It is the wrong answer for: interactive chat with tight latency budgets (the 4.6 second restore is unacceptable), workloads where context is regenerable cheaply (RAG with small retrieval windows that prefill in under a second), and workloads where session count is small enough that pure in memory caching fits.

The cost story scales the same way the cluster's other patterns scale. A single 128GB M5 Max plus 8TB of internal NAND, depreciated over three years, costs roughly $0.08 per active session hour at 64 concurrent sessions. The equivalent capacity on a GPU server (one H200 plus the storage layer needed for offload, or two H100s sharded) costs roughly $1.40 per active session hour. The order of magnitude advantage holds for workloads where the active GPU compute is far below the memory and storage demand, which is exactly the shape of multi tenant long context inference.

If your team is running into the KV cache memory wall on Apple Silicon (or thinking about a GPU server purchase to escape it), we work on local inference architecture for engineering teams that need ultra long context inference at multi tenant scale without paying GPU server prices. The pattern above is one of several that change the unit economics by an order of magnitude.

FAQ

Does KV cache offload to NVMe work for vLLM or llama.cpp the same way?

Both runtimes support paged attention with disk backed page swapping in 2026, but the integration points differ. vLLM's --swap-space flag enables disk swap for the KV cache pages and works on Linux GPU servers; the equivalent path on Apple Silicon is mlx-lm with a custom session cache layer, as in the code above. llama.cpp added a --cache-type-k disk experimental option in the 2026.4 release, which trades simplicity for a slightly higher restore latency than the safetensors based approach.

Does the offload pattern work on the M4 Max or M5 Pro?

It does, with smaller numbers. The 64GB M4 Max ceiling is roughly 16 concurrent sessions at 256K context with Q8 KV. The 48GB M5 Pro ceiling is roughly 8 concurrent sessions at the same context. The internal NAND is the same generation, so restore latencies are comparable. The bottleneck is the active memory budget for the model plus one full session, which is the same constraint that limits concurrent sessions in the in memory only mode.

Is the SSD wear a real concern at 64 sessions writing 80GB caches multiple times per day?

It is a concern worth measuring. The internal Apple NAND on the M5 Max is rated for roughly 1,200 terabytes written over the warranty period. At 64 sessions writing 80GB each twice per day, the daily write volume is 10TB, and the drive lifetime under that load is roughly 120 days. The mitigations are Q8 KV (halves the writes), longer idle timeouts (fewer offload events per session), and write coalescing (only write when the cache has actually changed since the last offload). On production deployments we run Q8 KV with a 5 minute idle timeout and see daily writes closer to 2.4TB, which puts drive lifetime past two years.