KV Cache Eviction on an M5 Max: Sliding Window vs Attention Sinks for Unbounded Chat Sessions

Every local chat assistant that gets real use eventually hits the same wall. The session does not reset between messages, so each turn appends the user's text and the model's reply to the KV cache, and that cache grows every single turn. On an M5 Max the unified memory is generous but finite, and the model's context window is a hard ceiling regardless of memory, so a conversation that runs long enough will fill the cache and force a decision: what do you throw away to make room. The obvious answer is to drop the oldest tokens and keep the newest, and it is wrong in a way that is invisible until it happens, at which point the model stops making sense while the recent context still looks perfectly intact. Understanding why that failure occurs is the whole game, because the fix is small and cheap once you know where to look.

Why the cache grows and why you cannot just let it

The KV cache stores the key and value tensors for every token the model has already processed, so that each new token can attend to the full history without recomputing it. That is what makes decode fast; it is also what makes the cache grow linearly with the length of the conversation. For a model in the 8B class the per token cache cost runs into the tens of kilobytes once you account for every layer and attention head, so a session that accumulates tens of thousands of tokens across many turns climbs from megabytes into gigabytes of resident memory. The same growth is what pushes decode latency up as context lengthens, the effect covered in the long context decode breakdown, and it is the reason the cache cannot be allowed to grow forever.

Two ceilings arrive. The first is the model's trained context window, past which attention has no meaning because the position encodings were never trained that far. The second is physical memory on the machine, shared between weights, activations, and cache under the unified memory model. Whichever you hit first, the outcome is identical: you must evict tokens from the cache to continue the conversation, and the policy you choose to evict by determines whether the model survives the operation.

The trap: dropping the oldest tokens breaks the model

The intuitive eviction policy is a sliding window. Keep the most recent N tokens, drop everything older, and the reasoning seems sound because a conversation's relevant context is usually recent. In practice a pure sliding window causes the model's output to collapse the moment the window slides past the very start of the sequence. The generations become repetitive, then incoherent, and perplexity spikes even though the recent tokens the model can still see are perfectly ordinary.

The cause is a structural quirk of transformer attention. Models learn to dump a large share of their attention weight onto the first few tokens of the sequence, regardless of whether those tokens carry meaning, because the softmax over attention scores has to put its mass somewhere and the initial tokens become a convenient sink for it. These anchor tokens are load bearing not for their content but for their position, and when a naive sliding window evicts them the attention distribution has nowhere to land and the model destabilizes. This is the attention sink phenomenon, and it means the first handful of tokens in any long session are the ones you can least afford to drop.

The three policies, measured

Here is a controlled comparison on an M5 Max running an 8B class chat model quantized to Q4 through a local server, driven by a synthetic conversation of two hundred short turns designed to overflow a modest working window. The measured quantities are output coherence, judged by perplexity on held out continuations, the resident KV cache size once the session reaches steady state, and the decode speed at that steady state. These are harness figures meant to show the shape of each policy, not a guaranteed number for your model or prompt mix.

Eviction policy	Coherence over 200 turns	Steady state cache	Decode speed at steady state
No eviction (grow until OOM)	stable until crash	unbounded, then failure	degrades as cache grows
Pure sliding window	collapses after first eviction	flat and small	fast and flat
Sliding window plus attention sinks	stable across all turns	flat and small	fast and flat

The middle row is the trap. A pure sliding window gives you exactly the memory and speed profile you want, flat and cheap, and pays for it with a model that falls apart the first time the window advances past the opening tokens. The bottom row keeps the same flat memory and speed while preserving coherence, and the only difference between the two is that it retains a small number of sink tokens from the very start of the sequence alongside the recent window. The top row, letting the cache grow without eviction, holds coherence right up until it exhausts memory and the process dies, and along the way its decode speed sags as the growing cache costs more to attend over every step.

The eviction loop that works

The working policy keeps two regions of the cache and drops everything between them. The first region is a small fixed block of sink tokens anchored at position zero, typically four to eight tokens, which almost always includes the start of your system prompt. The second is a sliding window of the most recent tokens. When the total exceeds your budget, you evict from the middle, never from the sinks and never from the freshest window.

# Evict from the middle: preserve attention sinks + recent window.
SINK_TOKENS = 4        # anchor tokens kept from position 0
WINDOW = 3072          # most recent tokens always retained
BUDGET = SINK_TOKENS + WINDOW

def evict_if_needed(cache):
    n = cache.length
    if n <= BUDGET:
        return cache                      # still under budget, nothing to do

    # Keep [0:SINK_TOKENS] and the trailing WINDOW; drop the span between.
    keep_front = cache.slice(0, SINK_TOKENS)
    keep_back = cache.slice(n - WINDOW, n)
    cache.rebuild(keep_front, keep_back)  # concat KV blocks, no recompute

    # Positions must stay monotonic for RoPE; renumber the kept tokens
    # so the recent window's positions continue from the sink block.
    cache.reindex_positions()
    return cache

Two details decide whether this holds up. The first is position handling. Rotary position encodings care about the relative distance between tokens, so after you splice out the middle you have to renumber the retained tokens into a continuous sequence, otherwise the model sees a gap in positions that it was never trained to interpret. The second is that the sink block should sit at the true start of the session, which in a chat assistant means the system prompt, so preserving the sinks also preserves the instructions the model is supposed to be following. That overlap is convenient but not automatic; if your system prompt is longer than the sink block, treat the whole prompt as pinned and slide only the conversational tail. This is a different lever from reusing an unchanged prefix across requests, which the prefix caching discussion covers, though the two compose cleanly: pin the prefix, sink the start, slide the rest.

What eviction cannot do

Eviction keeps a session running at flat cost, but it is genuinely forgetting. Anything in the evicted middle is gone from the model's attention, so a fact the user mentioned a hundred turns ago and that fell outside the window is simply not available anymore, no matter how cleanly the sinks preserve fluency. Attention sinks buy coherence, not memory. If your application needs the model to recall arbitrary earlier content across a very long session, eviction is the wrong tool on its own and you need a retrieval layer that stores the conversation externally and re injects the relevant pieces into the window on demand. The size of the window you keep is the real knob for how much recent context stays live, and that is the same context length tradeoff examined in the KV cache context length analysis. Eviction decides what stays cheap; retrieval decides what stays remembered, and they are not substitutes.

When this applies to your stack

Reach for sink aware eviction the moment you run a persistent local assistant that holds a session open across many turns, which describes almost any internal chat tool, coding companion, or support agent built on a local model. Set the window to the amount of recent conversation you actually need live, keep four to eight sink tokens pinned at the start alongside your system prompt, and never ship a pure sliding window, because it will look fine in a short demo and collapse in the first real hour of use. If your assistant needs to recall content older than the window, pair eviction with external retrieval rather than widening the window until you run out of memory, because a larger window slows every decode step and only delays the same wall. The failure this prevents is the worst kind, an assistant that degrades slowly and silently after working perfectly at launch, and the fix costs a handful of preserved tokens.

Getting eviction right is the difference between a local assistant a team trusts for long working sessions and one that mysteriously loses the plot after lunch. Designing that eviction policy, and the retrieval layer that sits behind it when a session has to remember more than it can hold, is the local inference work Contra Collective takes on when a promising prototype has to run all day without a restart.

FAQ

What are attention sinks? Attention sinks are the first few tokens of a sequence, which transformer models learn to attend to heavily regardless of their content. The softmax in attention must distribute its weight somewhere, and the initial tokens become a stable place to put the excess. Because the model relies on them as anchors, evicting them destabilizes the whole attention distribution, which is why a KV cache eviction policy has to preserve them.

Why does a pure sliding window break the model? A pure sliding window eventually evicts the opening tokens of the sequence, and those tokens are the attention sinks the model depends on. Once they are gone the model's attention has no anchor, and the output degrades into repetition and incoherence even though the recent tokens still in the window are perfectly valid. Keeping a small block of sink tokens alongside the window fixes it.

How many sink tokens should I keep? Four to eight is the usual range, and the gain from keeping more than that is negligible. The point of the sink block is to give attention a stable anchor, not to preserve meaning, so a handful is enough. In a chat assistant, make sure the sink block or a pinned prefix covers your system prompt so eviction does not silently discard the model's instructions.

Does eviction let the model remember the whole conversation? No. Eviction keeps a session running at flat memory and speed, but anything dropped from the cache is genuinely forgotten. Attention sinks preserve coherence, not recall. If you need the model to reference content older than its window, add an external retrieval layer that stores the conversation and re injects relevant pieces into the prompt.

Does this apply to models that already use sliding window attention natively? Partly. Some models are trained with a windowed attention pattern, which changes their tolerance for eviction, but the position handling and the need to preserve anchor tokens still apply when you splice the cache. Always confirm how your specific model treats positions after eviction, because a mismatch there is the most common cause of subtle degradation.

This post reports results from a controlled local test harness, not a production benchmark, and the figures will vary with your model, quantization, window size, and conversation profile; measure your own sessions before fixing an eviction policy.