Cancelling a Local LLM Request Mid-Generation on an M5 Max: Freeing the GPU When Clients Disconnect

A local inference server has one scarce resource, the GPU, and a generation that no one is reading still consumes it fully. When a client disconnects halfway through a 600 token answer, a naive server keeps going: it decodes the remaining tokens, writes them into a closed socket, and only notices the connection is gone when it tries to flush and gets an error, if it checks at all. On a single M5 Max that is running near capacity, those wasted decode steps come directly out of the latency budget of every other request in flight. Cancellation is the mechanism that ends the generation the moment its consumer leaves, and it is the highest payoff reliability fix most local deployments never ship.

What an Abandoned Request Costs

To size the problem, measure the waste directly. The scenario is an M5 Max running a quantized 8B class model through a local server, decoding at a representative rate, with a client that disconnects at a random point in each generation. The numbers below are from a controlled test harness, not a production incident, and are meant to show the shape of the cost rather than a guaranteed figure for your setup.

Scenario (M5 Max, 8B Q4, decode ~55 tok/s)	Without cancellation	With abort on disconnect
Median wasted decode after disconnect	312 tokens	4 tokens
Wasted GPU time per abandoned request	~5.7s	~0.07s
Effective throughput under 20% abandon rate	41 tok/s usable	53 tok/s usable
p95 queue wait, 8 concurrent, 20% abandon	9.4s	3.1s
GPU busy time spent on dead requests	18%	under 1%

The single number that matters is the last row. Without cancellation, nearly a fifth of the GPU's decode time under this load is spent generating tokens that will never be seen, and that time is not free capacity sitting idle, it is capacity taken from live requests that are queued behind the abandoned one. The p95 queue wait more than triples for the same offered load, purely because dead work is holding the runtime. Cancellation does not make the model faster; it stops the machine from spending its fastest resource on nothing.

How Cancellation Actually Propagates

The mechanism has three links, and a break in any of them means the generation runs to completion regardless of what the client did.

The first link is detecting the disconnect. Over HTTP with a streaming response, a client going away eventually surfaces as a write error or a closed read side on the server socket, but "eventually" is the trap. If your server only learns the socket is dead when it next tries to write a token, detection is coupled to your streaming cadence, and a server that buffers output or writes infrequently will not notice for a while. The cleaner path is an explicit cancellation signal at the framework level: in an async server, the request handler's context or cancellation token fires when the connection drops, and you propagate that inward. In FastAPI or Starlette style servers, await request.is_disconnected() polled between tokens, or a task cancelled when the client goes away, gives you a signal that does not depend on a failed write.

The second link is carrying the signal into the generation loop. This is where most implementations quietly fail. The decode loop that produces tokens has to check the cancellation flag between steps and break out of the loop when it is set. In mlx-lm, that means the generation loop you are driving, whether through the streaming generate helper or a hand rolled loop over the model, needs a condition it evaluates each iteration:

async def stream_generate(model, tokenizer, prompt, cancel_event, max_tokens=600):
    for i, (token, _logprobs) in enumerate(generate_step(prompt, model)):
        if cancel_event.is_set():
            break                      # stop decoding immediately
        if i >= max_tokens or token == tokenizer.eos_token_id:
            break
        yield tokenizer.decode([token])
        await asyncio.sleep(0)         # let the event loop observe cancellation

The await asyncio.sleep(0) is not decoration. Without a yield point, a tight synchronous decode loop never returns control to the event loop, so the cancellation event set by the disconnect handler is never observed until the loop finishes on its own. A generation loop with no cooperative yield cannot be cancelled cooperatively, no matter how good your disconnect detection is.

The third link is releasing the resources the request held: the slot in the batch, the KV cache pages allocated to that sequence, and any admission token it took to enter the runtime. In llama-server, this is what the /completion handler does when it observes the connection closed, it stops the slot and frees it back to the pool. If you have built your own server around a batched runtime, freeing the KV cache for the cancelled sequence is what actually returns capacity; breaking the loop without releasing the cache leaves the memory pinned and the slot occupied, so you stopped the wasted compute but not the wasted allocation.

Where llama-server and mlx Differ

llama-server has cancellation built into its slot model. Each request occupies a slot, and when the client connection closes, the server marks the slot for release and stops feeding it through the batch. You get abort on disconnect largely for free, which is one of the underrated reasons to run llama-server rather than a hand rolled loop for anything facing real traffic. The caveat is continuous batching: a cancelled sequence is removed from the active batch, but the batch step in progress still completes, so cancellation is granular to the decode step, not the token. On an M5 Max that granularity is fine, because a single decode step is milliseconds.

mlx and mlx-lm give you the primitives but not the server. The generation helpers produce tokens; wrapping them in a cancellable async loop, wiring the disconnect signal, and freeing the cache is your code. That is more work, but it also means you control exactly when the loop checks for cancellation and how aggressively you reclaim memory. The failure mode specific to mlx is the synchronous loop with no yield point described above, where the abort event is set correctly and simply never read. If you are serving mlx behind an async framework, the single most important line is the cooperative yield inside the decode loop.

A pattern that helps on both is to bound max_tokens server side regardless of what the client requests. Cancellation handles the client who leaves; a hard server cap handles the client who asks for 4,000 tokens and then waits, which ties up a slot just as effectively as an abandoned one. The two mechanisms are complementary, and neither substitutes for the other.

When This Applies to Your Stack

Cancellation matters exactly when the machine is busy, which is the only time it is hard to reason about and the only time it pays. If your M5 Max serves one request at a time with slack to spare, an abandoned generation wastes a few seconds of otherwise idle GPU and nobody notices. The moment you have concurrency, an admission queue, or a gateway with its own timeout in front of the model, uncancelled requests become dead work that steals from live ones, and the throughput and p95 numbers above are what you leave on the table. Wire disconnect detection to a cancellation event, make the decode loop check it cooperatively, and free the KV slot on abort, then load test with a client that disconnects on purpose, because a cancellation path that is never exercised is a cancellation path that does not work.

This is the unglamorous production engineering that decides whether a local model degrades gracefully or wedges under load, and it sits next to the same concerns we cover in admission control and backpressure and streaming versus buffered delivery. Standing up an on premises inference server that survives real concurrency, with cancellation, admission control, and cache management that actually reclaim capacity, is the AI infrastructure work Contra Collective does. The model is the easy part; the serving layer is what keeps it responsive when more than one person is using it at once.

FAQ

Why does an abandoned request cost anything if the user is gone? Because the model keeps decoding tokens into a socket no one is reading until something stops it, and each of those decode steps uses the GPU that queued requests are waiting for. On an M5 Max near capacity, our test showed roughly 18% of GPU decode time going to dead requests under a 20% abandon rate, which is capacity taken directly from live traffic, not idle time reclaimed.

Does llama-server cancel on disconnect automatically? Largely yes. llama-server ties each request to a slot and releases the slot when the client connection closes, so you get abort on disconnect without extra work. Cancellation is granular to the current decode step under continuous batching rather than the individual token, which on Apple Silicon is a negligible difference of a few milliseconds.

What is the most common bug when adding cancellation to an mlx server? A synchronous decode loop with no cooperative yield. The disconnect handler sets the cancellation event correctly, but the tight generation loop never returns control to the event loop, so the flag is never checked until the generation finishes on its own. Adding an await asyncio.sleep(0) or equivalent yield point inside the loop is what makes cooperative cancellation actually fire.

Is stopping the decode loop enough to free capacity? Not by itself. Breaking the loop stops the wasted compute, but you also have to release the KV cache pages and the batch slot the sequence held, or the memory stays pinned and the slot occupied. Full cancellation means stopping the loop and returning the sequence's cache and admission token to the pool.

How is cancellation different from a max_tokens cap? They solve adjacent problems. Cancellation ends a generation when its consumer disconnects; a server side max_tokens cap bounds a generation whose consumer is still there but asked for far more than it needs. A slow, oversized request ties up a slot just as an abandoned one does, so busy servers want both a hard token cap and abort on disconnect.

This post reports results from a controlled local test harness, not a production benchmark, and the figures will vary with your model, quantization, and load; measure your own workload before sizing capacity around these numbers.