Streaming vs Buffered Responses on a Local LLM: Does SSE Cost You Throughput on an M5 Max? (2026)

Token streaming is the default for any local model a human will read, and it should be, because a response that materializes at reading speed feels instant while a nine second wait for a finished paragraph feels broken even when the total time is identical. But the first time you profile a local server under real load, a suspicion creeps in. Streaming turns one response into hundreds of tiny socket writes, each one flushed immediately, each one a separate pass through the HTTP layer, and it is natural to wonder whether all that per token bookkeeping is quietly eating throughput the GPU could have spent decoding. On a hosted API the question is moot because the overhead is someone else's problem. On a single M5 Max where you own every cycle, it is worth answering with a measurement instead of a shrug. This post does that, and the result is more useful than a single number, because streaming turns out to change the two things that actually matter in different directions than you would expect.

What Streaming Actually Costs

Start by separating the two halves of serving a token. There is the generation half, where the model runs a forward pass, samples the next token, and appends it to the sequence, and that work happens on the GPU inside the inference engine regardless of how the result leaves the building. Then there is the delivery half, where that token is encoded, written to a socket, and flushed to the client. Buffered mode does the delivery half exactly once, at the end, after the whole response is generated. Streaming does the delivery half once per token, wrapping each one in a server sent events frame and flushing it so the client sees it immediately.

The worry is that the delivery half competes with the generation half for the same resources. It does not, or at least not in the way people fear, because on an M5 Max the two halves live on different hardware. Generation is GPU bound, saturating the Metal backend and the unified memory bandwidth that feeds it. Delivery is CPU bound, a matter of string encoding and syscalls on the performance cores, which sit idle during decode because the GPU is doing the heavy work. Streaming asks the CPU to do a little work between each GPU step, and the CPU has the headroom to spare. The cost is real but it is paid in a currency the decode loop is not spending.

The one place the two can collide is the flush. Flushing a socket per token is a syscall per token, and syscalls are not free. At eighty tokens a second that is eighty flushes a second per request, which is nothing. At eight concurrent requests each streaming at speed, it is several hundred flushes a second across the box, still comfortable for the CPU but no longer invisible, and this is the term that grows with concurrency rather than with the length of any single response.

The Measurement

These are directional figures from a 128GB M5 Max running a 14B class model at 4-bit under llama-server, driving each client with a roughly 2K token prompt and asking for a roughly 500 token completion, comparing streamed server sent events against a single buffered response for the same requests. They show the shape of the effect, not universal constants, because the exact numbers move with model size, quantization, prompt length, and how the client reads the socket.

Mode	Concurrency	Aggregate throughput (tok/s)	Time to first token	p50 total latency
Buffered	1	83	n/a (arrives whole)	6.1s
Streaming	1	82	0.4s	6.2s
Buffered	8	236	n/a	~17s
Streaming	8	231	0.5s	~17s

Read the throughput column first, because it settles the headline question. At one request, streaming costs about one percent of aggregate throughput, which is inside the noise of the measurement. At eight concurrent it costs a couple of percent, the flush overhead becoming faintly visible but nowhere near enough to matter. If your worry was that turning on streaming would meaningfully slow token generation on an M5 Max, the worry is unfounded. The decode loop is GPU bound and streaming is a CPU side tax, so the two mostly stay out of each other's way, and the box has plenty of CPU headroom to pay the tax while the GPU is busy.

Now read the time to first token column, because that is where streaming earns its keep. In buffered mode there is no such thing as a first token; the client waits the full six seconds and then receives everything at once. In streaming mode the first token lands in under half a second, and every token after it arrives as it is produced. The total latency is the same either way, but the experience is not remotely the same, and for anything a person is watching, the perceived speed of streaming is the entire reason to use it. You are not buying throughput with streaming. You are buying the appearance of immediacy at almost no throughput cost, which is a good trade whenever a human is on the other end.

Where Streaming Changes Concurrency Behavior

The subtle effect is not in the throughput table at all; it is in how streaming changes the shape of a request's lifetime, and that interacts with the admission control that keeps a single box alive under load. A buffered request occupies a generation slot for the duration of decode and then releases it the instant the last token is produced, because delivery is a single write that happens after the slot is already free. A streaming request holds its slot until the client has actually consumed the stream, and if the client reads slowly, or sits behind a slow network, or is a browser that throttles a backgrounded tab, the request lingers, holding a slot and the KV cache that comes with it, long after the model finished the real work.

This matters because concurrency on an M5 Max is bounded by memory, not compute, and every slot held is KV cache reserved in unified memory that no other request can use. We covered why that ceiling is a memory ceiling and how to size a bounded queue and return honest backpressure in the admission control writeup, and streaming quietly raises the stakes of getting that sizing right. A slow consumer on a streamed connection is a slot leak in slow motion. Eight fast clients streaming at reading speed are fine. Eight clients that open a stream and then read it lazily can pin eight slots for far longer than the generation actually took, and the ninth request waits on capacity that is being held hostage by transport, not compute.

The fix is a server side timeout on stream duration, decoupled from generation time, so a slot cannot be held indefinitely by a client that stopped reading. Set it generously above the expected completion time for your longest responses, but set it, so that a stalled or abandoned stream releases its slot and its cache instead of parking them until the socket eventually dies. This is the one operational cost of streaming that is genuinely worth engineering around on a single box, and it is an availability concern rather than a throughput one.

# llama-server: streaming is on when the client sends stream:true
# the flag that protects you is a per request timeout, not a throughput knob
llama-server \
  --model model-14b-q4.gguf \
  --ctx-size 32768 \
  --parallel 8 \
  --cont-batching \
  --timeout 120   # cap how long a slot can be held, including slow readers

# client side: consume the stream promptly; do not open it and wander off
import httpx, json

with httpx.stream("POST", "http://localhost:8080/v1/chat/completions",
                  json={"model": "local", "stream": True,
                        "messages": [{"role": "user", "content": prompt}]},
                  timeout=120) as r:
    for line in r.iter_lines():          # read as fast as tokens arrive
        if line.startswith("data: "):
            chunk = line[6:]
            if chunk == "[DONE]":
                break
        # do not block here; slow consumption holds a server slot

When to Buffer Instead

Streaming is the right default for interactive traffic, but it is the wrong default for two other patterns, and the distinction is worth drawing because teams tend to stream everything out of habit. The first is machine to machine calls where nothing downstream can act on a partial response. If a backend service calls the model to extract structured data, classify a message, or produce a JSON object that a parser will consume whole, streaming buys nothing, because the consumer cannot use half a JSON object and will wait for the end regardless. Buffer those, skip the per token flush overhead entirely, and free the slot the moment generation finishes rather than after a transport round trip.

The second is batch and offline work, where you are processing a backlog for maximum aggregate throughput and no one is watching any individual response. There the small flush tax is pure waste, and worse, streaming's slot holding behavior works directly against the goal of keeping every generation slot saturated with real decode work. Buffer, or better, use a batch oriented path that does not open a per request stream at all, and let the continuous batching machinery pack the GPU, which is the mechanism we broke down in the continuous batching post. The rule is simple: stream when a human reads the output live, buffer when a machine consumes it whole or when you are optimizing a backlog.

When This Applies to Your Stack

If you are running a local model on Apple Silicon and wondering whether streaming is costing you throughput, the measurement says stop worrying about that specific thing. On an M5 Max, server sent events cost one to three percent of aggregate tokens per second because delivery is CPU bound and decode is GPU bound, and the box has the CPU headroom to stream without stealing from the GPU. What streaming actually changes is time to first token, which it improves dramatically for interactive traffic, and slot lifetime, which it can extend dangerously when clients read slowly, so the real engineering work is a per request timeout that protects your concurrency ceiling rather than any attempt to claw back throughput. Stream for humans, buffer for machines and batches, and put a duration cap on every stream so a lazy reader cannot hold memory your next request needs.

Getting this right is the difference between a local inference service that feels fast and stays up and one that mysteriously runs out of capacity under load that the GPU could easily handle. When we help teams put local models behind a real backend, this transport and admission layer is where a surprising amount of the reliability lives, and it is the kind of AI infrastructure work Contra Collective does when a local deployment needs to move from a working demo to something that holds under production traffic.

FAQ

Does token streaming slow down a local LLM? On an M5 Max, only marginally. In our measurements streaming server sent events cost about one percent of aggregate throughput at a single request and a couple of percent at eight concurrent. Token generation is GPU bound while stream delivery is CPU bound, so the two run on different hardware and the CPU has headroom to stream while the GPU decodes.

Why use streaming if total latency is the same as buffered? Because time to first token is not the same. Buffered responses arrive all at once after the full generation completes, while streaming delivers the first token in under half a second and the rest as they are produced. For anything a person watches, that perceived immediacy is the entire point even though the total time to finish is identical.

When should I buffer instead of stream on a local server? Buffer for machine to machine calls where the consumer needs the whole response, such as structured JSON extraction or classification, and buffer for batch and offline processing optimized for aggregate throughput. In both cases streaming adds flush overhead and holds generation slots longer without any user facing benefit.

How does streaming affect concurrency on a single Apple Silicon box? A streamed request holds its generation slot and KV cache until the client finishes reading the stream, not just until generation ends. A slow or abandoned reader can pin a slot far longer than the real work took, which starves your concurrency ceiling. Set a server side stream timeout so a stalled connection releases its slot and memory.

What is the single most important setting when streaming from llama-server? A per request timeout that caps how long a slot can be held, decoupled from generation time. It protects the memory bound concurrency ceiling on a single box by ensuring a client that stops reading cannot hold KV cache indefinitely. It is an availability control, not a throughput knob.