Serving Concurrent Requests to a Local LLM on Apple Silicon: Admission Control and Backpressure on an M5 Max (2026)

Every local inference service works fine until the second user shows up. On an M5 Max a single request against a 14B class model at 4-bit streams faster than a person reads, and the temptation is to declare the thing production ready and point traffic at it. Then concurrency arrives and the service does not degrade politely. Latency climbs, then the machine starts paging, then a request dies with an out of memory error partway through a response. The instinct is to reach for a throughput knob, but throughput is not the problem. The problem is that a single Apple Silicon box has a hard ceiling on how many requests it can hold at once, that ceiling is set by unified memory, and a server with no admission control will happily accept work past the ceiling and take the whole process down with it.

Why the Ceiling Is Memory, Not Compute

On a Linux GPU server you think about concurrency in terms of compute: how many sequences the GPU can decode in parallel before throughput saturates. On an M5 Max the binding constraint shows up earlier and it is memory. Unified memory is one pool shared by the model weights, the operating system, every other process, and critically the KV cache for every request currently in flight. Each concurrent sequence holds its own KV cache, and that cache grows with context length, so ten requests each carrying a long prompt reserve ten separate caches that all draw from the same pool the model already sits in.

The model weights are a fixed cost you pay once. The KV cache is a variable cost you pay per concurrent request, and it is the term that scales with load. A 14B model at 4-bit occupies roughly 8GB of weights, which is comfortable on a 128GB machine. But if each in flight request at a long context reserves a couple of gigabytes of cache, then the concurrent working set is weights plus cache times concurrency, and that sum is what has to fit under the wired memory limit, the ceiling macOS places on how much memory the GPU is allowed to pin. We covered where that limit sits and how to raise it in the unified memory wired limit writeup, and it is the number that governs this entire discussion. Cross it and the allocator cannot pin more, so the system falls back to swapping GPU adjacent memory to disk, which for an inference workload is indistinguishable from the service dying.

This is why the failure is sudden. Compute degrades on a slope, memory falls off a cliff. Right up until the working set exceeds the wired limit, adding a request mostly costs you some shared throughput. The moment it crosses, you are paging, and paging a KV cache is so slow that every request on the box stalls at once.

Measuring the Cliff

These are directional figures from a 128GB M5 Max, a 14B class model at 4-bit, running under llama-server with continuous batching enabled, driving each concurrent client with a roughly 4K token prompt so the KV cache per request is substantial rather than trivial. They show the shape of the curve, not universal constants, because the exact crossover moves with model size, quantization, context length, and how high you have set the wired limit.

Concurrency	Aggregate throughput (tok/s)	Per request p50 latency	Peak pinned memory	State
1	82	1.0x	~11GB	Healthy
4	190	1.7x	~19GB	Healthy
8	240	3.1x	~30GB	Near ceiling
12	210	6.4x	~41GB	Paging begins
16	70	18x+	swap	Collapsed

Read the throughput column and the state column together. From one to eight concurrent requests, aggregate throughput rises because continuous batching is doing its job, packing multiple sequences into the same decode step, which is the mechanism we broke down in the continuous batching post. Per request latency creeps up over that range, which is the expected cost of sharing the GPU, and it is fine. Then somewhere past eight, the working set approaches the wired limit, the twelfth request tips it over, and throughput does not just stop improving, it reverses, because the machine is now spending its time moving cache pages instead of decoding tokens. By sixteen the service has collapsed. The useful capacity of this box for this model is around eight concurrent requests, and the entire engineering task is to make the server refuse the ninth cleanly instead of accepting it and dying at the twelfth.

Admission Control: Cap, Queue, Reject

The pattern that keeps a single box predictable has three parts and they map directly onto the curve above. First, a hard concurrency cap set to the number of requests the box can actually hold, which from the table is eight, not the theoretical maximum the server will let you configure. Second, a bounded queue in front of that cap, so a short burst above steady state waits briefly instead of failing, but the queue has a fixed depth so it cannot grow without limit. Third, honest backpressure: when the queue is full, the service returns a 503 with a retry hint immediately rather than accepting a request it has no capacity to serve.

llama-server exposes the first piece directly. The --parallel flag sets the number of concurrent slots, and each slot reserves a share of the KV cache, so setting it to your measured ceiling is how you tell the server not to admit more sequences than memory allows.

# size --parallel to the MEASURED ceiling, not the max the flag accepts
# each slot reserves KV cache, so this is a memory decision
llama-server \
  --model model-14b-q4.gguf \
  --ctx-size 32768 \
  --parallel 8 \
  --cont-batching \
  --port 8080

Setting --parallel 8 with a 32K total context means the context budget is split across slots, so size the context to what a request actually needs rather than the largest number that fits, because an oversized per slot context is wasted cache that lowers how many slots you can run. The server will now batch up to eight sequences and no more, but on its own it does not give you a clean queue and reject behavior for the ninth caller. For that, put a small gateway in front.

# asyncio admission gateway: cap in flight, bounded wait, then 503
import asyncio, httpx
from fastapi import FastAPI, Request, HTTPException

MAX_INFLIGHT = 8          # matches --parallel
MAX_QUEUE = 8             # short burst absorber, bounded
QUEUE_TIMEOUT = 2.0       # seconds a request will wait for a slot

app = FastAPI()
slots = asyncio.Semaphore(MAX_INFLIGHT)
waiting = 0

@app.post("/v1/chat/completions")
async def proxy(req: Request):
    global waiting
    if waiting >= MAX_QUEUE:
        raise HTTPException(503, "server at capacity, retry shortly")
    waiting += 1
    try:
        await asyncio.wait_for(slots.acquire(), timeout=QUEUE_TIMEOUT)
    except asyncio.TimeoutError:
        raise HTTPException(503, "queue timeout, retry shortly")
    finally:
        waiting -= 1
    try:
        async with httpx.AsyncClient(timeout=None) as client:
            body = await req.body()
            r = await client.post("http://127.0.0.1:8080/v1/chat/completions", content=body)
            return r.json()
    finally:
        slots.release()

The semaphore enforces the cap, the waiting counter bounds the queue, and the timeout converts a caller who would otherwise wait forever into a fast 503. The value of the 503 is that it is a signal a client can act on. A load balancer can shed it, a retry layer can back off, an autoscaler in front of a pool of Macs can add a box. A stalled request that eventually times out at the socket level is none of those things, it is just a bad experience with no information attached.

Where the Numbers Come From, and When to Grow Instead of Tune

The ceiling is specific to the box and the model, so measure it rather than copy the table. Load the model, drive concurrency up in steps with a fixed prompt length that matches your real traffic, and watch pinned memory with powermetrics or a GPU memory readout while you watch p50 latency. The right cap is the concurrency just before pinned memory approaches the wired limit and just before p50 latency stops rising linearly and starts to hockey stick. Set --parallel there, size the gateway to match, and leave headroom because the operating system needs the pool too.

There is an honest boundary to this. Admission control makes one box predictable, it does not make it bigger. If your steady state concurrency genuinely exceeds what a single Mac holds, the answer is horizontal: a small pool of machines behind the same gateway, or a move to a shorter context, a smaller model, or heavier quantization to raise the per box ceiling. Running several models on one machine is a related but different problem, the memory competition between distinct models rather than between requests to one model, and we treated that in the concurrent multi model serving post. What admission control buys you is that the single box degrades to a clean rejection instead of a collapse, which is the precondition for scaling it out sanely, because you cannot load balance across boxes that lie about their capacity.

When This Applies to Your Stack

If you are serving a local model on Apple Silicon to more than one caller, you need a concurrency cap sized to memory, a bounded queue, and a backpressure response, and you need to set the cap from a measurement rather than from the maximum the server flag will accept. The failure mode you are engineering against is specific: unified memory means every concurrent request competes with the model weights and the operating system for the same pool, and a server with no admission control will accept requests past the wired limit and take the whole process down when the working set crosses it. Cap at the measured ceiling, queue a bounded burst, return 503 when full, and treat the rejection as a scaling signal rather than an error to hide. Measure pinned memory and p50 latency together as you raise concurrency, because the crossover is where they both turn at once.

If your team is taking a local inference service from a laptop demo to something that has to hold real concurrent traffic, sizing that ceiling and building the admission layer around it is the infrastructure work we do at Contra Collective when we help teams make local inference operationally boring instead of quietly fragile.

FAQ

Why does my local model server fall over under concurrency instead of just getting slower? Because the binding constraint on Apple Silicon is unified memory, not compute. Each concurrent request reserves its own KV cache from the same pool that holds the model weights and the operating system, so past a certain concurrency the working set crosses the wired memory limit and the machine starts paging to disk, which stalls every request at once rather than slowing them gradually.

How do I find the right concurrency cap for my box and model? Measure it. Drive concurrency up in steps with a prompt length that matches your real traffic while watching pinned memory and p50 latency together. The cap is the concurrency just before pinned memory approaches the wired limit and just before latency stops rising linearly. Set --parallel to that number and leave headroom for the operating system.

What is the point of returning a 503 instead of just letting requests wait? A 503 is an actionable signal. A load balancer can shed it, a client can back off and retry, and a pool autoscaler can add a machine. A request that waits indefinitely and then times out at the socket carries none of that information and gives every other in flight request a chance to stall with it. Fast honest rejection keeps the box predictable.

Does continuous batching remove the need for admission control? No. Continuous batching raises aggregate throughput up to the memory ceiling by packing sequences into shared decode steps, but it does not stop the server from admitting more sequences than memory can hold. Batching and admission control are complementary: batching makes the slots efficient, admission control decides how many slots exist and what happens when they are full.

When should I stop tuning one box and add more machines? When your steady state concurrency exceeds what a single Mac can hold at your required context and model size. Admission control makes one box degrade cleanly, it does not increase its capacity. If the measured ceiling is below your load, scale horizontally behind the same gateway, or lower the per box cost with a smaller model, shorter context, or heavier quantization.