oMLX on Apple Silicon: What the SSD KV Cache Actually Buys Coding Agents

Tool	Setup	Concurrency	Cache persistence	Best fit
mlx lm (bare)	Fastest single request, scripting only	Single request	None	Benchmarking, custom pipelines
Ollama (MLX backend)	Easiest install, largest model library	Serializes requests	None	Trying models quickly
vllm mlx	API focused, batching aware	Good	Partial	Production API serving
oMLX	Native menu bar app plus OpenAI and Anthropic compatible endpoints	Default limit of 8 concurrent requests	Full, survives server restarts	Coding agents, multi user, long repeated prompts

oMLX is an Apache 2.0 inference server built on MLX for Apple Silicon, distributed as a native SwiftUI menu bar app with a local web dashboard and OpenAI compatible endpoints alongside an Anthropic compatible /v1/messages route. Search volume for the term has grown into the thousands of monthly queries this year with essentially no competing content, which tells you the engineering community found this tool before most of the content marketing world did. The pitch that is driving the traffic is a cache architecture claim: time to first token on a long, previously seen prompt can fall from roughly 90 seconds to as little as 3, because the key value cache for that prompt was never thrown away, only moved.

Two tiers, one goal: stop recomputing what you already computed

Every local inference server has to decide what happens to a session's KV cache once that session goes idle. The default answer, in most servers, is to evict it and recompute the prefill from scratch the next time that prompt shows up, even if the prompt is 90 percent identical to the last one. For a coding agent that resends a large system prompt, a long file, and the growing conversation history on every single turn, that recomputation is the single biggest source of latency a user actually feels.

oMLX splits the cache into a hot tier and a cold tier. The hot tier keeps frequently used KV blocks resident in unified memory, exactly like any other server. The cold tier writes less active blocks to local SSD as safetensors files instead of discarding them. When a new request arrives with a matching prefix, oMLX restores the relevant blocks from disk rather than recomputing them through the model, and because the cache lives on disk rather than only in process memory, it survives a server restart, the same durability goal behind the KV cache offload to NVMe SSD work we measured on a 128GB M5 Max. The project's own numbers put this in concrete terms: a storage rework in an early 0.6.x release cut one mixed prefix workload's resident cache size from 282.7GB down to 15.3GB, which is the kind of reduction that actually determines whether a Mac with 64GB or 128GB of unified memory can hold enough context for a busy coding agent at all.

It is worth being precise about what this does not do. The SSD tier does not let you run a larger model than your unified memory supports. Model weights still have to fit in RAM in full. What moves to SSD is only the cache of already computed attention keys and values for prompts the server has seen before, which is a narrower and much more defensible claim than "run bigger models on less memory."

The headline number, and what actually backs it

The 90 second to 3 second claim traces back to a Show HN thread and the project author's own MLX community posts, reinforced by user reports on long context workloads rather than a single controlled benchmark run from a neutral third party. Read generously, it describes a coding agent sending a large, mostly unchanged context on a long session, where the alternative to cache restore is a full prefill recompute that can genuinely take the better part of a minute on consumer Apple Silicon. Read skeptically, every published restore speed figure so far comes from an internal NVMe drive, and the write up itself is inconsistent about whether the restored case lands at about 1 second or somewhere in a 1 to 3 second range. Treat the 90 to 3 framing as a real but best case illustration, not a number you can quote to a client as a guaranteed latency floor.

The concurrency data is more consistent. One independent test found aggregate throughput rising by a factor of roughly 1.3 to 1.4 when moving from a single concurrent request to four, on a backend that holds flat under the same load. A separate test on an M4 Max in low power mode measured close to a 4x aggregate throughput gain at four concurrent requests, which matters directly for any team serving more than one developer off a single Mac. Single request decode speed is roughly tied with bare mlx lm, with the meaningful difference being consistency: oMLX's run to run variance was measured at about plus or minus 0.4 tokens per second, against roughly plus or minus 2.3 for the bare library.

Where it still loses to the alternatives

Cold start time to first token is reported at two to three times worse than bare mlx lm, a direct consequence of lazy model loading on first request rather than an eager load at server startup. The practical fix is a deliberate warm up request immediately after the server starts, before real traffic arrives, which erases most of the penalty for a long running service. oMLX also requires macOS 15 or newer and an M1 through M4 family chip, so there is no path for Intel Macs, Windows, or Linux, and no larger context ceiling than what unified memory and the model's own limits allow.

A representative request to the Anthropic compatible endpoint looks like a normal Claude API call pointed at a local port:

curl http://localhost:8000/v1/messages \
  -H "content-type: application/json" \
  -d '{
    "model": "qwen3.8-coder-30b",
    "max_tokens": 1024,
    "messages": [{"role": "user", "content": "Summarize the diff in file auth.py"}]
  }'

Because the endpoint is compatible rather than a reimplementation, most Anthropic SDK based agent tooling can point at a local oMLX server with a one line base URL change, which is the detail that makes it a realistic swap in point for a team already building against Claude's API shape.

When this applies to your stack

If your team runs a coding agent, an internal retrieval pipeline, or any workload that resends a large, mostly stable context on every turn, the SSD tiered cache is the specific feature worth testing, not the headline latency number by itself. Run it against your own prompts for a week before trusting it in front of customers. The project has shipped several releases in rapid succession, which is normal for a young tool and also a reason to pin a version and test upgrades deliberately rather than tracking the latest release automatically. If you are weighing local inference against API spend for an engineering organization, that evaluation, concurrency planning, and cache architecture decision is exactly the kind of AI infrastructure work Contra Collective does for clients moving workloads off pure API spend and onto owned hardware.

FAQ

Does oMLX let me run bigger models on the same Mac? No. Model weights still have to fit entirely in unified memory. Only the KV cache for previously seen prompts can spill to SSD.

Is oMLX faster than bare mlx lm for a single user? Single request decode speed is roughly the same. The advantage shows up in concurrency and in repeated or resumed long prompts, not in raw single stream speed.

What hardware does oMLX actually need? macOS 15 or newer on an M1 through M4 family chip. A Mac Mini M4 Pro with 64GB or a Mac Studio are the commonly recommended starting points for serving more than one user.

Can I point existing Claude API code at a local oMLX server? Yes, through its Anthropic compatible /v1/messages endpoint, which typically only requires changing the base URL in whatever SDK or agent framework you already use.

Should I run oMLX in production today? Only after testing it against your own workload for a representative period. It is a fast moving young project, and the more dramatic latency claims are community reported rather than independently benchmarked.