Qwen3-Coder 30B Local Agentic Loop on M5 Max: Prefill, Decode, and Tool-Call Throughput Tested (June 2026)

Qwen3-Coder 30B is the first open weight coder model that holds together inside a real agentic loop on a single M5 Max. The 480B flagship is the headline number, but the 30B A3B variant (30B total parameters, 3B active per token under MoE routing) is the one that earns a slot in production because it fits 128GB unified memory with KV cache, prompt cache, and a working tool-call buffer, and because the active parameter count lets decode throughput stay above the point where an agent feels responsive. A synthetic decode benchmark on the same model gives a number that flatters the runtime; a real agent loop, with tool calls, file reads, and an expanding context, is where local inference either works or quietly falls apart.

We ran Qwen3-Coder 30B A3B on a 128GB M5 Max (40 core GPU, macOS 15.4) under both MLX 0.21 and llama.cpp b4900, against a corpus of 200 agent transcripts pulled from real coding sessions: file edits, shell commands, grep and read tool calls, and multi turn debugging. Context lengths ranged from 8K (single file edit) to 96K (multi file refactor with full repo grep results). We measured prefill latency, decode tokens per second, KV cache memory, tool-call round trip time, and the wall clock cost of one complete agent step.

Headline Comparison

Both runtimes deliver an agent loop that is usable. MLX is faster on prefill and on prompt cache hits, which compounds across an agent loop where every tool call returns context that the next step must consume. llama.cpp is more flexible on KV cache management and is the only path that supports speculative decoding cleanly today on this model. The choice is not about peak decode; it is about which side of the cache hit ratio the workload sits on.

Why MoE Active Parameter Count Matters More Than Total Parameters

Qwen3-Coder 30B A3B activates 3B parameters per token through Mixture of Experts routing. The total 30B parameter weight stays resident in unified memory, but each forward pass touches the routed experts only, which collapses the decode compute envelope to something closer to a 3B dense model. A 70B dense model on the same hardware decodes at 18 to 22 tokens per second; the 30B A3B decodes at 78 tokens per second on MLX. Both are useful, but only one keeps an agent loop under the latency threshold where a developer keeps the loop running instead of pausing it.

The trade is paid in prefill. MoE routing has to fan out across experts on every prefill token, and the routing cost shows up at long context. At 8K context, prefill is 1,920 tokens per second; at 96K, it falls to 940. A 70B dense model with similar quantization prefills at 380 tokens per second at 96K. The crossover is around 256K context, and below that the MoE model is the better choice for an interactive agent.

Prompt Cache Is the Hidden Variable

Every benchmark conversation about Apple Silicon local inference starts with decode throughput and stops there. Inside a real agent loop, the prefill cost is paid on every step that does not get a prompt cache hit, and a 32K context cold prefill at 1,580 tokens per second is still a 20 second time to first token. That latency on every step turns a usable agent into an unusable one.

MLX 0.21 prefix cache and llama.cpp slot persistence both fix this for the common case where the system prompt, tool definitions, and recent file reads stay stable across steps. Our measurement: with prefix cache hit on a 32K context window where only the last 1,500 tokens (the new user message plus the prior tool output) changed, MLX delivered first token in 340 ms. The same workload without prefix cache reuse delivered first token in 9.4 seconds. The cache hit ratio in a real coding session sits between 70 and 85 percent depending on how the agent harness manages tool result truncation, and that ratio multiplies through the whole step budget.

# MLX 0.21 prefix cache pattern for an agent loop
from mlx_lm import load, generate
from mlx_lm.models.cache import make_prompt_cache

model, tokenizer = load("mlx-community/Qwen3-Coder-30B-A3B-Instruct-4bit")
cache = make_prompt_cache(model)

# First step pays full prefill cost on the system prompt + tool definitions
system_and_tools = build_system_prompt() + render_tool_schemas()
prompt = system_and_tools + user_turn_1
response_1 = generate(model, tokenizer, prompt=prompt, prompt_cache=cache, max_tokens=2048)

# Second step: the cache covers system_and_tools + user_turn_1 + response_1
# Only the new tool output and the next user turn pay prefill
prompt_2 = system_and_tools + user_turn_1 + response_1 + tool_output + user_turn_2
response_2 = generate(model, tokenizer, prompt=prompt_2, prompt_cache=cache, max_tokens=2048)

The trap: if the harness truncates or rewrites earlier context (a common pattern when context windows blow past 64K), the prefix cache invalidates and every step pays the cold prefill again. The fix is to design the harness around append only context for the steady state, with truncation deferred until the cache savings stop paying for themselves.

llama.cpp Has the Cleaner Path to Speculative Decoding Today

llama.cpp b4900 supports speculative decoding for Qwen3-Coder with a Qwen3-Coder 0.5B draft model. The setup pays off on long generations (the model emits long edit diffs) and is roughly neutral on short tool call generations. Our measurement on a 32K context generation with 2,048 token output: greedy decode at 71 tokens per second, speculative with the 0.5B draft at 122 tokens per second (acceptance rate 64 percent), end to end 1.7x speedup.

MLX 0.21 has a speculative path, but the Qwen3-Coder draft pairing is not yet packaged, and the acceptance rate on a hand assembled MLX draft we tested was below the threshold where the prefill cost on the draft pays off. For workloads dominated by long output (code generation, multi file refactor diffs), llama.cpp is the better choice today. For workloads dominated by short tool call output and where prefix cache hits dominate, MLX is the better choice.

# llama.cpp speculative decoding setup, Qwen3-Coder 30B + 0.5B draft
./llama-server \
  --model qwen3-coder-30b-a3b-q4_k_m.gguf \
  --model-draft qwen3-coder-0.5b-q8_0.gguf \
  --draft-max 8 \
  --draft-min 4 \
  -c 98304 \
  -ngl 99 \
  --cache-type-k q8_0 \
  --cache-type-v q8_0 \
  --port 8080

KV Cache Quantization Is the Other Lever

At 96K context, KV cache for Qwen3-Coder 30B A3B at FP16 is 18.6 GB. With Q8_0 KV cache (the safe default for code workloads), it drops to 11.2 GB on MLX and 12.8 GB on llama.cpp. With Q4_0 KV cache, it drops to 7.4 GB on llama.cpp, with a measurable quality cost on long context recall (pass@1 on long context retrieval drops 3.2 percentage points). For an agent loop that holds 96K of repo context, Q8_0 KV cache is the right default. Q4_0 is the right choice only if the workload truly needs 128K and the memory headroom is otherwise gone.

When This Stack Earns Its Slot

A local Qwen3-Coder 30B agent loop on M5 Max is not a replacement for a frontier model on every workload. It is the right pick when one of three constraints holds: the codebase is sensitive enough that the team will not send it to a hosted API; the per developer API spend on a frontier model has crossed the point where local inference amortizes the hardware in under 12 months; or the team has an agent harness that benefits from microsecond level prompt cache hits that the round trip latency of a hosted API cannot match. The 30B A3B variant is the size where unified memory headroom, decode throughput, and agent loop responsiveness all clear the production threshold at once.

The 480B flagship is the right pick for offline batch evaluation, code review queues that can tolerate seconds per token, or distributed inference across a multi node Mac cluster. The 30B A3B variant is the right pick for the synchronous developer in the loop case, which is where the actual hours move.

When This Applies to Your Stack

If your team is evaluating local coding agents on Apple Silicon and the question on the table is whether the latency budget actually closes inside a real agent loop, the numbers above are the starting point. We build AI infrastructure for teams running local inference in production: prompt cache design, agent harness throughput tuning, MLX and llama.cpp deployment patterns, and the unified memory budget work that decides whether a 30B model fits with the rest of the stack. If a local coding agent is on your roadmap and the path from a synthetic decode benchmark to a working production loop is the gap, that is the work we do.

FAQ

Does Qwen3-Coder 30B A3B fit on a 64GB M5 Pro? Yes for the model weights (16.8 GB on MLX at Q4) and yes for a 32K context KV cache at Q8 (3.7 GB). At 96K context the KV cache pushes total memory past the practical headroom for a single user with any other process running. For interactive agent use on 64GB, cap context at 48K and use prompt cache aggressively.

MLX or llama.cpp for a production agent loop? MLX if prompt cache hits dominate the workload and prefill is the bottleneck. llama.cpp if long generations dominate and speculative decoding pays off. For most coding agents in 2026, MLX is the slightly better default because prefix cache hit ratios in a well designed harness sit above 70 percent.

Is 3B active parameter quality enough for real coding work? For code completion, single file edits, and tool call routing, yes. For complex multi file refactors that require deep cross file reasoning, the 480B flagship or a frontier hosted model still leads on pass rate. The 30B A3B is the developer in the loop tier, not the autonomous PR generation tier.

What is the realistic developer hours per month break even versus a hosted frontier model? At a typical heavy agent user spend of $400 per month on Claude or GPT API calls, a 128GB M5 Max amortizes in roughly 10 to 14 months for a single user, faster for a small team sharing the box through an mlx-lm or llama-server endpoint.