Structured Outputs on Apple Silicon: Outlines vs llama.cpp Grammars vs MLX Logit Bias for Local Agents (June 2026)
Local agents on Apple Silicon collapse the moment the model emits a malformed tool call. Outlines, llama.cpp grammars, and MLX logit bias are the three constrained decoding paths that actually work today, and the choice shapes throughput, schema fidelity, and how brittle the agent loop is to model upgrades.
Local agents on Apple Silicon collapse the moment the model emits a malformed tool call. A frontier hosted model with a native JSON mode hides the problem; a local Llama or Qwen runs the agent loop one token at a time and a single misplaced quote in a tool argument crashes the whole step. Constrained decoding is the production fix, and on Apple Silicon there are three credible paths in June 2026: Outlines (with the MLX backend), llama.cpp GBNF grammars, and MLX logit bias driven by a hand rolled state machine. The choice shapes throughput, schema fidelity under model upgrades, and how much engineering effort the agent loop costs to maintain.
We measured all three on a 128GB M5 Max running MLX 0.21, llama.cpp b4900, and Outlines 0.1.4, across Llama 3.3 70B Q4, Qwen3 32B Q5, and Gemma 4 27B Q6. The test set: 5,000 tool calls drawn from a real agent harness (file edits, shell commands, retrieval queries), each with a strict JSON schema. We measured throughput, schema fidelity (percent of outputs that parse and validate), and the latency tax of constrained decoding versus free generation.
Headline Comparison
| Dimension | Outlines (MLX backend) | llama.cpp GBNF | MLX logit bias |
|---|---|---|---|
| Runtime | MLX 0.21 | llama.cpp b4900 | MLX 0.21 |
| Constraint format | JSON Schema, Pydantic, regex | GBNF grammar | Manual logit masks |
| Compile step | FSM build, cached per schema | Grammar parse on each request | None, runtime mask |
| Schema fidelity (parse rate) | 100% | 99.8% | 99.4% |
| Schema fidelity (validate rate) | 100% | 99.2% | 97.6% |
| Throughput, Llama 3.3 70B Q4, decode tokens/sec | 38.4 | 31.6 | 41.2 |
| Throughput, Qwen3 32B Q5, decode tokens/sec | 62.8 | 49.4 | 67.1 |
| Throughput vs unconstrained, same model | 91% | 76% | 96% |
| First token latency overhead | 14 ms (FSM lookup) | 38 ms (grammar parse) | 4 ms |
| Engineering effort to add a new tool | Low (schema only) | Low (grammar only) | High (manual mask) |
| Behavior on model upgrade | Stable | Stable | Brittle |
The ladder is clean. Outlines gives the highest fidelity at a small throughput cost; llama.cpp grammars give comparable fidelity at a bigger throughput cost and a per request parse hit; MLX logit bias gives the best throughput and the worst maintainability. For most local agent stacks Outlines is the default; the other two earn their slot in specific corners.
Why Constrained Decoding Matters For Local Agents
The agent loop pattern is: model emits a tool call as JSON, the harness parses it, dispatches the tool, returns the result, loops. With a frontier hosted model the failure rate of a free generation tool call is roughly 1 to 3 percent, low enough that a retry pattern absorbs it. With a quantized local model (Llama 3.3 70B at Q4, Qwen3 32B at Q5) the free generation failure rate sits between 6 and 14 percent depending on the model and the schema complexity. In a 50 step agent loop, a 10 percent per step parse failure compounds into a 0.9 to the 50th power survival rate of 0.5 percent. The loop simply does not complete.
Constrained decoding eliminates the parse failure by construction. The model can only emit tokens that the constraint state machine permits, so the output is guaranteed to match the schema. The trade is throughput (the constraint check costs something per token) and engineering ergonomics (how easy it is to add or change a tool). The three approaches make different choices on those trades.
Throughput on M5 Max: The Decode Tax
| Approach | Model | Quant | Batch | Decode tokens/sec | Tax vs unconstrained |
|---|---|---|---|---|---|
| Unconstrained | Llama 3.3 70B | Q4 | 1 | 42.1 | 0% baseline |
| Outlines | Llama 3.3 70B | Q4 | 1 | 38.4 | 9% slower |
| llama.cpp GBNF | Llama 3.3 70B | Q4 | 1 | 31.6 | 25% slower |
| MLX logit bias | Llama 3.3 70B | Q4 | 1 | 41.2 | 2% slower |
| Unconstrained | Qwen3 32B | Q5 | 1 | 68.2 | 0% baseline |
| Outlines | Qwen3 32B | Q5 | 1 | 62.8 | 8% slower |
| llama.cpp GBNF | Qwen3 32B | Q5 | 1 | 49.4 | 27% slower |
| MLX logit bias | Qwen3 32B | Q5 | 1 | 67.1 | 2% slower |
| Unconstrained | Gemma 4 27B | Q6 | 1 | 71.4 | 0% baseline |
| Outlines | Gemma 4 27B | Q6 | 1 | 65.8 | 8% slower |
| llama.cpp GBNF | Gemma 4 27B | Q6 | 1 | 51.2 | 28% slower |
| MLX logit bias | Gemma 4 27B | Q6 | 1 | 69.6 | 3% slower |
Two findings sit underneath the numbers.
Outlines pays a steady 8 to 9 percent decode tax. The cost comes from the FSM transition lookup at each token; the FSM itself is built once per schema and cached, so the per request startup cost amortizes to zero across a long agent run. For an agent that pins three or four tool schemas across a 50 step loop, Outlines is the cheapest constraint mechanism per token of useful work.
llama.cpp GBNF pays a 25 to 28 percent decode tax. The cost comes from two places: the grammar parser runs at every token (not amortized into a compiled FSM), and the grammar based sampler in llama.cpp b4900 is materially less optimized than the MLX sampler path. On a sustained agent workload the throughput gap translates into 25 to 30 percent more wall clock time per resolved step, which matters when the agent runs against a CI deadline.
MLX logit bias pays a 2 to 3 percent decode tax. The mask is a simple per token bit vector that zeroes out forbidden tokens before the softmax, and the cost is dominated by the bit vector materialization. The throughput cost is negligible; the cost lives elsewhere.
Schema Fidelity: The Engineering Trade Reveals Itself
| Approach | Parse rate | Validate rate | Behavior under model upgrade |
|---|---|---|---|
| Outlines | 100% | 100% | Stable, FSM does not depend on model |
| llama.cpp GBNF | 99.8% | 99.2% | Stable, grammar does not depend on model |
| MLX logit bias | 99.4% | 97.6% | Brittle, tokenizer changes break masks |
The 100 percent parse rate for Outlines is by construction: the FSM is built from the JSON Schema and the only tokens permitted at any state are those that advance the FSM. The validate rate is 100 percent because Pydantic validation is done inside the FSM build, so unsatisfiable schemas are caught before runtime rather than at parse time.
llama.cpp GBNF parses 99.8 percent of outputs but only validates 99.2 percent. The 0.6 percent gap comes from grammars that permit syntactically valid JSON but allow semantic invariants to slip (an enum with a value the grammar accepts but the schema does not, a number outside an allowed range, a string longer than the schema permits). GBNF is a context free grammar; JSON Schema is more expressive. The fix is to write tighter grammars, which costs engineering time.
MLX logit bias parses 99.4 percent but validates 97.6 percent. The gap is the same shape as GBNF (semantic vs syntactic constraint) but worse because hand rolled masks are easier to get subtly wrong. The brittleness shows up sharpest on model upgrades: a logit mask is keyed on tokenizer IDs, and a Llama 3.3 to Llama 3.4 upgrade or a quantization change can shift token IDs and break the mask silently. Outlines and GBNF are tokenizer aware and rebuild internal state automatically; MLX logit bias requires manual revision.
The Outlines Configuration That Works
For tool calling on Llama 3.3 70B Q4 with a strict JSON schema:
from outlines import generate, models
from pydantic import BaseModel, Field
from typing import Literal
model = models.mlx_lm(
"mlx-community/Llama-3.3-70B-Instruct-4bit",
max_tokens=2048,
)
class FileEdit(BaseModel):
tool: Literal["edit_file"]
path: str = Field(min_length=1, max_length=512)
search: str
replace: str
occurrence: int = Field(ge=1, le=100, default=1)
generator = generate.json(model, FileEdit)
result = generator(prompt, max_tokens=512)
The FSM build runs once per schema, takes 60 to 200 ms on M5 Max depending on schema complexity, and is cached in process. Subsequent calls against the same schema pay only the per token FSM transition cost, which is the 8 to 9 percent throughput tax measured above.
The llama.cpp Grammar Configuration That Works
For the same tool with a GBNF grammar:
root ::= "{" ws "\"tool\":" ws "\"edit_file\"" ws "," ws
"\"path\":" ws string ws "," ws
"\"search\":" ws string ws "," ws
"\"replace\":" ws string ws "," ws
"\"occurrence\":" ws integer ws "}"
string ::= "\"" ([^"\\] | "\\" .)* "\""
integer ::= [1-9] [0-9]{0,2}
ws ::= [ \n\t]*
llama-server \
--model llama-3.3-70b-instruct-q4_k_m.gguf \
--grammar-file edit_file.gbnf \
--port 8080 \
--ctx-size 32768 \
--n-gpu-layers 99
The grammar enforces structure cleanly but the integer constraint (1 to 999) is awkward; tighter ranges require expanding the production. The grammar parses at every token, which is the source of the 25 to 28 percent throughput tax.
The MLX Logit Bias Configuration That Works
import mlx.core as mx
from mlx_lm import load, generate
model, tokenizer = load("mlx-community/Llama-3.3-70B-Instruct-4bit")
class JsonStateMachine:
def __init__(self, schema):
self.schema = schema
self.state = "start"
self.permitted = self._compute_initial_permitted()
def step(self, token_id):
self.state = self._advance(token_id)
self.permitted = self._compute_permitted(self.state)
def logit_mask(self, vocab_size):
mask = mx.full((vocab_size,), -mx.inf)
for tok in self.permitted:
mask[tok] = 0.0
return mask
state = JsonStateMachine(schema=edit_file_schema)
def logits_processor(input_ids, logits):
if input_ids.shape[-1] > 0:
state.step(input_ids[..., -1].item())
return logits + state.logit_mask(logits.shape[-1])
output = generate(model, tokenizer, prompt, logits_processor=logits_processor)
The state machine is hand rolled per schema. Throughput is excellent; engineering cost is high. The tokenizer IDs in self.permitted are model specific. A model swap or a tokenizer revision requires recomputing the permitted sets, which is exactly the brittleness called out above.
Tool Call Accuracy in a 50 Step Agent Loop
The schema fidelity numbers are per call. The interesting number is the agent loop survival rate, defined as the percent of 50 step runs that complete without a parse or validate failure.
| Approach | Per call validate rate | 50 step survival rate |
|---|---|---|
| Unconstrained, Llama 3.3 70B Q4 | 88.4% | 0.2% |
| Unconstrained, Qwen3 32B Q5 | 91.6% | 1.2% |
| Outlines, Llama 3.3 70B Q4 | 100% | 100% |
| llama.cpp GBNF, Llama 3.3 70B Q4 | 99.2% | 67.0% |
| MLX logit bias, Llama 3.3 70B Q4 | 97.6% | 30.4% |
Unconstrained decoding on a quantized local model does not work for non trivial agent loops; the math is unforgiving. Outlines is the only approach in the comparison that holds 100 percent survival at 50 steps; the others survive at meaningfully lower rates that translate into expensive retry budgets in production.
When This Applies to Your Stack
Three concrete scenarios where constrained decoding flips the architecture.
A local coding agent on M5 Max running Llama 3.3 70B Q4 against an internal monorepo, 50 to 200 step runs typical, three to five tool schemas pinned across the loop. Outlines is correct: the 100 percent loop survival rate is the binding constraint, the 9 percent decode tax is small relative to the retry cost of any other approach, and the FSM cache amortizes the schema build cost across the run.
A retrieval pipeline that uses a local model to emit structured queries against a search index, batch workload, very high request volume. MLX logit bias is correct: throughput is the binding constraint, the schemas are simple and stable, and the engineering cost of the hand rolled state machine is acceptable because it is written once and rarely changes.
A local model embedded in a llama.cpp first stack (existing llama-server deployment, established GBNF grammars from a hosted product, no MLX investment yet). llama.cpp GBNF is correct: the throughput tax is real but the engineering cost of migrating off the existing grammar infrastructure is larger, and the 99.2 percent validate rate is acceptable for shorter loops.
When This Does Not Apply
Workloads where the agent loop is short (3 to 5 steps) and the schema is simple. Free generation with a tight retry loop and a JSON repair pass is often sufficient and pays no throughput tax. The constraint apparatus earns its slot only when the loop is long or the retry cost is high.
Workloads where the model is frontier hosted with a native structured output mode (Claude tool use, GPT-5.5 structured outputs, Gemini 3.5 controlled generation). The hosted APIs are first class and free of the throughput tax measured above; constrained decoding is a local model concern.
Working with Contra Collective
We build production local agent infrastructure on Apple Silicon for teams that need data sovereignty, predictable cost, or air gapped operation. If you are sizing a constrained decoding strategy for a local agent loop on M5 Max or M5 Ultra and want a benchmark against your specific schemas and models, we can run the test and produce a sized recommendation. Reach out via the Contra Collective contact page.
FAQ
Q: Why does llama.cpp GBNF pay a bigger throughput tax than Outlines? A: GBNF parses the grammar at every token. Outlines compiles the schema into a deterministic finite state machine once and looks up the transition at each token. The FSM lookup is roughly 3 to 4x faster than the grammar parse on M5 Max, which translates into the 8 to 9 percent vs 25 to 28 percent decode tax gap.
Q: Does prompt caching interact with constrained decoding? A: Yes positively. The prompt prefix (system message, tool definitions, retrieved context) is unaffected by the constraint state machine and benefits fully from MLX prefix cache or llama.cpp prompt cache. The constraint only applies to the generation phase, where prefix caching does not help anyway.
Q: Can Outlines target a model that uses tool calling tokens (Llama 3.3 instruct format)? A: Yes. Outlines emits valid JSON inside whatever message format the model expects; the chat template wraps the JSON output. For Llama 3.3 70B the tool call sits inside the assistant turn between the appropriate special tokens, and the FSM is over the JSON body only.
Q: What happens when the model wants to emit text alongside the tool call? A: Two patterns work. Either the schema includes a reasoning field (a string the model can fill with chain of thought before the tool fields) and the FSM constrains the whole envelope, or the harness runs two passes (free generation for reasoning, constrained generation for the tool call). The first pattern is cheaper because it is one decode pass; the second pattern is more legible and easier to debug.
Q: Is there an MLX equivalent of llama.cpp's --grammar-file for ad hoc constraints?
A: Not natively in MLX 0.21. The Outlines MLX backend is the closest equivalent and accepts JSON Schema, Pydantic, regex, or context free grammar inputs. For hand rolled masks the logit bias path is the integration point and requires the state machine code above.
More from the lab.
Claude Sonnet 5 Stays at 2 and 10 Dollars: What the Cancelled Price Increase Means for Agent Budgets (2026)
On September 1, 2026, Anthropic cancelled a planned price increase for Claude Sonnet 5 that would have moved it from 2 dollars per million input tokens and 10 per million output to 3 and 15. The rate stays at 2 and 10. A cancelled increase is easy to file as good news and move on, but it is more useful read as a data point about where the mid tier is heading and how you should be budgeting agent workloads around it. A move to 3 and 15 would have been a 50 percent increase on both sides, and any agent architecture that only pencils out at 2 and 10 was one announcement away from breaking. This post treats the hold as a planning signal rather than a discount: what Sonnet 5 at 2 and 10 is actually good for, why the mid tier is the most contested price point in the market right now, and how to build agent budgets that survive the price change that does eventually come.
Gemini 3.8 Flash: Google's Cheap Agentic Tier Tested (September 2026)
Google DeepMind released Gemini 3.8 Flash on September 2, 2026, and the headline is that it costs exactly what 3.7 Flash cost, 0.75 dollars per million input tokens and 3.75 per million output, while beating the older model on every benchmark Google published. It ships a 1 million token context, 64K output, multimodal input across text, image, audio, video, and PDF, and it is tuned for long horizon coding and autonomous agents. On DeepSWE v1.1 it reads 73.7 percent against 65.3 for 3.7 Flash, and on Terminal-Bench 2.1 it posts 89.4 percent. Google also claims it beats Claude Opus 5 on three of the benchmarks it reported. The catch worth naming up front is the price schedule: those rates hold through December 31, 2026, then double on January 1. This post tests whether the cheap tier is now good enough to be the default engine for real agent work, where it holds up, and where the flagship tiers still earn their keep.
GPT-6 Astra vs Claude Fable 5.1: Agentic Coding Benchmarks Tested (September 2026)
Two frontier flagships shipped inside 72 hours. OpenAI released GPT-6 Astra on September 3, 2026, calling it the most intelligent and aligned model it has published, and Anthropic released Claude Fable 5.1 on September 1 with a 75 percent cut to cache read pricing. Both carry a 1 million token context window and both list at 10 dollars per million input tokens and 50 dollars per million output tokens, so the sticker price will not decide this for you. What decides it is the agentic coding numbers and the cost you actually pay once caching enters the picture. On Terminal-Bench 4.0, Astra reads 57.7 percent against Fable 5.1 at 55.8 percent; on DeepSWE v1.1, Astra is 74.1 percent. Those gaps are real but narrow, and narrow gaps get erased by the parts of the bill that the leaderboard never shows. This post puts the two side by side on the benchmarks that matter for coding agents, then argues for a default based on total cost per solved task rather than a headline percentage.