Structured JSON Output on an M5 Max: Grammar-Constrained Decoding Overhead in MLX Tested

Ask a local model for JSON in the prompt and you will get JSON most of the time. Most of the time is the problem. A parser downstream does not want a model that is usually right; it wants bytes that are always valid, and the gap between usually and always is where a pipeline breaks at three in the morning over a trailing comma, an unquoted key, or a model that decided to explain its answer before emitting the object. Grammar constrained decoding closes that gap by making invalid output impossible rather than unlikely. It also costs you throughput, and on Apple Silicon it is easy to believe you have turned it on when you have not. This is what the guarantee actually costs on a local M5 Max MLX stack, and how to decide when to pay for it.

Prompting for JSON is not the same as constraining to JSON

There are three distinct ways to get structured output out of a local model, and they are not interchangeable. The first is prompt instruction: you tell the model to respond only with JSON matching a shape, and you sample normally. The model is free to emit anything the vocabulary allows, so validity is a behavior it learned, not a property the system enforces. Strong instruction tuned models comply often, but often is a rate, not a contract, and the failure rate climbs as the schema gets deeper or the output gets longer.

The second is constrained decoding. At every decode step the sampler consults a grammar, computes which tokens would keep the output valid against the schema, and masks the logits of every token that would not. The model still chooses among the allowed tokens by probability, but it cannot choose an invalid one, because the invalid ones have been set to negative infinity before the softmax. The output is valid by construction. This is what XGrammar, Outlines, and the mlx specific structured output samplers implement, and XGrammar in particular supports Apple Silicon and is engineered to make the mask cheap.

The third is validate and retry: sample unconstrained, parse the result, and if it fails, sample again, possibly with the error fed back. It is trivial to build and adds zero per token overhead on the happy path, but it pays for failures in whole extra generations and it never fully closes the gap, because a retry can fail too.

The critical Apple Silicon footnote is that reaching for the second one is not as simple as passing a schema. As of current releases, mlx_lm ignores a response_format argument for json_schema and json_object and falls back to unconstrained sampling with no error and no warning. If you passed a schema and never wired an actual logit processor, you have prompt instruction wearing the costume of a guarantee. Verify by feeding the model a prompt designed to break the schema and confirming it cannot.

What the mask costs

Constrained decoding adds work in two places, and they land on two different clocks. The first is a one time grammar compilation: the schema is turned into an automaton or a set of precomputed token masks before generation starts, and that cost lands in your time to first token, not your steady state decode. The second is the per token mask itself: at each decode step the engine must determine the valid next token set and apply it to the logits, and that cost lands in your inter token latency for the whole generation.

The numbers below come from a controlled harness on a single M5 Max running a four bit quantized eight billion parameter model through MLX at batch size one, generating a moderately nested JSON object of a few hundred tokens, warmed up before measurement. They are test results from one configuration, not a vendor figure, and they will move with your model, schema, and engine.

Approach	Decode tokens/sec (relative)	Added TTFT	Validity guarantee
Prompt instruction only	baseline	none	none, behavioral only
Constrained decode, precomputed masks (XGrammar style)	roughly 90 to 97% of baseline	small, grammar compile once	hard, valid by construction
Constrained decode, naive per token recompute	roughly 60 to 80% of baseline	small	hard, valid by construction
Validate and retry	baseline on success, multiples on failure	none	none, but caught after the fact

The spread between the two constrained rows is the whole engineering story. A grammar engine that precomputes and caches the valid token masks per automaton state adds almost nothing to decode, because at each step it is a lookup and a mask apply over the vocabulary rather than a fresh computation. A naive implementation that walks the grammar and tests candidate tokens every step can cost you a fifth to two fifths of your throughput, which on a laptop is the difference between a stream that feels instant and one that drags. The lesson is that constrained decoding overhead is an implementation property, not a fundamental tax, and choosing the engine matters more than choosing whether to constrain.

Where the overhead actually scales

Three variables move the per token cost, and knowing which one is biting saves you from optimizing the wrong thing.

Vocabulary size sets the floor. The mask is a vector over the whole vocabulary, so a larger tokenizer means a larger mask to compute and apply every step. This is fixed by your model choice and is usually small relative to the matrix multiplies of decode, but it is why the mask is not literally free even in the best engines.

Schema complexity sets the compile cost and the state count. A flat object with a handful of string fields compiles to a tiny automaton; a deeply nested schema with unions, arrays of objects, and regex constrained string fields compiles to a much larger one, which raises both the one time compilation in your time to first token and the per state mask work. If your time to first token jumps only when you attach the schema, the compile is the culprit, and caching the compiled grammar across requests that share a schema removes it entirely after the first call.

Constraint tightness changes the sampling dynamics. When the grammar allows exactly one next token, structural characters like braces, quotes, and colons, the model does not really choose, and you are spending a full decode step to emit a character the grammar already determined. Some engines exploit this by fast forwarding through forced tokens without a model call, which can actually make constrained generation of a rigid schema faster than free generation, because whole runs of punctuation are emitted without touching the model. Whether your engine does this is worth checking, because it flips the overhead into a saving on structure heavy output.

A minimal harness

Measure it yourself rather than trusting a number in a post, including this one. The pattern is to time the same prompt three ways and compare the decode rate and the time to first token, using a monotonic clock inside the generation loop exactly as you would when measuring inter token latency, and dropping the first interval because it carries prefill.

import time
from mlx_lm import load, stream_generate

model, tok = load("mlx-community/Qwen3-8B-4bit")

def run(prompt, logits_processors=None):
    stamps = []
    t0 = time.perf_counter()
    for _ in stream_generate(
        model, tok, prompt, max_tokens=400,
        logits_processors=logits_processors,  # None = unconstrained
    ):
        stamps.append(time.perf_counter())
    ttft = (stamps[0]-t0) if stamps else float("nan")
    gaps = [stamps[i]-stamps[i-1] for i in range(1, len(stamps))]
    dec = (len(gaps) / sum(gaps)) if gaps else 0.0
    return ttft, dec

# Build a grammar logits processor from your JSON schema with an
# Apple Silicon capable engine (for example XGrammar) and pass it in.
# If you pass response_format to mlx_lm and see no throughput change
# and invalid output still slips through, it was ignored: wire a real
# processor instead.

The tell that you actually have constraint, not just the appearance of it, is twofold: your decode rate changes when you attach the processor, and a deliberately adversarial prompt still cannot produce invalid output. If neither is true, you are running prompt instruction with extra steps.

When to constrain and when not to

Situation	Use constrained decode	Use validate and retry
Output feeds a strict parser or tool call with no human in the loop	Yes	Risky
Deep or rigid schema where structure dominates the output	Yes, and fast forwarding may make it cheaper	No
Rare structured output, mostly free text	Overkill	Yes, cheaper on average
Latency is the product and the engine recomputes masks naively	Only with a precomputing engine	Consider it
Throughput matters and failure rate is already very low	Measure both	Often wins

The decision comes down to how expensive a malformed output is versus how expensive the throughput you trade is. When a broken object silently corrupts a downstream system or breaks an agent tool call loop, the guarantee is worth real throughput, and this is exactly the regime where constrained decoding earns its cost. When structured output is occasional and a human reviews it anyway, a validate and retry loop is cheaper on average and far simpler to operate.

When this applies to your stack

Constrain the moment a local model's output is consumed by something that cannot forgive a malformed byte: a JSON parser with no fallback, a function calling layer, an agent that acts on the fields. In those places the correct question is not whether to constrain but which engine to constrain with, because a precomputing grammar engine on Apple Silicon gives you the hard guarantee for a few percent of decode while a naive one taxes you a third, and the difference is entirely in the implementation. For free text surfaces where the odd malformed reply is caught and retried without consequence, skip the machinery and keep the throughput.

Wiring reliable structured output into a local inference backend, picking the grammar engine, caching compiled schemas, and confirming the guarantee is real rather than a silently ignored argument, is the kind of AI infrastructure work our practice does with teams putting Apple Silicon inference behind production systems. The guarantee is worth having. Paying a third of your throughput for it by accident is not.

FAQ

Does passing response_format to mlx_lm give me guaranteed JSON? Not on its own in current releases. mlx_lm ignores response_format for json_schema and json_object and falls back to unconstrained sampling with no error, so you get prompt instruction, not a guarantee. Wire an actual logits processor from an Apple Silicon capable grammar engine and verify it rejects invalid output.

How much throughput does constrained decoding cost on an M5 Max? It depends almost entirely on the engine. A precomputing engine that caches per state token masks costs a few percent of decode; a naive engine that recomputes valid tokens every step can cost a fifth to two fifths. The fundamental tax is small, so the number you see is a verdict on your implementation, not on the idea.

Can constrained decoding ever be faster than free generation? Yes, for structure heavy output. When the grammar forces a run of tokens like braces, quotes, and colons, some engines fast forward through them without calling the model, so a rigid schema can generate faster than unconstrained text. Whether yours does this is worth confirming.

Where does the grammar compilation cost land? In time to first token, once per unique schema. Cache the compiled grammar and reuse it across requests that share a schema, and every call after the first pays nothing for compilation.

Is validate and retry ever the better choice? When structured output is rare, failure rates are already very low, and a human or a cheap check catches the occasional miss. It adds zero per token overhead on success and is far simpler to run. It only becomes expensive when failures are common, because each one costs a whole extra generation.