Reproducible Local LLM Inference on an M5 Max: Why Temperature Zero Is Not Deterministic

You set temperature to zero, you pin the seed, you send the identical prompt twice, and the model returns two different answers. Not always, and usually not by much, but often enough that a snapshot test fails intermittently, a semantic cache misses when it should hit, or a nightly regression suite goes red on a diff that touched only a comment. The reflex is to distrust your own harness, because everyone was taught that greedy decoding is deterministic. Greedy decoding is deterministic given identical logits. The logits are the thing that is not identical, and understanding why is the difference between chasing a ghost and shipping a reproducible pipeline.

Two different problems wear the same costume

There are exactly two sources of run to run variation in a local generation, and they need completely different fixes, so the first job is telling them apart.

The first is sampling nondeterminism. When temperature is above zero the model draws from a probability distribution, and that draw uses a random number generator. If the generator is not seeded, or is seeded from wall clock time, every run picks different tokens. This one is easy: seed the generator and set temperature to zero so decoding is greedy, meaning it always takes the highest probability token. Most people stop here and are surprised the problem does not fully go away.

The second is numerical nondeterminism, and it survives everything you just did. Even in pure greedy decoding with a fixed seed, the logits themselves can change between runs because floating point addition is not associative. The sum a plus b plus c can land on a different last bit depending on the order the hardware adds the terms, and the order is not fixed. It depends on how the runtime tiled the matrix multiply, how many rows were in the batch, and which Metal kernel was selected for that particular shape. When two candidate tokens have logits that are close, a one bit wobble is enough to flip which one is the argmax, and greedy decoding then commits to a different token and the generation diverges from that point forward. The math is correct in both runs. It is just not bit identical.

Where each source comes from, and what actually fixes it

Source	Trigger	Fixable?	The fix
RNG sampling	temperature above zero, unseeded generator	fully	fix the seed, or set temperature to zero for greedy
Reduction order	matmul tiling and accumulation order	partially	fix batch size, pin threads, accept residual float wobble
Batch invariance	request batched with different neighbors	partially	serialize the request, or use a batch invariant kernel path
Kernel selection	runtime picks a kernel per input shape	partially	pin the runtime version, keep shapes stable, disable autotuning
Quantization dequant	low bit formats dequantize per block	negligible on its own	keep the same quant file; the format is deterministic, the accumulation is the wobble

The pattern worth internalizing: sampling nondeterminism is fully removable, and everything below it in the table is a numerical effect you can shrink but not always drive to zero on general purpose hardware. Note that quantization does not add nondeterminism by itself. A Q4 weight file dequantizes the same way every time; what varies is the accumulation into higher precision that happens afterward, which is the same reduction order problem, not a property of the format.

The measurement, on an M5 Max

Here is what the two failure modes look like in practice. The harness runs an 8B class instruct model quantized to Q4 through a local server on an M5 Max, sends one hundred fixed prompts, and repeats each prompt ten times under different configurations. The metric is the exact match rate: the fraction of prompt pairs whose generations are byte for byte identical across repeats. These are harness figures to show the shape of the effect, not a guaranteed number for your model or runtime.

Configuration	Exact match rate	What is happening
Temperature 0.7, unseeded	near zero	pure sampling variation, every run differs
Temperature 0.7, fixed seed, single request at a time	high but not perfect	seed pins sampling; reductions still wobble
Temperature 0, single request at a time	very high	greedy plus no batching neighbors is the stable case
Temperature 0, under concurrent load	noticeably lower	requests get batched with different neighbors run to run

The jump from the first row to the third is the part people expect. The drop in the last row is the one that generates bug reports. Nothing about your request changed. What changed is that under load the server batched your prompt alongside a different set of other prompts than last time, the batch had a different number of rows, the matmul tiled differently, and the reduction order shifted. This is why a model that is perfectly reproducible on your laptop in a quiet loop becomes flaky the moment it is behind a busy batching server. The nondeterminism is not in the model. It is in the concurrency.

A configuration that gets you as close as the hardware allows

The working recipe has three layers, applied in order of how much they buy you.

import mlx.core as mx

# Layer 1: kill sampling nondeterminism.
mx.random.seed(20260818)      # pin the RNG explicitly, do not rely on defaults
TEMPERATURE = 0.0             # greedy: always take the argmax token

# Layer 2: keep the numerics stable for the reduction path.
#   pin the runtime version so kernel selection does not change under you,
#   keep prompt and batch shapes consistent so the same kernel is chosen,
#   and avoid autotuning that picks a different tiling per shape.

# Layer 3: control batching, the biggest lever for a server.
MAX_BATCH = 1                 # serialize when you need bit-for-bit repeats

Layer one removes sampling variation entirely and is non negotiable for reproducibility; if you need diversity, get it from a controlled seed sweep rather than an unseeded generator, which keeps the diversity itself reproducible. If you have to run above temperature zero, keep the rest of the sampler settings fixed too, because a threshold tuned at one temperature says nothing at another. Layer two pins the version and the shapes so the runtime does not silently swap kernels on you between deploys. Layer three is the one that actually decides whether a production server is reproducible: batching your request with a fixed set of neighbors, or with no neighbors at all, is what stabilizes the reduction order. Serializing to a batch of one is the sledgehammer, and it costs you throughput, so the real engineering choice is whether reproducibility is worth that throughput on this endpoint. For an evaluation harness or a snapshot test, yes. For a high traffic assistant, almost never, and you should design the test around the variance instead.

What you can guarantee, and what you cannot

Be honest with yourself about the ceiling. On a single machine, with a pinned runtime, fixed seed, greedy decoding, and a batch of one, you can get to bit for bit reproducible generations for the same prompt, and that is enough for snapshot testing and for a cache that keys on exact output. What you cannot promise for free is reproducibility across two different machines, across a runtime upgrade, or across a change in batch composition, because all three change the numerical path. The mature move is to stop treating any two generations as needing to be byte identical and instead assert on what actually matters: that the extracted answer, the parsed field, or the tool call is the same, not that all one thousand tokens match. A test that pins the answer and tolerates token level wobble is both more honest about the hardware and far less flaky, and it keeps passing through a runtime bump that a byte comparison would break on noise.

When this applies to your stack

Reach for strict reproducibility exactly where a byte difference has a cost: golden file tests for prompts, an exact match cache, a compliance trail that has to replay a past generation, or a regression gate you want to trust. In those places, run greedy with a pinned seed and serialize the request, accept the throughput hit on that path, and pin your runtime version so a silent kernel change does not reopen the problem. Everywhere else, design for the variance instead of fighting it: assert on the parsed result rather than the raw tokens, seed your diversity so even your randomness is reproducible, and treat the numerical wobble as the ordinary behavior of floating point hardware rather than a bug in your stack. The teams that stay calm about this are the ones who separated the two sources early and fixed the sampling problem completely while bounding the numerical one deliberately.

Getting a local pipeline reproducible enough to test and audit, without paying the serialize everything tax on every endpoint, is the kind of infrastructure work Contra Collective takes on when a local model has to be trusted in production rather than demoed. The seed is the easy part. Knowing which guarantees the hardware can actually keep is the engineering.

FAQ

Does temperature zero make a local model deterministic? It removes sampling nondeterminism, which is the largest and easiest source, but not numerical nondeterminism. In pure greedy decoding the logits can still shift in their last bits because floating point reductions are not run in a fixed order, and when two candidate tokens are close that shift can flip the argmax and diverge the generation. Temperature zero is necessary for reproducibility, not sufficient.

Why does my model become nondeterministic only under load? Because a busy server batches your request with a different set of neighbors each time, and the batch shape changes how the matrix multiplies tile and accumulate, which changes the reduction order and the low order bits of the logits. Alone in a quiet loop your prompt is stable; batched with different company it is not. Serialize the request or use a batch invariant kernel path when you need repeats.

Is quantization the reason my outputs vary? No. A quantized weight file dequantizes identically every run, so the format itself is deterministic. What varies is the higher precision accumulation after dequantization, which is the same reduction order effect that also hits unquantized models. Keeping the same quant file does not introduce variance and switching formats does not remove it.

How do I write a stable test against a local model? Assert on the extracted answer or parsed field rather than on the full token stream. A byte for byte comparison is fragile because a runtime upgrade or a batch change will wobble the low order bits, while the meaningful output stays correct. Pin the seed and use greedy decoding to remove sampling variance, then test the thing you actually care about.

Can I get identical outputs across two Macs? Not reliably for free. Different chips, runtime versions, or Metal kernel choices change the numerical path, so cross machine bit identity is not something to promise. Within one machine and one pinned runtime you can reach bit for bit repeats; across machines, assert on the parsed result instead.

This post reports results from a controlled local test harness, not a production benchmark, and exact match rates will vary with your model, quantization, runtime version, and batching; validate reproducibility on your own stack before relying on it for tests or caching.