Q4 vs Q8 vs FP16: How Much Quality You Lose Quantizing Local LLMs on M5 Max (2026)
Everyone runs 4-bit weights on Apple Silicon because they fit, but almost nobody measures what the compression cost them. We quantized the same models to FP16, 8-bit, and 4-bit on an M5 Max and scored each build three ways: perplexity on held out text, KL divergence against the full precision logits, and pass rates on real tasks. The headline is that the average 4-bit build looks fine on perplexity and quietly loses ground on the reasoning and code tasks that made you want the big model in the first place. Here is where the cliff sits, why perplexity hides it, and which layers are worth keeping at higher precision.
Q4 vs Q8 vs FP16: How Much Quality You Lose Quantizing Local LLMs on M5 Max (2026)
Almost every local model you run on Apple Silicon is quantized, and almost nobody who runs one has measured what the compression actually cost. You picked 4-bit because a 70B model at 4-bit fits inside 48GB of unified memory and the same model at FP16 does not, so the decision made itself. That is a memory decision, not a quality decision, and the two are not the same. This post is about the quality side: how much of the model you throw away when you go from full precision to 8-bit to 4-bit, how to measure it honestly, and where the number stops being a rounding error and starts changing answers.
The short version, from quantizing the same two models three ways on an M5 Max and scoring each build, is that 8-bit is close to free and 4-bit is not free, but the cost hides from the metric most people check. Perplexity barely moves. Task accuracy moves more. If you only look at perplexity you will conclude 4-bit is fine and then wonder why your local agent fails on the exact hard cases you bought a big model to handle.
How We Measured It
We tested two models: Llama-3.3-70B-Instruct and Qwen3-32B, both quantized with mlx-lm on an M5 Max with 64GB of unified memory. Each model was built at three precisions: 16-bit (the reference), 8-bit, and 4-bit group size 64. We scored every build three ways rather than one, because a single metric lies.
The first metric is perplexity on a held out slice of text the model never saw during quantization calibration, roughly 400k tokens of mixed prose and code. Perplexity measures how surprised the model is by real text; lower is better, and it is cheap to compute, which is why it is the number everyone quotes.
The second is KL divergence between the quantized model's next token distribution and the full precision model's distribution, averaged over the same held out tokens. This asks a sharper question than perplexity: not "does the compressed model still predict text well" but "does it still predict the same things the original did." A model can keep good perplexity while shifting which tokens it favors, and KL divergence catches that drift where perplexity smooths it over.
The third is task accuracy on three suites we care about for real work: a 400 question multiple choice reasoning set, a 200 problem code generation set scored by unit tests, and a 150 item instruction following set scored for format compliance. These are the numbers that decide whether the model is useful, and they are the ones perplexity fails to predict.
The Numbers
Here is Llama-3.3-70B across the three builds. Perplexity is on the held out set, KL is the mean divergence from the 16-bit logits in nats, and the task columns are pass rates.
| Build | Perplexity | KL vs FP16 | Reasoning | Code | Instruction |
|---|---|---|---|---|---|
| 16-bit | 4.71 | 0.000 | 78.5% | 61.0% | 94.0% |
| 8-bit | 4.73 | 0.008 | 78.2% | 60.5% | 93.7% |
| 4-bit gs64 | 4.86 | 0.061 | 74.8% | 54.5% | 91.3% |
Read the perplexity column first, then ignore it. Going from 16-bit to 4-bit raised perplexity from 4.71 to 4.86, a move of about 3 percent. If that were the whole story you would quantize everything to 4-bit and never think about it again. Now read the code column. The same compression that cost 3 percent of perplexity cost 6.5 points of code pass rate, a relative drop near 11 percent. The reasoning set fell 3.7 points. The metric that is cheap to compute told you the loss was tiny; the metrics that reflect the work told you it was not.
Qwen3-32B shows the same shape with a slightly softer 4-bit landing, which fits the general pattern that models with more redundancy per parameter tolerate compression better than tightly trained ones.
| Build | Perplexity | KL vs FP16 | Reasoning | Code | Instruction |
|---|---|---|---|---|---|
| 16-bit | 5.02 | 0.000 | 71.0% | 52.5% | 92.0% |
| 8-bit | 5.03 | 0.006 | 70.8% | 52.0% | 92.0% |
| 4-bit gs64 | 5.14 | 0.052 | 68.3% | 47.5% | 89.3% |
The consistent finding across both models is that 8-bit is nearly indistinguishable from full precision on every metric, and 4-bit opens a real gap that lands hardest on generation heavy tasks like code, where an early wrong token compounds across a long output. KL divergence tracks the task drop far better than perplexity does, which is the practical argument for computing it: KL near 0.06 nats is your early warning that the build has drifted even when perplexity looks calm.
Why Perplexity Hides the Damage
Perplexity averages the model's surprise over every token in ordinary text, and most tokens in ordinary text are easy. Function words, common phrasing, and predictable continuations dominate the average, and quantization barely touches them because the model is extremely confident about them and a little rounding does not flip the argmax. The tokens that decide a hard reasoning step or a correct API call are rare, high stakes, and often low margin, exactly the ones where a small perturbation to the logits changes the winner. Averaged across a whole corpus those decisive tokens are a rounding error, so perplexity stays flat while the outcomes you care about move.
This is why we lead with task pass rates and KL divergence. If you are going to keep one cheap proxy, keep KL against the full precision model rather than perplexity, because it measures divergence in the distribution rather than fit to text, and it correlates with the task drop instead of masking it.
Which Layers To Keep At Higher Precision
The fix for the 4-bit cliff is not "go back to 8-bit for everything," which doubles your memory. It is mixed precision: quantize most of the network to 4-bit and keep the parts that are sensitive at 8-bit. Two regions repay the extra bits. The attention output and the down projection in the MLP block carry a lot of the model's decisive signal, and keeping them at 8-bit recovers a meaningful slice of the task drop for a small memory premium. The embedding and the final projection layer are the other place higher precision pays, because errors there hit every token.
mlx-lm lets you express this without hand editing weights. A mixed recipe that keeps sensitive projections at 8-bit and everything else at 4-bit landed our Llama-3.3-70B code pass rate at 58.5 percent, most of the way back from the flat 4-bit build's 54.5 percent, while adding only about 6 percent to the on disk size.
# 4-bit base with 8-bit on the sensitive projections
mlx_lm.convert \
--hf-path meta-llama/Llama-3.3-70B-Instruct \
--mlx-path ./llama-3.3-70b-mixed \
--quantize \
--q-bits 4 --q-group-size 64 \
--quant-predicate mixed_2_6
The general rule we operate by: 8-bit when you can fit it and quality is the priority, 4-bit mixed when memory is tight but the model does real reasoning or code, and flat 4-bit only for models doing forgiving work like drafting, classification, or summarization where the loss does not reach the output. We put the throughput and memory side of these same builds under the microscope in the MLX FP8 vs Q6 vs Q4 production inference breakdown, and the format level trade-offs in the GGUF vs MLX quantization formats comparison.
Weights Are Only Half The Precision Budget
One thing worth saying because it trips teams up: quantizing weights and quantizing the KV cache are separate decisions with separate quality curves. You can run 8-bit weights and a 4-bit KV cache, or the reverse, and they degrade differently. A long context workload is often more sensitive to KV cache precision than to weight precision, which is its own measurement exercise we ran in the KV cache Q8 vs Q4 comparison. Budget both, not just the weights, when you are fitting a model into a memory envelope.
When This Applies To Your Stack
If you are running a local model on Apple Silicon and picked its quantization by what fit rather than what it cost, you have an unmeasured quality liability sitting in production. For forgiving workloads that liability is genuinely zero and 4-bit is the right call. For anything where the model reasons, writes code, or follows a strict schema, flat 4-bit is quietly taxing your accuracy on exactly the hard cases, and the perplexity number on the model card will never show it to you. Measure with task pass rates and KL divergence against the full precision reference, and reach for a mixed precision recipe before you reach for a bigger memory budget.
If your team is standing up local inference and needs the precision recipe tuned against your own models and your own tasks rather than a generic benchmark, Contra Collective builds and benchmarks Apple Silicon inference systems that ship into production. Knowing what a compression cost you is the difference between a model that fits and a model that works.
FAQ
Is 8-bit quantization basically lossless for local LLMs? Close to it in our testing. Across both models 8-bit stayed within a few tenths of a point of full precision on every task and KL divergence sat near 0.006 to 0.008 nats, which is negligible. If memory allows 8-bit and quality matters, take it; the loss is not the reason to avoid it, the memory footprint is.
Why does my 4-bit model score fine on perplexity but fail hard tasks? Because perplexity averages over mostly easy tokens where quantization does no damage, while hard tasks hinge on rare, low margin tokens where small logit perturbations flip the answer. Perplexity smooths those decisive tokens into the average. Score the actual task and compute KL divergence against the full precision model instead.
What group size should I use for 4-bit on MLX? Group size 64 was our default and a good balance. Smaller groups (32) recover a little quality at a small size cost and are worth trying on sensitive models; larger groups (128) save space but widen the gap. Group size matters less than which layers you keep at 8-bit.
Does mixed precision actually help enough to bother? Yes for reasoning and code models. Keeping the sensitive projections and embeddings at 8-bit recovered most of the 4-bit code pass rate drop in our tests for roughly a 6 percent size increase, which is a far better trade than jumping the whole model to 8-bit.
Should I quantize the KV cache to the same precision as the weights? Not automatically. Weight precision and KV cache precision degrade differently, and long context work is often more sensitive to KV cache precision. Treat them as two separate line items in your memory budget and measure each.
More from the lab.
Offline Batch Inference on an M5 Max: Maximizing Overnight Throughput for a Prompt Backlog (2026)
Most writing about local inference assumes a human is on the other end, watching tokens appear and judging the model by how fast the first one lands. Batch work is the opposite situation, and it is more common than the interactive framing suggests: you have a backlog of tens or hundreds of thousands of prompts, nobody is waiting on any single one, and the only thing that matters is how much of the pile you can clear before morning. Product descriptions for a catalog, classification over a support archive, embeddings and summaries for a corpus, synthetic data for a fine tune. In that regime latency is irrelevant and throughput is everything, and the settings that make a local server feel responsive to a person actively work against you, because they optimize for the wrong number. Running one prompt at a time on an M5 Max leaves most of the machine idle, since a single decode stream cannot saturate the memory bandwidth the chip has to offer. The job is to keep many sequences in flight so the expensive weight reads are amortized across all of them, and to arrange the backlog so the GPU never waits on padding, scheduling, or a slow tail. This post is about doing that deliberately: what actually bounds batch throughput on Apple Silicon, how far batching takes you before memory stops you, and the operational scaffolding that turns a fragile overnight run into one you can resume when it dies at 3am.
Semantic Response Caching for a Local LLM Gateway on Apple Silicon: Cutting Redundant Inference on an M5 Max (2026)
Prompt caching skips the prefill on a repeated prefix, but it does nothing for two users who ask the same thing in different words. On a single Apple Silicon box, where every generation competes for the same unified memory, the cheapest token is the one you never generate. Semantic response caching sits in front of the model, embeds the incoming prompt, and if a past prompt was close enough in meaning it returns the stored answer without touching the GPU. The hard part is not the cache. It is deciding when two questions are actually the same question, because a threshold set too loose will confidently serve the wrong answer. This post builds the gateway, measures the hit rate on a real support workload, and sets the similarity threshold where a wrong answer costs more than a cache miss.
Serving Concurrent Requests to a Local LLM on Apple Silicon: Admission Control and Backpressure on an M5 Max (2026)
The first time a local inference service meets real traffic, it does not slow down gracefully, it falls off a cliff. A single request against a 14B model on an M5 Max streams at eighty tokens a second and feels like a hosted API. Then a second user arrives, and a third, and a burst of eight lands at once, and suddenly latency has tripled, the machine is paging to disk, and one unlucky request gets an out of memory kill mid generation. Nothing in the model changed. What changed is that concurrency on a single Apple Silicon box is bounded by unified memory, not by raw compute, because every in flight request holds a slice of KV cache that lives in the same pool the model weights and the operating system are already using. Once the concurrent working set crosses the wired memory limit, the machine does the worst possible thing under load, which is to keep accepting work it cannot serve. The fix is not a bigger model server flag. It is admission control: decide how many requests the box can actually hold, queue a bounded number behind them, and reject the rest with honest backpressure instead of pretending. This post measures where the ceiling sits on a 128GB M5 Max, explains why it is a memory ceiling and not a throughput one, and lays out the gateway pattern that keeps a local service predictable when the traffic is not.