NVFP4 on Apple Silicon: 4-bit Float Quantization on the M5 Max with MLX

NVFP4 started as a data center format. It is NVIDIA's 4-bit floating point encoding, designed around Blackwell class hardware, and for most of its life the local inference crowd treated it as something that happened on someone else's GPU. That framing is now out of date. The format has landed in Apple Silicon tooling, and the Ollama MLX preview benchmark that ran Qwen3.5-35B-A3B in NVFP4 on an M5 Max gave the community its first concrete look at how it behaves on a Mac. This post does the honest breakdown: what the format is, why 4 bits of float beats 4 bits of integer, and what actually runs on Apple Silicon today.

What NVFP4 actually encodes

A quantization format is a deal about how you spend bits. At 4 bits per weight you get 16 distinct values, and the entire game is choosing which 16 and how you map them back to real numbers at compute time. Integer formats like int4 pick 16 evenly spaced levels and attach a scale, often one scale per channel or per group of weights. Floating point formats like NVFP4 spend those 4 bits as a tiny float, in this case one sign bit, two exponent bits, and one mantissa bit, which the literature writes as E2M1. Because it has an exponent, the representable values are not evenly spaced. They cluster near zero and spread out toward the extremes, which matches how weight distributions actually look.

The part that makes NVFP4 more than just E2M1 is the scaling. NVFP4 groups weights into blocks of 16 and gives each block its own scale stored in FP8, specifically the E4M3 variant. That is the difference that matters. A per tensor integer scale has to cover the whole weight matrix with one number, so any single outlier stretches the range and starves the rest of the values of precision. A 16 element block scale in FP8 tracks local dynamic range so tightly that an outlier only degrades the 15 weights next to it, not the entire tensor. This is the same idea behind the microscaling formats, taken to a smaller block and a more precise scale.

NVFP4 versus MXFP4 versus int4

If you have run 4-bit models on a Mac you have almost certainly used int4 or an MLX group quantization scheme, and possibly MXFP4. Here is how the three line up on the axes that decide accuracy.

Format	Element type	Block size	Scale type	Practical read
int4 (grouped)	4-bit integer	32 to 128 typical	integer or FP16 per group	fast, mature, loses tails
MXFP4	E2M1 float	32	E8M0 (power of two)	OCP standard, coarse scale
NVFP4	E2M1 float	16	E4M3 (FP8)	tightest scale, best 4-bit accuracy

The story the table tells is that NVFP4 makes two choices that both push toward accuracy: a smaller block (16 instead of 32) and a scale that is a real FP8 float rather than a power of two. MXFP4's E8M0 scale can only represent powers of two, so it snaps the block scale to the nearest doubling, which is cheap in hardware but wasteful of range. NVFP4 spends more on the scale and gets meaningfully closer to the FP16 baseline on the models where 4-bit int quantization visibly hurts, which is usually the reasoning and long context cases where small errors compound. We walked through the broader format landscape in our comparison of GGUF and MLX quantization formats, and NVFP4 is the format that finally makes the accuracy argument for 4 bits without asking you to jump to 6 or 8.

Why the M5 changes the hardware side

Quantization is half the story. The other half is whether your hardware can multiply those low precision numbers quickly. This is where the M5 generation matters. Apple put Neural Accelerators inside every GPU core on the M5 family, which are dedicated matrix multiply units in the same spirit as the tensor cores on discrete GPUs, but wired directly into the unified memory pool. Low precision matrix multiply is precisely the workload those units exist to accelerate, so a 4-bit format that decodes into a compact matmul is a natural fit.

The measured effect shows up most in prompt processing. In controlled tests the M5 Max runs prefill roughly 35 to 40 percent faster than the M4 Max, because prefill is a big batched matmul over the whole prompt and that is what the accelerators chew through. Decode gains are smaller, closer to a high 20s percentage overall throughput improvement, because decode is one token at a time and stays largely bound by memory bandwidth rather than matmul throughput. We covered that bandwidth ceiling in detail in the memory bandwidth and tokens per second analysis, and NVFP4 does not change the decode ceiling; it changes how much accuracy you keep while living under it.

The numbers that exist so far

Be careful with early benchmarks, because the tooling is a preview and the coverage is narrow. The clearest public data point comes from the Ollama MLX backend benchmark on an M5 Max running Qwen3.5-35B-A3B, a Mixture of Experts model with a small active parameter count, in NVFP4. In that run prefill measured around 1,810 tokens per second and decode around 112 tokens per second, up from roughly 1,154 and 58 on the older Metal path. A separate int4 configuration on the same class of machine reported figures in the neighborhood of 1,851 prefill and 134 decode.

Read those two side by side and the tradeoff is visible. The int4 decode number is higher, because integer decode is a touch cheaper and more mature in the kernels, while NVFP4 is buying accuracy that a throughput benchmark does not measure. The honest way to frame this today is that NVFP4 is an accuracy play with competitive speed, not a speed play. If your workload is sensitive to quantization error, reasoning chains, structured extraction, long context recall, NVFP4 is worth the slightly slower decode. If you are serving short chat turns where int4 already passes your evals, int4 remains the pragmatic default. These are controlled test results on specific models and builds, not a promise about your prompts, so measure on your own traffic before committing.

What runs today, and what does not

The gap between what NVFP4 allows in principle and what you can load this afternoon is real. The MLX backend preview that carries the fastest NVFP4 path launched with support for a single architecture family, the Qwen3.5 line, and required a Mac with more than 32GB of unified memory. Smaller Macs stayed on the older Metal path with no change. Architecture coverage is expanding, but if your target model is not on the supported list you will fall back to a slower generic path or to a different format entirely.

# sketch of a local NVFP4 serving config, values you would set
runtime:        mlx backend (preview)
model:          Qwen3.5-35B-A3B-NVFP4
min_unified_mem: 32GB
block_size:     16          # NVFP4 micro-block
scale_dtype:    e4m3        # FP8 per-block scale
fallback:       int4 grouped if arch unsupported

The practical workflow is to check three things before you plan around NVFP4: whether your exact model architecture has a supported kernel path, whether your machine clears the memory floor, and whether an NVFP4 checkpoint exists or whether you would have to quantize one yourself. Any of those can be the blocker, and today the architecture path is the one that most often is.

When this applies to your stack

NVFP4 earns its place when you are running a capable open weights model locally and you have been forced to choose between int4 that slightly dents quality and 8-bit that eats memory and bandwidth. It gives you a 4-bit memory footprint with accuracy that lands much closer to the higher precision baseline, and on M5 class hardware the matrix accelerators keep the speed competitive. It is not the right default for every deployment yet, because coverage is uneven and int4 is still faster to decode, but it is the format to reach for when quantization error is showing up in your evals and you do not want to pay the memory cost of going wider.

If you are standing up local or on device inference and trying to decide which format and which hardware tier actually fit your accuracy and latency budget, this is the kind of tradeoff we work through with clients building AI infrastructure. The right answer depends on your models, your context lengths, and your tolerance for quantization error, and it is worth measuring rather than guessing.

FAQ

Is NVFP4 always more accurate than int4? On most models it holds accuracy closer to the FP16 baseline at 4 bits, because the FP8 block scale tracks dynamic range better than an integer scale. The gap is largest on reasoning and long context work and smallest on short, easy prompts where int4 already passes.

Does NVFP4 run faster than int4 on an M5 Max? Not usually on decode. Early tests show int4 decoding a little faster, while NVFP4 trades that for accuracy. Prefill is strong for both because the M5 Neural Accelerators accelerate the batched matmul.

Do I need a specific Mac to use it? The fastest NVFP4 path in the MLX preview required more than 32GB of unified memory and, at launch, supported a limited set of architectures. Check your model's support before planning around it.

How is NVFP4 different from MXFP4? Both use the E2M1 element type, but NVFP4 uses 16 element blocks with an FP8 scale, while MXFP4 uses 32 element blocks with a power of two scale. NVFP4's finer, more precise scaling is why it holds accuracy better.