Memory Bandwidth Is the Local Inference Bottleneck: M5 Pro vs Max vs Ultra Tokens per Second (2026)

If you are choosing an Apple Silicon machine to run models on, the spec that predicts your token generation speed is not the GPU core count and it is not the neural engine. It is memory bandwidth. Single stream decode, the token by token generation that happens when one person is chatting with a local model, is bound by how fast the chip can read weights out of unified memory, and almost nothing else. We measured decode across the M5 Pro, M5 Max, and M5 Ultra and put the numbers next to the rated bandwidth so you can see how tightly they track. Here is the result first.

Headline Comparison

Chip	Rated memory bandwidth	8B model, 4-bit, decode tokens/sec	8B model, 8-bit, decode tokens/sec	32B model, 4-bit, decode tokens/sec
M5 Pro	350 GB/s	61	34	16
M5 Max	700 GB/s	118	66	31
M5 Ultra	1400 GB/s	214	124	58

Read the bandwidth column and any tokens per second column together, top to bottom, and the pattern is almost embarrassing in how clean it is. The M5 Max has twice the rated bandwidth of the M5 Pro and generates roughly twice the tokens per second on the same model and quantization. The M5 Ultra doubles the Max again and again roughly doubles the rate. The absolute numbers fall as the model or the weight precision grows, because there are more bytes to read per token, but the ratio between chips holds. That is the whole thesis: decode speed scales with GB/s, not with cores. The methodology, the arithmetic behind it, and the two cases where this stops being true are below.

Why Decode Is Memory Bound

Generating one token with a dense transformer requires reading every active weight in the model exactly once, multiplying it against a small activation, and moving on. The amount of arithmetic per weight is tiny, on the order of two floating point operations per parameter, while the weight itself has to be fetched from memory. That ratio, a lot of bytes moved for very little math done, is the definition of a memory bound workload. The GPU cores spend most of their time waiting for weights to arrive rather than computing, so the ceiling on tokens per second is set by how fast memory can feed them.

The back of the envelope is simple enough to keep in your head. Take the model size in bytes, which for a quantized model is roughly the parameter count times the bits per weight divided by eight, and divide the chip bandwidth by it. An 8B model at 4-bit is about 4 GB. On a 700 GB/s M5 Max that is a theoretical ceiling near 175 tokens per second, and real world decode lands at 60 to 70 percent of that once you account for the KV cache reads, attention, and overhead that the simple model ignores. Our measured 118 tokens per second on that configuration is exactly where the roofline says it should be. This is also why the quantization format you choose changes decode speed directly: fewer bits per weight means fewer bytes to read per token, which is why every 4-bit column in the table beats its 8-bit neighbor by close to two to one.

What This Means for Buying a Machine

The practical consequence is that you should spec the tier for the bandwidth, then check that the memory capacity holds the model you want resident. Core count barely enters the single user decode decision. If your workload is one person at a time talking to a model, the M5 Ultra is not faster because it has more GPU cores; it is faster because it has more bandwidth, and you would get almost the same decode uplift from bandwidth alone if such a part existed.

Your primary use	Spec on
Single user local chat and coding assistant	Highest bandwidth tier you can justify
Running the largest model that fits	Memory capacity first, then bandwidth
Serving several users at once	GPU cores and bandwidth together
Long context sessions	Bandwidth plus capacity for the KV cache

The capacity caveat matters because bandwidth is useless if the model does not fit. A 32B model at 8-bit needs around 32 GB just for weights before the KV cache, so the machine has to have the unified memory to hold it resident, or you spill and the numbers collapse. Capacity decides which models you can run at all; bandwidth decides how fast they decode once they fit. Long context makes this sharper, because the KV cache grows with sequence length and competes for the same memory and the same bandwidth, a dynamic we covered in the KV cache and context length analysis.

Where the Bandwidth Rule Breaks: Batching

The clean bandwidth story is a single stream story. The moment you serve more than one request at a time, it starts to bend, and eventually it breaks in your favor. Batching works because the weights you read out of memory can be reused across several sequences in the same forward pass. You pay the bandwidth cost of reading a weight once and amortize it over every request in the batch, so the arithmetic per byte climbs. As the batch grows, the workload crosses from memory bound to compute bound, and now the GPU cores that were idle during single stream decode become the thing that matters.

Concurrent requests	Bound by	What raises throughput
1	Memory bandwidth	GB/s
2 to 8	Transitional	Bandwidth and cores both help
16 and up	Compute	GPU cores

This is why an M5 Ultra pulls further ahead than its bandwidth ratio alone would predict once you batch: it brings both more bandwidth and more cores, and batched serving rewards the cores that single user chat ignores. If you are building a service that fans out to many users on one box, do not spec it the way you would spec a personal machine. The core count you could ignore for one user is now load bearing.

Where It Breaks Again: Prefill

The other exception is prefill, the phase where the model reads your prompt before it generates anything. Prefill processes all the prompt tokens in parallel, which is a compute bound matrix multiply rather than a bandwidth bound stream, so it behaves like batching does. A machine with more GPU cores chews through a long prompt faster even at batch size one. This is why time to first token on a long document does not track the decode numbers in the headline table. If your workload front loads huge prompts, retrieval augmented generation stuffing thousands of tokens of context, long document analysis, code review over a big file, the prefill compute matters as much as decode bandwidth, and the two phases want different things from the chip.

Reading Your Own Numbers

If your measured decode is far below what the roofline predicts for your chip, the usual causes are a model that does not fully fit and is partially spilling, thermal throttling on a sustained run, or a runtime that is not using the GPU path efficiently. The first two are capacity and cooling problems, not bandwidth problems, and they show up as numbers that start near the prediction and then sag. The roofline is the sanity check: compute model bytes, divide bandwidth by it, expect 60 to 70 percent of that, and if you are far under, look for the leak. Energy tells the same story from another angle, since a memory bound workload spends most of its watts moving bytes rather than computing, which is why decode efficiency in joules per token also tracks the memory system more than the compute units.

Test Setup

Every number came from the same methodology so the cross chip ratios are honest. We ran a llama family 8B model and a 32B model through MLX at 4-bit and 8-bit, decoding at batch size one from a short fixed prompt, and recorded steady state tokens per second after the first hundred tokens so warmup does not skew the figure. Each machine had enough unified memory to hold the model and KV cache fully resident, so nothing spilled. Runs were kept short enough to measure sustained decode without crossing into thermal throttling, which is a separate effect we treat on its own. Bandwidth figures are the rated peak for each tier; real achievable bandwidth is lower, which is already baked into the 60 to 70 percent efficiency you see against the roofline. The point of the test was not to crown a single fastest number but to show the relationship, and the relationship is that decode tokens per second rises with memory bandwidth in near lockstep.

When This Applies to Your Stack

If you are putting local inference into a real backend, size the hardware to the shape of your load. One user at a time is a bandwidth problem, so buy the bandwidth and confirm the model fits. Many users at once is a compute problem, so the GPU cores you could ignore for a single stream become the thing that sets your ceiling, and you should spec and batch accordingly. Long prompts are a prefill problem that lives on the compute side too, so a retrieval heavy workload wants cores for the prompt and bandwidth for the generation. The mistake we see most often is buying core count for a single user assistant and being disappointed that decode did not speed up, or buying for single stream and then trying to serve a team off it and hitting a wall. Match the chip to the phase that dominates your workload.

If your team is designing a local or hybrid inference tier and wants the hardware sized to the actual request pattern rather than a spec sheet, Contra Collective builds AI infrastructure that models the bandwidth and compute budget against your real traffic, so the box you buy is the box the workload needs.

FAQ

Does GPU core count affect local LLM speed on Apple Silicon? For single stream decode, barely. Token by token generation is bound by memory bandwidth, so a chip with more cores but the same bandwidth generates at nearly the same rate. Core count starts to matter when you batch several requests together or process long prompts during prefill, both of which are compute bound rather than bandwidth bound.

How do I estimate tokens per second before buying a Mac? Compute the model size in bytes, roughly the parameter count times bits per weight over eight, then divide the chip's rated bandwidth by it for a theoretical ceiling. Expect real decode to land at about 60 to 70 percent of that. An 8B 4-bit model is around 4 GB, so a 700 GB/s part predicts a ceiling near 175 tokens per second and delivers closer to 118.

Why is a 4-bit model faster than an 8-bit model at decode? Because decode speed is set by how many bytes have to be read from memory per token, and a 4-bit weight is half the bytes of an 8-bit weight. Halving the bytes per token roughly doubles the decode rate on a memory bound workload, which is why every 4-bit result beats its 8-bit counterpart by close to two to one.

When does memory bandwidth stop being the bottleneck? When you batch multiple requests or run prefill on a long prompt. Batching reuses each weight read across several sequences, so the workload becomes compute bound and rewards GPU cores. Prefill processes all prompt tokens in parallel as a compute bound matrix multiply, so it also depends on cores rather than bandwidth.

Should I buy an M5 Ultra for a personal coding assistant? Only if you want the raw decode speed its bandwidth provides, since for one user the Ultra's extra GPU cores mostly sit idle during generation. The Ultra is genuinely faster at single stream decode because it has more bandwidth, not because it has more cores. If you also plan to serve several people or process very long prompts, the cores start earning their place.