M5 Ultra Mac Studio With 512GB: Running 120B Class Models in FP16 Locally (2026)

The Mac Studio with M5 Ultra ships on September 22, 2026, and the configuration that matters for local inference is the 512GB one. It is the first consumer machine whose unified memory can hold a 120B class model at full precision with enough left over for a real context window. That crosses a line no prior Apple Silicon box crossed. It is also the kind of headline that hides a bandwidth story underneath, so here is the memory math first, then what the throughput actually looks like, then the honest cases where you should buy something cheaper.

The Memory Math That Sets the Ceiling

Start with weights, because that is what decides whether a model fits at all. At FP16, every parameter is two bytes, so the weight footprint is roughly two times the parameter count in gigabytes.

def weight_gib(params_billion, bytes_per_param):
    return params_billion*bytes_per_param

fp16 = 2
print(weight_gib(70, fp16))    # 140 GiB for a 70B model
print(weight_gib(120, fp16))   # 240 GiB for a 120B class model

Weights are only the floor. Every token of context you hold adds to the KV cache, and the cache grows with context length, layer count, and head dimension. On a 120B class model a working context window can add tens of gigabytes on top of the 240 for weights. Add the runtime, the framework overhead, and the operating system, and a 120B FP16 setup with a long context lands in the high 200s to low 300s of gigabytes. That is why 512GB is the number: 256GB could technically hold the weights but leaves no comfortable room for context, and 128GB machines were never in the conversation for 120B at full precision. This is the same headroom argument we made for dense versus mixture of experts models in the Qwen3 235B MoE vs Llama 3.3 70B on M5 Ultra comparison, where capacity, not compute, is the thing that decides what you can even load.

Prefill, Decode, and Why Bandwidth Is the Real Constraint

Fitting a model and running it fast are different problems. Once the weights are resident, decode speed is governed by memory bandwidth, because generating each token means streaming the whole active weight set through the compute units once. A 120B FP16 model moves roughly 240GB per token of work, so even enormous bandwidth yields a modest tokens per second at full precision. Prefill is the friendlier phase: it is compute bound and parallel, and it is where the M5 generation actually changed the picture. Apple's own MLX and M5 research describes the M5 Neural Accelerators as dedicated matrix multiplication units, and matmul is exactly what prefill is made of, so time to first token on long prompts improves in a way decode does not.

You can see the same shape in the tooling. When the MLX backend landed in Ollama earlier in 2026, the published numbers on an M5 Max running a 35B class mixture of experts model showed prefill jumping from about 1,150 to 1,810 tokens per second and decode from about 58 to 112, a much larger relative gain on the prefill side. Scale the model up to 120B FP16 and the asymmetry gets sharper: prefill stays usable because it is compute bound and the accelerators help, while decode is capped by how fast 240GB can be streamed per token. The lived experience on the 512GB box is a machine that ingests a long document quickly and then generates at a deliberate pace, which is fine for batch and analysis workloads and less fine for a low latency chat agent.

When FP16 at 120B Is Worth It, and When It Is Not

The clean case for the 512GB machine at full precision is when precision itself is the requirement. Numerically sensitive work, evaluation harnesses where you want the reference behavior of the model rather than a quantized approximation, research on a model's exact outputs, fine tuning experiments that need full precision activations: these are the workloads that justify holding 240GB of FP16 weights instead of a quantized copy. If your goal is faithful behavior and you cannot tolerate the small quality drift that quantization introduces, the 512GB box is the only consumer way to get it for a 120B class model.

The case against is throughput per dollar. For most production style serving, a well chosen 4 bit or 8 bit quantization of the same model runs on far less memory, streams a quarter to a half of the bytes per token, and therefore decodes several times faster, at a quality level that is often indistinguishable on real tasks. We laid out that tradeoff in detail in the GGUF vs MLX quantization formats guide, and the summary is that quantization usually buys you more usable tokens per second than raw precision buys you quality. If you are serving a model to users, a quantized 120B on a 256GB machine, or even a smaller high quality model on a 128GB machine, will frequently beat the 512GB FP16 setup on the metric that matters, which is responses per second per dollar. Our general M5 Ultra inference numbers and the broader setup live in the M5 Ultra local AI inference walkthrough.

FAQ

Can the 512GB M5 Ultra really run a 120B model in FP16? Yes. At FP16 a 120B class model needs roughly 240GB for weights, and 512GB of unified memory holds that plus a real context window and runtime overhead with headroom. Smaller configurations cannot do it at full precision.

Will it be fast? Prefill will be quick because the M5 Neural Accelerators help the matmul heavy ingestion phase. Decode will be deliberate, because generating each token streams the full 240GB of weights and is bandwidth bound. Expect strong document ingestion and modest generation speed at full precision.

Should I buy 512GB or run a quantized model on less memory? Buy 512GB FP16 if you need reference precision: evaluation, research, numerically sensitive work. If you are serving users, a quantized copy on a smaller machine usually gives more responses per second per dollar at comparable quality.

Does more unified memory increase decode speed? No. Memory capacity decides what fits; memory bandwidth decides decode speed. A larger model on a bigger machine is not faster per token, it is just able to load.

When This Applies to Your Stack

The 512GB M5 Ultra is a capability unlock, not a speed unlock. It lets a 120B class model live on a desk at full precision for the first time, which is exactly right for evaluation, research, and precision sensitive experiments, and usually the wrong tool for high throughput serving, where quantization on cheaper hardware wins on responses per second per dollar. Decide by asking whether you are paying for faithful behavior or for throughput, because those two goals point at different machines.

If you are sizing local inference hardware for a team, or deciding where model workloads should live between an Apple Silicon box and cloud GPUs, that capacity versus bandwidth tradeoff is the core of the decision. Contra Collective does AI infrastructure consulting for teams running inference in production, and getting the hardware sizing right early saves a great deal of money later.