Running Mixture of Experts Models on M5 Max: Why Total Parameters Decide Memory, Not Active Parameters (2026)

Mixture of Experts is the architecture that made big open models feel affordable to run. Instead of pushing every token through every parameter, a MoE model splits its feed forward layers into many experts and routes each token to a small handful of them. A router picks, say, two experts out of a hundred, and only those experts do work for that token. The headline number that comes out of this is the active parameter count: Llama 4 Scout has 109B parameters total but activates roughly 17B per token, Qwen3-Coder-30B has 30B total but activates about 3B, and newer designs push the ratio further with 256 experts and single digit billions active. The pitch writes itself. You get the knowledge capacity of a large model at the compute cost of a small one.

The pitch is true about compute and false about memory, and on an Apple Silicon box that difference decides whether the model runs at all. We loaded several MoE models on a 64GB M5 Max under mlx-lm to watch two numbers at once: resident memory and decode tokens per second. The pattern is consistent and it is the thing to internalize before you buy a machine. Total parameters set the floor on how much memory the model needs to sit resident. Active parameters set how fast it decodes. Sizing by the active count, which is the number everyone quotes, is how you end up with a model that will not load.

Why Every Expert Has To Be Resident

The router in a MoE model is data dependent. Which experts fire for a given token is decided at inference time by the token itself, and across a real prompt every expert gets used. You cannot know in advance which experts a conversation will need, so you cannot keep only a favored subset in memory and evict the rest without paying for it later. That means the full set of expert weights has to be resident, or reachable fast enough that the model does not stall waiting for one.

This is the crux. A dense 70B model reads all 70B parameters every token, so its memory footprint and its compute cost track each other. A MoE model decouples them. It reads only the active experts each token, which is cheap, but it must hold all experts in memory, which is not. The quantized weight footprint of a MoE model is a function of its total parameters, exactly like a dense model of the same total size. A 109B MoE at 4-bit occupies roughly the same memory as a hypothetical 109B dense model at 4-bit. The MoE part buys you speed, not space.

MLX makes the speed side land cleanly because it was built for unified memory. Tensors live in one pool that the GPU addresses directly, so there is no copy from host to device, and for MoE that matters because only the active expert weights get pulled into the compute pipeline each token. The inactive experts sit in the same pool doing nothing until a token routes to them. They cost memory the whole time and cost compute only when selected. That is the architecture working as intended, and it is also the reason the memory bill is set by the total, not the active, count.

The Two Numbers, Measured

Here is what the split looks like on a 64GB M5 Max, single stream, 4-bit weights, a short context so the KV cache is not yet the story. Treat the throughput figures as our test results on this specific machine, not as guarantees for yours.

Model	Total params	Active params	4-bit resident weights	Decode tokens per sec
Dense 70B	70B	70B	about 40 GB	10 to 12
MoE, Scout class	109B	17B	about 58 GB	26 to 34
MoE, Qwen3-Coder class	30B	3B	about 17 GB	55 to 70
Dense 8B	8B	8B	about 5 GB	60 to 75

Read the table across, not down. The Scout class MoE needs more memory than the dense 70B, roughly 58GB against 40GB, because it has more total parameters. On a 64GB machine that is genuinely tight once you add a KV cache and the reserve the system needs, which is a point we will come back to. But it decodes two to three times faster than the dense 70B, because each token only reads the 17B of active experts rather than the full 70B. The 30B MoE is the sweet spot for a 64GB box: it fits comfortably at 17GB and decodes in the range of a dense 8B model while carrying far more parameters worth of knowledge. That combination, small model speed with large model breadth, is the entire reason to reach for MoE locally.

The failure mode hides in the same table. A team reads that the Scout class model activates 17B parameters and reasons that it should run in the memory an 8B or a 13B dense model needs. The active count says small. The resident weights column says 58GB. The model either refuses to load or loads and immediately pushes the machine into swap, at which point decode collapses by an order of magnitude, which is the same cliff we mapped in the unified memory oversubscription analysis. The fix is not a setting. It is sizing by the correct number.

Why Decode Is Fast: Memory Bandwidth, Not Flops

Local decode is bound by memory bandwidth, not arithmetic. Generating one token means reading the weights the token needs out of memory and doing a comparatively small amount of math on them. On a dense model you read every weight every token, so decode speed is roughly the model size divided by the memory bandwidth. On a MoE model you read only the active experts plus the always on attention and router weights, so the effective bytes moved per token track the active parameter count, and decode speed tracks that smaller number.

That is why the Scout class MoE, with 17B active, decodes near what a 17B dense model would rather than near a 70B, even though it occupies more memory than the 70B. The bandwidth of the M5 Max is fixed. What changes between architectures is how many bytes you have to move per token, and MoE moves the active slice. This is also why the throughput advantage of MoE is real and not a benchmark artifact: it comes straight out of the bandwidth bound nature of decode, the same physics we leaned on when reasoning about dense model quantization in the Q4 versus Q8 versus FP16 quality piece.

Sizing A MoE Model So It Fits

Budget the residency the same way you would for a dense model of the total size, then enjoy the speed as a separate gift. The resident footprint is the quantized total weights, plus the KV cache at your longest context and highest concurrency, plus a reserve for macOS and your own applications.

def moe_resident_gb(total_params_b, bits, kv_cache_gb, os_reserve_gb=8):
    # quantized weights: every expert is resident, so use TOTAL params
    weight_gb = total_params_b * (bits / 8)
    return round(weight_gb + kv_cache_gb + os_reserve_gb, 1)

# Scout-class 109B MoE, 4-bit, ~6 GB KV cache at a working context
print(moe_resident_gb(109, 4, 6))   # -> ~68.5 GB, over a 64 GB machine

# Qwen3-Coder-class 30B MoE, 4-bit, ~3 GB KV cache
print(moe_resident_gb(30, 4, 3))    # -> ~26.0 GB, comfortable on 64 GB

The important line in that snippet is total_params_b. Feed it the total, never the active count. If the result is over your working headroom, you have the usual levers before buying a bigger machine: drop to a smaller MoE, quantize the KV cache to buy back context, or cap context length so the cache stays bounded. The one lever that does not exist is pretending the active count is the memory number.

Expert Streaming Is Real But It Is A Throughput Trade

There is a way to run a MoE model whose total weights exceed your memory: expert streaming, which keeps the experts on the internal drive and pages only the active experts for each token into a small memory cache. Recent work has run a 122B parameter MoE quantized to 3-bit on a 16GB Mac this way, with the model occupying about 54GB on disk and a resident footprint that peaks near 9GB. That is a genuinely impressive result and it changes what is possible on small machines.

It also changes the performance profile completely, and you should reach for it with eyes open. Streaming turns every token into a potential disk read, because if the experts a token needs are not in the cache they have to come off the drive before decode can proceed. For batch or throughput tolerant work, where you care about tokens per hour and not tokens per second, that trade can be worth it. For an interactive feature where a user is watching output stream, the cold expert stalls are exactly the latency spikes that make a local model feel broken. Streaming is the tool for fitting a model that otherwise would not fit, not the tool for making a resident model faster. If your machine can hold the total weights, hold them, and use streaming only when the alternative is not running the model at all. The general shape of choosing between MoE architectures for a backend is where we started in the working with MoE models breakdown.

When MoE Wins And When It Does Not

Situation	Reach for	Why
64GB M5 Max, want large model breadth at usable speed	30B class MoE at 4-bit	Fits at about 17GB, decodes like a small dense model
Need the biggest knowledge capacity and have 64GB to spend	Scout class MoE, watch the cache	Fits only just, so budget KV cache carefully
Latency sensitive interactive feature, limited memory	Small dense model, not streamed MoE	Streaming stalls break interactive latency
Throughput batch job that must run a huge model on little RAM	Streamed MoE off SSD	Trades tokens per second for fitting at all
Uniform load, memory to spare, simplest ops	Dense model near your memory ceiling	No router variance, predictable footprint

The row that trips teams up is the second one. A Scout class MoE on a 64GB machine is not a comfortable fit, it is a careful one, and the KV cache is what eats the remaining headroom during a long session. Size it as if you are running a 109B model, because for the purpose of memory you are.

When This Applies To Your Stack

If you are putting a local model behind a product feature, MoE is often the right architecture because it delivers large model quality at small model decode speed, which is the combination that makes local inference feel responsive. The catch is entirely in the sizing. The active parameter count is a compute and speed property. The total parameter count is a memory property. Provision the machine on the total, confirm the KV cache at your real peak context still fits, and you get the speed for free. Provision on the active count and the model will not load, or worse, it will load and swap on the first long conversation and present as a hang rather than an error.

If your team is choosing between a dense model and a MoE model for a backend and wants the memory budget, the decode targets, and the fallback behavior designed against real hardware rather than a spec sheet, Contra Collective builds AI infrastructure that fits the machine it runs on. The active parameter count is the number that sells the model. The total parameter count is the number that decides whether it runs.

FAQ

Does a MoE model use less memory than a dense model? No. A MoE model uses roughly the same memory as a dense model of the same total parameter count, because every expert has to be resident. What it uses less of is compute per token, since only the active experts run. Memory tracks total parameters, speed tracks active parameters.

Can I run a 109B MoE model on a 64GB M5 Max? Just barely, and only with care. At 4-bit the weights alone are close to 58GB, which leaves little room for the KV cache and the system reserve. It works for short and medium contexts and gets tight fast at long context. A 30B class MoE is the more comfortable choice on 64GB.

Why does my MoE model decode so much faster than a dense model of the same size? Because decode is memory bandwidth bound and a MoE model only reads its active experts each token. A 109B MoE with 17B active moves about 17B parameters worth of bytes per token, so it decodes near a 17B dense model even though it occupies memory like a 109B model.

What is expert streaming and should I use it? Expert streaming keeps experts on the SSD and pages only the active ones into a small cache per token, which lets a machine run a model larger than its memory. It works well for throughput tolerant batch work and poorly for interactive use, because cold experts cause disk read stalls. Use it only when the model otherwise would not fit.

How do I size a machine for a MoE model? Add the quantized total weights, the KV cache at your longest context and highest concurrency, and a reserve for the OS and your applications. Use the total parameter count for the weight term, never the active count, and confirm the sum fits your working headroom.