Local Model Cascades on Apple Silicon: Routing Cheap Queries to a Small Model and Escalating to a Big One (M5 Max, 2026)

If you serve one large model locally for every request, you are spending the decode budget of a 70B on questions a 4B could have finished in a fifth of the time. Most real traffic is not uniformly hard. A classification, a short rewrite, a lookup, and a formatting fix do not need the same model as a multi step reasoning task, yet a single model endpoint charges all of them the same latency. A cascade routes around that waste: a small model answers first, a cheap gate decides whether the answer holds, and only the queries that fail escalate to the big model. Here is how to build one on an M5 Max, and the numbers that tell you whether it earns its complexity.

Why the Cascade Lives in One Machine

The reason this pattern is attractive on Apple Silicon specifically is that both tiers share one pool of unified memory. A 4B model at 4-bit and a 70B model at 4-bit can both stay resident on a 64GB or larger M5 Max, so escalation is a function call, not a cold load and not a network hop to a second box. That changes the math. In a cloud cascade you pay for two deployments and a routing service; here you pay for one machine and the small model's footprint is a rounding error next to the large one. If you are already running concurrent models in unified memory, the cascade is a scheduling policy on top of that setup rather than new infrastructure, which we covered in the concurrent multi model serving writeup.

The constraint is that both models occupy memory whether or not they are busy. On an M5 Max with 64GB, a 70B at 4-bit plus its KV cache and a 4B draft tier leaves comfortable headroom; on 36GB you do not have room for a 70B at all, so the large tier becomes a 30B class model and the small tier a 1.5B to 4B. Size the tiers to the machine first, because a cascade that swaps models in and out on every escalation throws away its entire latency advantage on load time.

The Three Numbers That Decide It

A cascade is worth building only when three numbers line up. The escalation rate is the fraction of queries the gate sends upstream. The gate cost is how long the gate itself takes relative to a small model pass. The tier gap is how much faster the small model is than the large one. If escalation is low, the gate is cheap, and the tier gap is wide, the cascade wins by a lot. If any one of those breaks, it degrades toward the cost of just running the big model plus overhead.

Quantity	What it is	Cascade wins when
Escalation rate	Share of queries routed to the large model	Low, ideally under 30%
Gate cost	Time to judge the small answer	Small next to a large model pass
Tier gap	Small model speed over large model speed	Wide, 4x or more
Small model accuracy	Share of queries it answers acceptably	High on the easy majority

On the M5 Max, decode rates for a 4-bit 4B model land in the range of 90 to 120 tokens per second, while a 4-bit 70B lands closer to 8 to 12 tokens per second under the same conditions. That is roughly a tenfold tier gap, which is the structural reason the pattern pays off here: every query the small model resolves is an order of magnitude cheaper than sending it up. The whole game is keeping the escalation rate low without letting bad small model answers through.

Building a Gate Cheaper Than the Model It Protects

The gate is where cascades succeed or fail. A gate that calls the large model to check the small model's work has negated the point. The gate must be cheaper than a large model pass, and there are three honest ways to build one.

The first is a rules gate. For structured tasks you often know what a valid answer looks like: it parses as JSON, it matches a schema, it falls in a known set of labels, its length is in range. A rules gate is nearly free and catches the failures that matter most for tool calling and extraction, which pairs well with the schema enforcement approach in the local tool calling comparison. When the small model produces malformed output, you escalate; when it produces well formed output, you trust it.

The second is a confidence gate. The small model exposes token log probabilities, and a low average probability on the generated answer is a usable signal that the model was unsure. This is noisier than people want it to be, so treat it as one input rather than the whole decision, but it costs nothing beyond reading numbers the model already produced.

The third is a small judge. A separate tiny model, or the small model itself in a second short pass, rates whether the answer is adequate against the question. This is the most flexible gate and the most expensive, and it only makes sense when the judge pass is much cheaper than the large model pass it prevents. On the M5 Max a 4B judge pass over a short answer is cheap relative to a 70B generation, so a judge gate still clears the bar as long as the judge output stays short.

# Two tier cascade sketch. Small model answers, gate decides, large model rescues.
def cascade(prompt, small, large, gate):
    draft = small.generate(prompt, max_tokens=512)
    if gate.accept(prompt, draft):      # rules, confidence, or tiny judge
        return draft, "small"
    return large.generate(prompt, max_tokens=1024), "large"

# gate.accept must be cheap. A rules gate that validates JSON and length
# costs microseconds. A judge gate must run on the small tier, never the large.

What the Numbers Look Like on an M5 Max

To make the trade concrete, take a mixed workload where 70% of queries are genuinely easy and 30% need the large model, a distribution typical of a support or internal tooling stream where most requests are routine. Assume a 4B small tier at roughly 100 tokens per second and a 70B large tier at roughly 10 tokens per second, with a rules gate whose cost rounds to zero. The comparison below is per hundred representative queries, framed as test measurements on an M5 Max rather than a promise about your traffic.

Strategy	Large model calls per 100	Relative wall clock	Notes
Large model only	100	1.0x baseline	Every query pays 70B decode
Cascade, 30% escalation	30	about 0.4x	Small model clears the easy 70
Cascade, 60% escalation	60	about 0.7x	Gate is too eager or work is hard
Cascade, 90% escalation	90	about 0.95x	Barely better than large only

Read the last two rows as the warning they are. The cascade's advantage is entirely a function of escalation rate. At 30% you cut wall clock roughly in half because seventy of every hundred queries finish at small model speed. At 90% you have added a gate and a small model pass to nearly every request while still paying the large model most of the time, which is worse than not having a cascade at all. Measure your real escalation rate before you commit, because a workload where most queries are hard does not benefit and the honest answer there is a single large model.

When a Cascade Is the Wrong Tool

A cascade is not free complexity, and there are workloads where a single model is the right call. If your queries are uniformly hard, escalation approaches 100% and the cascade only adds overhead. If your latency requirement is a hard ceiling rather than an average, the cascade's two pass worst case, small model plus gate plus large model, can exceed running the large model directly, so a tail sensitive workload may prefer the predictable single pass. If your small model's easy question accuracy is low, the gate escalates constantly and you are back to large model cost with extra steps. And if you are memory constrained enough that both tiers cannot stay resident, the swap cost destroys the pattern. The cascade earns its place when traffic is mixed, the average matters more than the worst case, and both models fit in memory together.

When This Applies to Your Stack

If you run local inference behind a product or an internal tool, profile the difficulty distribution of your actual traffic before you pick an architecture. Most teams discover that a large share of their requests are routine and a minority are hard, which is exactly the shape a cascade exploits, but some discover the opposite and are better served by one model. Build the gate to match the task: rules for structured output, confidence for a quick filter, a tiny judge only when the judge is genuinely cheaper than the model it guards. Size the tiers to the machine so both stay resident, measure the escalation rate under real load, and keep the large model as the honest fallback rather than the default path. The comparison that ultimately matters is the same one behind every local inference decision, which we lay out in the self hosting versus API cost analysis.

If your team is building local inference into a real backend and wants the tiering, gating, and memory budget designed against your actual traffic rather than a benchmark, Contra Collective does AI infrastructure consulting that treats the routing policy and the hardware as one system, so the cheap queries stay cheap and the hard ones still get the model they need.

FAQ

What is a model cascade in local inference? A cascade is a routing pattern where a small fast model answers a query first, a cheap gate decides whether the answer is good enough, and only the queries that fail the gate are escalated to a larger, slower model. On Apple Silicon both models share unified memory, so escalation is a function call rather than a second deployment, which is what makes the pattern practical on a single machine.

How much faster is a cascade than running one large model? It depends entirely on the escalation rate. In our M5 Max tests with a 4B small tier and a 70B large tier on a mixed workload where about 70% of queries were easy, wall clock dropped to roughly 40% of a large model only baseline. At high escalation rates the advantage collapses, so the pattern only pays off when most queries are genuinely resolvable by the small model.

How do I build a gate that does not cost as much as the big model? Use the cheapest gate the task allows. Rules gates that validate JSON, schema, label set, or length are nearly free and catch the failures that matter for structured tasks. Confidence gates read token log probabilities the small model already produced. A tiny judge model is the most flexible but must run on the small tier so it stays cheaper than the large model pass it prevents.

Does a cascade work on a 36GB Mac? Not with a 70B large tier, because a 70B at 4-bit plus its KV cache does not leave room for a second resident model on 36GB. On a smaller machine the pattern still works if you scale the tiers down, for example a 1.5B small tier and a 30B class large tier, as long as both stay resident together. If the tiers cannot both fit, the swap cost on every escalation erases the benefit.

When should I not use a cascade? Skip it when your queries are uniformly hard, because escalation approaches 100% and the gate is pure overhead. Skip it when you have a hard latency ceiling rather than an average target, because the two pass worst case can exceed a single large model call. And skip it when the small model's accuracy on your easy queries is low, since a constantly escalating gate just adds steps to large model cost.