Offline Batch Inference on an M5 Max: Maximizing Overnight Throughput for a Prompt Backlog (2026)

Almost everything written about running a model locally assumes someone is watching the output, which means it optimizes for time to first token and responsiveness. Batch work inverts that. You have a backlog: a hundred thousand product descriptions to generate, a support archive to classify, a corpus to summarize, and no human is waiting on any individual result. The only figures that matter are aggregate tokens per second and the cost per million tokens you process, and the settings that make a server feel snappy to a person are exactly the settings that leave your M5 Max half idle on a job like this. This is how to run the machine flat out instead.

Why One Prompt at a Time Wastes the Machine

Decode on Apple Silicon is bound by memory bandwidth, not by compute. Every token generated requires reading the model's weights out of unified memory, and for a single sequence that read feeds exactly one forward pass before the whole thing repeats. The GPU spends most of its time waiting on memory, and the arithmetic units sit largely unused. That is why a 14B class model might give you eighty tokens a second on a single stream and feel fast to a person, while the chip itself is nowhere near its ceiling.

Batching fixes this by reading the weights once and using them for many sequences at the same time. The expensive part, the weight read, is amortized across every sequence in the batch, so aggregate throughput climbs steeply as you add sequences even though each individual sequence gets no faster. This is the same mechanism behind continuous batching in a serving context, which we covered in the continuous batching writeup; the difference in offline work is that you control the entire input set up front, so you can pack the batch far more aggressively than a live server ever could.

How Far Batching Takes You Before Memory Stops It

Throughput does not climb forever. Two ceilings end the party. The first is memory: every sequence in flight holds a KV cache proportional to its context length, and on a 128GB M5 Max that pool is finite. Add enough concurrent sequences at long context and the KV cache alone exhausts unified memory, at which point you cannot grow the batch further no matter how much bandwidth is left unused. The second is the crossover from memory bound to compute bound: past a certain batch size the forward pass is doing enough parallel arithmetic that the arithmetic units, not the memory bus, become the limit, and throughput flattens.

These are directional figures from a 128GB M5 Max running a 14B class model at 4-bit under an MLX batch generation loop, with roughly 1K token prompts and 300 token completions, no streaming, sequences sorted by length. They show the shape, not universal constants, because the knee moves with model size, quantization, and context length.

Batch size	Aggregate throughput (tok/s)	Relative cost per 1M tokens	Notes
1	84	1.00x	GPU mostly idle, memory bound
4	288	0.29x	near linear scaling
8	486	0.17x	still climbing, memory has room
16	690	0.12x	approaching the compute knee
32	760	0.11x	KV cache pressure, diminishing returns

Read the throughput and the cost columns together, because they are the whole argument for doing this deliberately. Moving from batch one to batch sixteen does not make any single prompt finish faster, but it clears the backlog roughly eight times quicker and cuts the effective cost per million tokens by nearly an order of magnitude, because you are paying for the same wall clock hour of electricity and hardware but getting eight times the work out of it. The jump from sixteen to thirty two barely moves the needle, which tells you the knee is near sixteen for this configuration and pushing past it only trades memory headroom for a couple of percent. Find your own knee before committing an overnight run, because guessing wrong in either direction wastes the machine.

Sort the Backlog So the GPU Never Waits on Padding

The single largest avoidable loss in batch inference is padding. When you batch sequences of wildly different lengths, the shorter ones are padded to match the longest one in the batch, and every padded position is compute and memory spent generating nothing. A batch holding one 4K prompt and fifteen 200 token prompts runs at the pace of the 4K prompt while wasting most of its width. The fix costs nothing: sort the entire backlog by length before batching, so each batch groups sequences of similar size and padding waste collapses to almost zero.

Length sorting also lets you vary batch size by bucket. Short prompts pack many to a batch because their KV cache is small; long prompts pack few because each one reserves a large cache. A fixed batch size across a mixed backlog either over commits memory on the long buckets or under fills the short ones, so the better pattern is a target token budget per batch rather than a fixed sequence count, filling each batch until the combined context reaches your memory ceiling regardless of how many sequences that takes.

# mlx-lm batch generation over a sorted backlog, token-budgeted batches
from mlx_lm import load, batch_generate

model, tokenizer = load("mlx-community/model-14b-4bit")

prompts = [p for p in load_backlog()]                 # your 100k prompts
prompts.sort(key=lambda p: len(tokenizer.encode(p)))  # group similar lengths

TOKEN_BUDGET = 24000        # tune to your KV cache ceiling, not a fixed count
batch, budget = [], 0
for p in prompts:
    n = len(tokenizer.encode(p))
    if budget + n > TOKEN_BUDGET and batch:
        results = batch_generate(model, tokenizer, batch, max_tokens=300)
        checkpoint(batch, results)                    # persist before the next batch
        batch, budget = [], 0
    batch.append(p); budget += n
if batch:
    checkpoint(batch, batch_generate(model, tokenizer, batch, max_tokens=300))

Turn Off Everything That Serves a Human

Interactive defaults are pure overhead in a batch job. Streaming exists so a person sees tokens as they arrive, but there is no person, so streaming only adds a per token flush that buys nothing; generate whole and write the finished result. Any per request logging, tracing, or metrics tuned for a live API should be dialed down to batch level rather than per request, because at hundreds of sequences an hour the bookkeeping itself starts to matter. And unlike a live server, you do not want an admission control queue holding sequences back, because the whole point is to run at maximum concurrency deliberately; the backpressure machinery that protects an interactive box under unpredictable load, which we described in the admission control post, is the opposite of what an offline run wants.

One interactive feature is worth keeping when it fits: prefix caching. If your backlog shares a large common prefix, a long system prompt, a fixed instruction block, a shared context document, you can compute that prefix once and reuse its KV cache across every prompt that shares it, skipping the prefill for the shared portion entirely. On a backlog where the variable part is short and the shared part is long, this is a large saving, and it composes with batching rather than competing with it. We covered the mechanics in the prefix caching writeup; in batch work it is often the difference between prefill dominating the run and prefill disappearing.

Make the Run Resumable Before You Start It

An overnight batch job will be interrupted. The model will hit a prompt that triggers an edge case, the machine will be rebooted, a thermal event will pause it, or you will simply want to stop and change something. A run that has no memory of its progress forces you to start the hundred thousand prompts over from zero, and that is how a two hour job becomes a two day ordeal. Checkpoint after every batch, writing completed results keyed by a stable prompt identifier, so a restart skips everything already done and picks up exactly where it stopped.

Idempotency is the companion requirement. Key results by a hash of the prompt and generation parameters, not by position in the list, so that reordering the backlog, retrying a failed batch, or running the job twice never produces duplicates or drops work. This costs a few lines and it converts the run from something you have to babysit into something you can start, walk away from, and trust to finish or resume cleanly. Sustained runs also raise the thermal question, because a batch job holds the GPU near saturation for hours in a way interactive traffic never does, and that changes the throughput you actually hold over a long run versus the burst number a short test reports.

When to Skip Batching Entirely

Batching is the right default for a genuine backlog, but it is the wrong tool for two nearby cases. The first is latency sensitive work that only looks like a batch: if results are consumed as they complete and someone downstream is waiting, you want streaming interactive serving, not a batch loop that returns everything at the end. The second is a backlog too small to amortize setup, where the model load and the sorting pass cost more than the handful of prompts would take run plainly. Below a few hundred prompts, just run them; the batching machinery earns its complexity only when the pile is large enough that a ten times throughput gain translates into hours saved.

When This Applies to Your Stack

If you are generating or classifying at corpus scale on Apple Silicon, stop treating it like interactive serving. Sort the backlog by length, batch to a token budget rather than a fixed count, find the throughput knee for your model and quantization before committing the full run, turn off streaming and per request overhead, reuse a shared prefix cache when your prompts have one, and checkpoint after every batch so an interruption costs minutes rather than the whole job. Done this way, an M5 Max clears a backlog at close to an order of magnitude lower cost per token than the one at a time approach, because the expensive weight reads are shared instead of repeated.

This is the kind of work that quietly determines whether running models locally is cheaper than paying a hosted API by the token or merely feels cheaper while the machine sits idle between requests. When we help teams stand up local inference for real workloads, the batch path is where a large share of the actual economics lives, and it is the AI infrastructure work Contra Collective does when a local deployment has to process volume rather than serve a demo.

FAQ

Does batching make each prompt finish faster? No. Batching increases aggregate throughput across all sequences, not the speed of any single one. An individual prompt in a batch of sixteen finishes at roughly the same wall clock time as it would alone, but the batch clears sixteen prompts in that time instead of one. Batch for total backlog speed, not per item latency.

What limits batch size on an M5 Max? Two ceilings. First, KV cache memory, since every concurrent sequence reserves cache proportional to its context length and unified memory is finite. Second, the crossover from memory bound to compute bound, where past a certain batch size the arithmetic units saturate and throughput stops climbing. Find the knee empirically before a long run.

Why sort prompts by length before batching? Because batches pad shorter sequences to the length of the longest, and every padded position is wasted work. Sorting groups similar lengths together so padding waste nearly disappears, and it lets you pack short prompt batches wide and long prompt batches narrow to respect the memory ceiling.

Should I use streaming for a batch job? No. Streaming exists to show tokens to a waiting human and adds a per token flush that buys nothing offline. Generate whole responses and write the finished result. Also disable per request logging and admission control queuing, which are interactive protections that only add overhead to a deliberate maximum concurrency run.

How do I make an overnight run survive an interruption? Checkpoint after every batch, writing results keyed by a stable hash of the prompt and its generation parameters. A restart then skips completed work and resumes cleanly, and idempotent keys mean retries and reordering never duplicate or drop results. This turns a fragile multi hour job into one you can stop and resume without losing progress.