Running Vision Language Models Locally on an M5 Max: What Image Tokens Actually Cost (2026)
The mental model most people bring to a vision language model is wrong in a way that costs them latency. They picture an image as a single input, roughly one unit of work, the way a text token is one unit of work. What actually happens is that the vision encoder chops the image into patches, and a high resolution photo can turn into hundreds or thousands of visual tokens that the language model must then prefill through before it generates anything at all. On a cloud endpoint you never feel this, because someone else eats the prefill and bills you a flat per image rate. Run the same model on your own M5 Max and the cost becomes visible immediately: the model loads fine, it holds in unified memory with room to spare, and then a single screenshot at native resolution takes several seconds to first token because you just asked the machine to prefill four thousand tokens it manufactured out of one picture. This post is about that gap. We look at where the memory actually goes when you load a VLM locally, why image tokens and not model weights are usually your latency problem, how the resolution setting is the one knob that moves both accuracy and speed, and when a local VLM is the right call versus a cloud vision API. The economics of local vision are real, but only if you understand that the image, not the prompt, is the expensive part.
Running Vision Language Models Locally on an M5 Max: What Image Tokens Actually Cost (2026)
A vision language model looks like a bargain when you load it. A 7B VLM at 4 bit sits in a few gigabytes of unified memory, a 32B one in the low twenties, and even a 72B model at 4 bit fits comfortably on a 128GB M5 Max at roughly 40GB resident. Then you hand it a screenshot and wait three seconds for the first token, and the bargain evaporates. The weights were never the problem. The image was. When you run a VLM locally you pay, in full and in real time, for a cost that cloud vision APIs hide inside a flat per image price: the prefill over every visual token the encoder produced. Here is where that cost comes from and how to control it.
The Weights Are Cheap, The Image Is Not
Loading a VLM has two memory components that a text only model does not force you to think about separately. There is the language model itself, which behaves exactly like any local LLM, and there is the vision stack: a vision encoder plus a projection layer that maps image features into the language model's embedding space. The encoder is small next to the language model, usually a few hundred megabytes to a couple of gigabytes, so it barely moves your resident footprint.
What moves your latency is what the encoder produces. A vision encoder patchifies an image into a grid, and each patch becomes a visual token that occupies a real position in the language model's context, complete with a KV cache entry. Models like Qwen3-VL use dynamic resolution, so the number of visual tokens scales with the pixel count of the image rather than being fixed. A small thumbnail might become a few hundred tokens; a full resolution screenshot or a photographed document can become several thousand. Those tokens are not free text you skim past. They are the prefill workload, and prefill is compute bound in a way decode is not.
| Cost center | Where it lives | Scales with | Felt as |
|---|---|---|---|
| Language model weights | Unified memory, resident | Model size and quant | Fixed load cost |
| Vision encoder weights | Unified memory, resident | Encoder size | Small fixed cost |
| Visual tokens (prefill) | KV cache plus compute | Image resolution | Time to first token |
| Text decode | Compute per token | Output length | Tokens per second |
Why Time To First Token Blows Up
Prefill and decode are different workloads, and image heavy prompts push almost all the pain into prefill. Decode generates one token at a time and is largely memory bandwidth bound, which is where Apple Silicon does well. Prefill processes the entire input in parallel and is compute bound, so a sudden expansion of the input length hits the part of the pipeline that Apple Silicon is comparatively weaker at than a discrete GPU. We covered this split in depth for text in the prefill versus decode writeup, and vision makes it sharper, because a single image can inject more prefill tokens than an entire page of text.
Put concrete numbers on it, framed as observations from a single M5 Max rather than a guarantee. A 32B class VLM at 4 bit decoding text runs at a comfortable interactive rate. Feed it one native resolution screenshot that expands to roughly three to four thousand visual tokens and time to first token climbs into the multi second range, because the machine must prefill all of those tokens before it emits a word. Halve the image resolution and the visual token count falls roughly with the pixel area, and time to first token falls with it. The decode speed after that first token is unchanged; the whole cost you added lives in the prefill of the image.
Resolution Is The One Knob
The practical lever is image resolution, and most VLM runtimes expose it as a minimum and maximum pixel budget rather than a literal width and height. This is the setting that decides how many visual tokens the encoder emits, which means it decides your prefill cost directly.
# Sketch with mlx-vlm. The pixel budget, not the file size,
# controls how many visual tokens the encoder produces.
from mlx_vlm import load, generate
from mlx_vlm.utils import load_config
model, processor = load("mlx-community/Qwen3-VL-32B-Instruct-4bit")
# Cap the image budget so a full res screenshot does not
# expand into thousands of tokens. Numbers illustrate the
# pattern; tune per model and per task.
processor.image_processor.min_pixels = 256 * 28 * 28
processor.image_processor.max_pixels = 1280 * 28 * 28
out = generate(
model, processor,
prompt="Extract every line item and total from this invoice.",
image="invoice.png",
max_tokens=512,
)
The trade-off is legibility. Downscale too far and small text, thin table borders, or fine detail dissolve, and the model starts guessing. Downscale sensibly and you cut prefill cost dramatically with no accuracy loss, because most photographs and UI screenshots carry far more pixels than the task actually needs. The discipline is to set the pixel budget to the smallest value at which the model still reads what matters, and to test that per task rather than leaving the runtime default, which is usually tuned for maximum fidelity at maximum cost.
Encode Once, Prompt Many Times
The second lever is not re-encoding an image you have already processed. In a multi turn conversation about the same picture, a naive loop runs the vision encoder again on every turn, paying the encoder cost repeatedly for an image that has not changed. A vision feature cache fixes this by storing the projected features keyed on the image so the encoder runs once per unique image and later turns reuse the result. This is the multimodal cousin of KV cache reuse for text, and it matters most for interactive tools where a user asks several questions about one document or one screen.
The pattern generalizes. If your workload is document extraction over a batch of pages, the win is caching and reusing the language model prompt scaffolding across pages, while the image portion necessarily differs per page. If your workload is a chat about one uploaded image, the win is caching the vision features so only the new text turns cost anything. Either way the principle is the same one that governs all local inference: identify the expensive computation, do it once, and protect the result. Our local document extraction writeup walks through the batch case where images change every call and the scaffolding is what you cache.
Choosing A Model Size
Model size for a local VLM follows the same memory arithmetic as any local model, with the encoder added on top. The decision table below is for a 128GB M5 Max, with resident memory as approximate 4 bit figures and the reminder that your real ceiling is lower once the KV cache for a long visual prompt is included.
| VLM size (4 bit) | Approx resident | Fits 128GB | Best for |
|---|---|---|---|
| 7B class | Several GB | Easily | Fast UI and simple image QA |
| 32B class | Low twenties GB | Comfortably | Document extraction, detailed reasoning |
| 72B class | Around 40GB | Yes, with headroom | Hardest visual reasoning, batch tolerant |
The smaller models are not just faster to decode; they also tend to emit their answer after a shorter think, which compounds with lower prefill to make them feel far snappier for interactive use. Reserve the 72B class for tasks where the visual reasoning genuinely fails at 32B, and accept that its prefill on a large image is where you will spend your seconds.
When This Applies To Your Stack
A local VLM earns its place when your images cannot leave the building, or when volume makes per image cloud pricing painful, and when your task tolerates the prefill cost of the resolution you actually need. Document extraction over sensitive records, screenshot understanding for an internal agent, and image QA on proprietary product photography are the shapes that fit. If your images are public, your volume is modest, and you need the absolute frontier of visual reasoning, a cloud vision API is often the cheaper answer once you price your own engineering time, because you skip the resolution tuning entirely.
If you are standing up local, private multimodal inference and want the pipeline built so image resolution is tuned to the task, features are cached, and prefill cost is a number you control rather than a surprise, that is the AI infrastructure work we do at Contra Collective. The models fit on the hardware you already own; the value is in the harness that keeps the image from becoming your bottleneck.
FAQ
Why is my local VLM slow to first token but fast after that? Because the image expands into many visual tokens that must be prefilled before generation starts, and prefill is compute bound. Once the first token is out, decode runs at normal text speed. Lowering the image resolution cuts the visual token count and the time to first token with it.
How many tokens does one image become? It depends on the model and the image resolution. Models with dynamic resolution scale visual tokens with pixel area, so a thumbnail might be a few hundred tokens and a full resolution screenshot several thousand. The pixel budget setting in the runtime is what controls this.
Can an M5 Max run a 72B vision model? Yes. At 4 bit a 72B VLM is roughly 40GB resident, which fits a 128GB M5 Max with headroom for the encoder and a reasonable visual prompt. The constraint you feel is prefill latency on large images, not whether the weights fit.
Should I downscale every image? Downscale to the smallest resolution at which the model still reads what the task requires, and verify that per task. Most photos and screenshots carry more detail than the job needs, so downscaling usually cuts cost with no accuracy loss, but fine print and dense tables set a floor you should not cross.
More from the lab.
Sampler Settings for Local LLMs on Apple Silicon: Temperature, Top-p, Min-p, and Repetition Penalty Measured (2026)
The model produces a probability distribution over the whole vocabulary at every step, and the sampler is the piece that turns that distribution into a single token. This is the part of the local inference stack teams tune last and understand least, which is odd because it is the setting that most directly decides whether output is repetitive, whether it hallucinates under pressure, and whether a JSON response parses. The defaults matter more than people think: most runners ship a temperature and a top-p tuned for open ended chat, and those same values are actively wrong for code generation, structured extraction, or an agent loop that needs deterministic tool calls. Worse, the parameters interact. Temperature reshapes the distribution before top-p or min-p truncate it, so changing one silently changes what the others do, and a repetition penalty stacked on top can push a model off a correct but repeated token into a wrong one. On Apple Silicon this is all essentially free to change: sampling happens on a handful of logits after the expensive forward pass, so it costs no extra memory and negligible time, which means there is no throughput reason not to tune it. This post walks through what each knob does to the distribution, how they interact, and what settings hold up for the three jobs most local deployments actually run, using an M5 Max as the test bench.
Serving Many LoRA Adapters on One M5 Max: Hot Swapping Fine Tunes Without Reloading the Base Model (2026)
The naive way to serve a fine tune is to merge the adapter into the base weights and load the merged model, which is fine until you have twenty fine tunes and a machine that can only hold two merged copies of a 70B model in memory at once. LoRA exists precisely so you do not have to do that. The adapter is a few hundred megabytes of low rank matrices that sit on top of a frozen base, which means one resident base model can back many adapters if your serving layer knows how to keep the base loaded and swap or batch the adapters around it. The gap between knowing that and running it in production is where most teams stall, because the obvious implementation reloads the whole base every time a request wants a different adapter, and that throws away the entire advantage. This post is about serving many LoRA fine tunes from a single M5 Max: how much memory each resident adapter actually costs, how fast you can bring a cold adapter in from disk, why merged serving falls over at scale, and the batched adapter pattern that lets requests for different fine tunes share the same forward pass. The economics only work if the base stays put and the adapters move cheaply, so the whole design is about protecting that invariant.
Hybrid Local and Cloud Inference: Bursting Overflow from an M5 Max to a Cloud API Under Load (2026)
A single M5 Max is a genuinely good inference box for a steady stream of requests, and if your load were flat you would never need anything else. Real load is not flat. It has a baseline you can size hardware against and spikes you cannot, and the spike is where a local only setup fails: the queue depth climbs, time to first token blows past your latency budget, and users wait behind a machine that is already at full decode. The fix is not a bigger Mac, because you would be buying capacity for a peak that shows up a few hours a week and sitting idle the rest of the time. The fix is a hybrid: keep the baseline on local hardware where each token is nearly free and no data leaves your network, and burst only the overflow to a cloud API that you pay for by the token and only when you actually need it. The design problem is the router. A router that spills too early throws away the cost advantage you built the local tier for, and one that spills too late lets the queue hurt users before it reacts. This post covers how to set the spill threshold against queue depth rather than raw request rate, how to keep the cloud path from leaking data you meant to keep in house, what the blended cost actually looks like, and when a hybrid is worth its extra failure modes versus when one tier is the honest answer.