Running Vision Language Models Locally on an M5 Max: What Image Tokens Actually Cost (2026)

A vision language model looks like a bargain when you load it. A 7B VLM at 4 bit sits in a few gigabytes of unified memory, a 32B one in the low twenties, and even a 72B model at 4 bit fits comfortably on a 128GB M5 Max at roughly 40GB resident. Then you hand it a screenshot and wait three seconds for the first token, and the bargain evaporates. The weights were never the problem. The image was. When you run a VLM locally you pay, in full and in real time, for a cost that cloud vision APIs hide inside a flat per image price: the prefill over every visual token the encoder produced. Here is where that cost comes from and how to control it.

The Weights Are Cheap, The Image Is Not

Loading a VLM has two memory components that a text only model does not force you to think about separately. There is the language model itself, which behaves exactly like any local LLM, and there is the vision stack: a vision encoder plus a projection layer that maps image features into the language model's embedding space. The encoder is small next to the language model, usually a few hundred megabytes to a couple of gigabytes, so it barely moves your resident footprint.

What moves your latency is what the encoder produces. A vision encoder patchifies an image into a grid, and each patch becomes a visual token that occupies a real position in the language model's context, complete with a KV cache entry. Models like Qwen3-VL use dynamic resolution, so the number of visual tokens scales with the pixel count of the image rather than being fixed. A small thumbnail might become a few hundred tokens; a full resolution screenshot or a photographed document can become several thousand. Those tokens are not free text you skim past. They are the prefill workload, and prefill is compute bound in a way decode is not.

Cost center	Where it lives	Scales with	Felt as
Language model weights	Unified memory, resident	Model size and quant	Fixed load cost
Vision encoder weights	Unified memory, resident	Encoder size	Small fixed cost
Visual tokens (prefill)	KV cache plus compute	Image resolution	Time to first token
Text decode	Compute per token	Output length	Tokens per second

Why Time To First Token Blows Up

Prefill and decode are different workloads, and image heavy prompts push almost all the pain into prefill. Decode generates one token at a time and is largely memory bandwidth bound, which is where Apple Silicon does well. Prefill processes the entire input in parallel and is compute bound, so a sudden expansion of the input length hits the part of the pipeline that Apple Silicon is comparatively weaker at than a discrete GPU. We covered this split in depth for text in the prefill versus decode writeup, and vision makes it sharper, because a single image can inject more prefill tokens than an entire page of text.

Put concrete numbers on it, framed as observations from a single M5 Max rather than a guarantee. A 32B class VLM at 4 bit decoding text runs at a comfortable interactive rate. Feed it one native resolution screenshot that expands to roughly three to four thousand visual tokens and time to first token climbs into the multi second range, because the machine must prefill all of those tokens before it emits a word. Halve the image resolution and the visual token count falls roughly with the pixel area, and time to first token falls with it. The decode speed after that first token is unchanged; the whole cost you added lives in the prefill of the image.

Resolution Is The One Knob

The practical lever is image resolution, and most VLM runtimes expose it as a minimum and maximum pixel budget rather than a literal width and height. This is the setting that decides how many visual tokens the encoder emits, which means it decides your prefill cost directly.

# Sketch with mlx-vlm. The pixel budget, not the file size,
# controls how many visual tokens the encoder produces.
from mlx_vlm import load, generate
from mlx_vlm.utils import load_config

model, processor = load("mlx-community/Qwen3-VL-32B-Instruct-4bit")

# Cap the image budget so a full res screenshot does not
# expand into thousands of tokens. Numbers illustrate the
# pattern; tune per model and per task.
processor.image_processor.min_pixels = 256 * 28 * 28
processor.image_processor.max_pixels = 1280 * 28 * 28

out = generate(
    model, processor,
    prompt="Extract every line item and total from this invoice.",
    image="invoice.png",
    max_tokens=512,
)

The trade-off is legibility. Downscale too far and small text, thin table borders, or fine detail dissolve, and the model starts guessing. Downscale sensibly and you cut prefill cost dramatically with no accuracy loss, because most photographs and UI screenshots carry far more pixels than the task actually needs. The discipline is to set the pixel budget to the smallest value at which the model still reads what matters, and to test that per task rather than leaving the runtime default, which is usually tuned for maximum fidelity at maximum cost.

Encode Once, Prompt Many Times

The second lever is not re-encoding an image you have already processed. In a multi turn conversation about the same picture, a naive loop runs the vision encoder again on every turn, paying the encoder cost repeatedly for an image that has not changed. A vision feature cache fixes this by storing the projected features keyed on the image so the encoder runs once per unique image and later turns reuse the result. This is the multimodal cousin of KV cache reuse for text, and it matters most for interactive tools where a user asks several questions about one document or one screen.

The pattern generalizes. If your workload is document extraction over a batch of pages, the win is caching and reusing the language model prompt scaffolding across pages, while the image portion necessarily differs per page. If your workload is a chat about one uploaded image, the win is caching the vision features so only the new text turns cost anything. Either way the principle is the same one that governs all local inference: identify the expensive computation, do it once, and protect the result. Our local document extraction writeup walks through the batch case where images change every call and the scaffolding is what you cache.

Choosing A Model Size

Model size for a local VLM follows the same memory arithmetic as any local model, with the encoder added on top. The decision table below is for a 128GB M5 Max, with resident memory as approximate 4 bit figures and the reminder that your real ceiling is lower once the KV cache for a long visual prompt is included.

VLM size (4 bit)	Approx resident	Fits 128GB	Best for
7B class	Several GB	Easily	Fast UI and simple image QA
32B class	Low twenties GB	Comfortably	Document extraction, detailed reasoning
72B class	Around 40GB	Yes, with headroom	Hardest visual reasoning, batch tolerant

The smaller models are not just faster to decode; they also tend to emit their answer after a shorter think, which compounds with lower prefill to make them feel far snappier for interactive use. Reserve the 72B class for tasks where the visual reasoning genuinely fails at 32B, and accept that its prefill on a large image is where you will spend your seconds.

When This Applies To Your Stack

A local VLM earns its place when your images cannot leave the building, or when volume makes per image cloud pricing painful, and when your task tolerates the prefill cost of the resolution you actually need. Document extraction over sensitive records, screenshot understanding for an internal agent, and image QA on proprietary product photography are the shapes that fit. If your images are public, your volume is modest, and you need the absolute frontier of visual reasoning, a cloud vision API is often the cheaper answer once you price your own engineering time, because you skip the resolution tuning entirely.

If you are standing up local, private multimodal inference and want the pipeline built so image resolution is tuned to the task, features are cached, and prefill cost is a number you control rather than a surprise, that is the AI infrastructure work we do at Contra Collective. The models fit on the hardware you already own; the value is in the harness that keeps the image from becoming your bottleneck.

FAQ

Why is my local VLM slow to first token but fast after that? Because the image expands into many visual tokens that must be prefilled before generation starts, and prefill is compute bound. Once the first token is out, decode runs at normal text speed. Lowering the image resolution cuts the visual token count and the time to first token with it.

How many tokens does one image become? It depends on the model and the image resolution. Models with dynamic resolution scale visual tokens with pixel area, so a thumbnail might be a few hundred tokens and a full resolution screenshot several thousand. The pixel budget setting in the runtime is what controls this.

Can an M5 Max run a 72B vision model? Yes. At 4 bit a 72B VLM is roughly 40GB resident, which fits a 128GB M5 Max with headroom for the encoder and a reasonable visual prompt. The constraint you feel is prefill latency on large images, not whether the weights fit.

Should I downscale every image? Downscale to the smallest resolution at which the model still reads what the task requires, and verify that per task. Most photos and screenshots carry more detail than the job needs, so downscaling usually cuts cost with no accuracy loss, but fine print and dense tables set a floor you should not cross.