Aug 8, 2026 / AI Engineering Most writing about local inference assumes a human is on the other end, watching tokens appear and judging the model by how fast the first one lands. Batch work is the opposite situation, and it is more common than the interactive framing suggests: you have a backlog of tens or hundreds of thousands of prompts, nobody is waiting on any single one, and the only thing that matters is how much of the pile you can clear before morning. Product descriptions for a catalog, classification over a support archive, embeddings and summaries for a corpus, synthetic data for a fine tune. In that regime latency is irrelevant and throughput is everything, and the settings that make a local server feel responsive to a person actively work against you, because they optimize for the wrong number. Running one prompt at a time on an M5 Max leaves most of the machine idle, since a single decode stream cannot saturate the memory bandwidth the chip has to offer. The job is to keep many sequences in flight so the expensive weight reads are amortized across all of them, and to arrange the backlog so the GPU never waits on padding, scheduling, or a slow tail. This post is about doing that deliberately: what actually bounds batch throughput on Apple Silicon, how far batching takes you before memory stops you, and the operational scaffolding that turns a fragile overnight run into one you can resume when it dies at 3am.
Aug 7, 2026 / AI Engineering Prompt caching skips the prefill on a repeated prefix, but it does nothing for two users who ask the same thing in different words. On a single Apple Silicon box, where every generation competes for the same unified memory, the cheapest token is the one you never generate. Semantic response caching sits in front of the model, embeds the incoming prompt, and if a past prompt was close enough in meaning it returns the stored answer without touching the GPU. The hard part is not the cache. It is deciding when two questions are actually the same question, because a threshold set too loose will confidently serve the wrong answer. This post builds the gateway, measures the hit rate on a real support workload, and sets the similarity threshold where a wrong answer costs more than a cache miss.
Aug 5, 2026 / AI Engineering The first time a local inference service meets real traffic, it does not slow down gracefully, it falls off a cliff. A single request against a 14B model on an M5 Max streams at eighty tokens a second and feels like a hosted API. Then a second user arrives, and a third, and a burst of eight lands at once, and suddenly latency has tripled, the machine is paging to disk, and one unlucky request gets an out of memory kill mid generation. Nothing in the model changed. What changed is that concurrency on a single Apple Silicon box is bounded by unified memory, not by raw compute, because every in flight request holds a slice of KV cache that lives in the same pool the model weights and the operating system are already using. Once the concurrent working set crosses the wired memory limit, the machine does the worst possible thing under load, which is to keep accepting work it cannot serve. The fix is not a bigger model server flag. It is admission control: decide how many requests the box can actually hold, queue a bounded number behind them, and reject the rest with honest backpressure instead of pretending. This post measures where the ceiling sits on a 128GB M5 Max, explains why it is a memory ceiling and not a throughput one, and lays out the gateway pattern that keeps a local service predictable when the traffic is not.
Aug 2, 2026 / AI Engineering The model produces a probability distribution over the whole vocabulary at every step, and the sampler is the piece that turns that distribution into a single token. This is the part of the local inference stack teams tune last and understand least, which is odd because it is the setting that most directly decides whether output is repetitive, whether it hallucinates under pressure, and whether a JSON response parses. The defaults matter more than people think: most runners ship a temperature and a top-p tuned for open ended chat, and those same values are actively wrong for code generation, structured extraction, or an agent loop that needs deterministic tool calls. Worse, the parameters interact. Temperature reshapes the distribution before top-p or min-p truncate it, so changing one silently changes what the others do, and a repetition penalty stacked on top can push a model off a correct but repeated token into a wrong one. On Apple Silicon this is all essentially free to change: sampling happens on a handful of logits after the expensive forward pass, so it costs no extra memory and negligible time, which means there is no throughput reason not to tune it. This post walks through what each knob does to the distribution, how they interact, and what settings hold up for the three jobs most local deployments actually run, using an M5 Max as the test bench.
Jul 30, 2026 / AI Engineering The mental model most people bring to a vision language model is wrong in a way that costs them latency. They picture an image as a single input, roughly one unit of work, the way a text token is one unit of work. What actually happens is that the vision encoder chops the image into patches, and a high resolution photo can turn into hundreds or thousands of visual tokens that the language model must then prefill through before it generates anything at all. On a cloud endpoint you never feel this, because someone else eats the prefill and bills you a flat per image rate. Run the same model on your own M5 Max and the cost becomes visible immediately: the model loads fine, it holds in unified memory with room to spare, and then a single screenshot at native resolution takes several seconds to first token because you just asked the machine to prefill four thousand tokens it manufactured out of one picture. This post is about that gap. We look at where the memory actually goes when you load a VLM locally, why image tokens and not model weights are usually your latency problem, how the resolution setting is the one knob that moves both accuracy and speed, and when a local VLM is the right call versus a cloud vision API. The economics of local vision are real, but only if you understand that the image, not the prompt, is the expensive part.
Jul 29, 2026 / AI Engineering The naive way to serve a fine tune is to merge the adapter into the base weights and load the merged model, which is fine until you have twenty fine tunes and a machine that can only hold two merged copies of a 70B model in memory at once. LoRA exists precisely so you do not have to do that. The adapter is a few hundred megabytes of low rank matrices that sit on top of a frozen base, which means one resident base model can back many adapters if your serving layer knows how to keep the base loaded and swap or batch the adapters around it. The gap between knowing that and running it in production is where most teams stall, because the obvious implementation reloads the whole base every time a request wants a different adapter, and that throws away the entire advantage. This post is about serving many LoRA fine tunes from a single M5 Max: how much memory each resident adapter actually costs, how fast you can bring a cold adapter in from disk, why merged serving falls over at scale, and the batched adapter pattern that lets requests for different fine tunes share the same forward pass. The economics only work if the base stays put and the adapters move cheaply, so the whole design is about protecting that invariant.
Jul 28, 2026 / AI Engineering A single M5 Max is a genuinely good inference box for a steady stream of requests, and if your load were flat you would never need anything else. Real load is not flat. It has a baseline you can size hardware against and spikes you cannot, and the spike is where a local only setup fails: the queue depth climbs, time to first token blows past your latency budget, and users wait behind a machine that is already at full decode. The fix is not a bigger Mac, because you would be buying capacity for a peak that shows up a few hours a week and sitting idle the rest of the time. The fix is a hybrid: keep the baseline on local hardware where each token is nearly free and no data leaves your network, and burst only the overflow to a cloud API that you pay for by the token and only when you actually need it. The design problem is the router. A router that spills too early throws away the cost advantage you built the local tier for, and one that spills too late lets the queue hurt users before it reacts. This post covers how to set the spill threshold against queue depth rather than raw request rate, how to keep the cloud path from leaking data you meant to keep in house, what the blended cost actually looks like, and when a hybrid is worth its extra failure modes versus when one tier is the honest answer.
Jul 27, 2026 / AI Engineering If you run a single large model locally for everything, you are paying the full decode cost of that model on questions a much smaller one could have answered in a fifth of the time. A cascade fixes that: a small fast model takes the first pass, a cheap gate decides whether the answer is good enough, and only the queries that fail the gate get escalated to the big model. On an M5 Max the economics are compelling because the small model and the large model share the same unified memory, so there is no extra hardware and no network hop, just a routing decision. The catch is that a cascade only pays off if the escalation rate is low and the gate is cheap and accurate, because a gate that sends everything upstream anyway just adds latency on top of the big model you were already running. This post walks through how to size the two tiers, how to build a gate that is cheaper than the model it protects, what the measured latency and throughput look like on an M5 Max, and when the pattern is worth the extra moving parts versus when a single model is the honest answer.
Jul 26, 2026 / AI Engineering You bought a Mac with 128GB of unified memory to run a big model locally, and then the model that should fit refuses to stay resident and your tokens per second collapse. The reason is that macOS does not hand the whole pool to the GPU. There is a wired limit, the amount of memory the graphics side is allowed to pin, and by default it sits well below the installed capacity so the CPU, the window server, and the rest of the system keep breathing room. When the weights plus the KV cache push past that ceiling, the runtime spills to slower paths and the decode rate falls off a cliff, which people misread as a bandwidth or quantization problem when it is really an allocation problem. This post measures the default headroom on M4 and M5 machines, shows exactly where common model sizes cross the line, and gives the one sysctl knob that raises the ceiling safely, along with how far you can push it before you starve the OS. If you are speccing or tuning a machine to hold a large model resident, this is the number that actually decides whether it fits.
Jul 25, 2026 / AI Engineering When people shop for a Mac to run models locally, they read the GPU core count and the neural engine numbers. Those are the wrong numbers. Single stream token generation is bound almost entirely by memory bandwidth, because producing each token means reading every active weight out of unified memory, and the arithmetic per byte is tiny. That is why a chip with twice the bandwidth gives you roughly twice the tokens per second on the same model, and why adding GPU cores without adding bandwidth barely moves the decode figure. We measured llama style decode across M5 Pro, M5 Max, and M5 Ultra, put the tokens per second next to the rated bandwidth, and show where the picture flips: batched serving and prefill are compute bound and reward the cores, while everyday single user chat is bandwidth bound and rewards GB/s. If you are speccing a machine or trying to explain why your throughput is what it is, this is the model to reason with.
Jul 24, 2026 / AI Engineering There is a real reason to run OCR on your own hardware rather than a cloud document API: invoices, purchase orders, and contracts are exactly the documents you do not want leaving your network, and per page cloud pricing turns a back office batch job into a recurring bill. The question is whether a Mac can do it fast and accurately enough to replace the API. We put Surya, docTR, and PaddleOCR on an M5 Max, measured pages per second and character error rate on real business documents, and compared them against a local vision language model doing the same job, so you can see which engine fits which document and where the accuracy actually falls apart.
Jul 23, 2026 / AI Engineering Speculative decoding is the standard way to make local generation faster, but it costs you a second model resident in the same unified memory you are already fighting over. Prompt lookup decoding is the draft model free version of the same trick. Instead of a small model guessing the next tokens, it guesses them by finding a matching n-gram already sitting in the prompt and proposing the text that followed. On an M5 Max we measured how much it helps on RAG, summarization, and code editing, and why it is close to useless for open ended chat.