Jul 29, 2026 / AI Engineering The naive way to serve a fine tune is to merge the adapter into the base weights and load the merged model, which is fine until you have twenty fine tunes and a machine that can only hold two merged copies of a 70B model in memory at once. LoRA exists precisely so you do not have to do that. The adapter is a few hundred megabytes of low rank matrices that sit on top of a frozen base, which means one resident base model can back many adapters if your serving layer knows how to keep the base loaded and swap or batch the adapters around it. The gap between knowing that and running it in production is where most teams stall, because the obvious implementation reloads the whole base every time a request wants a different adapter, and that throws away the entire advantage. This post is about serving many LoRA fine tunes from a single M5 Max: how much memory each resident adapter actually costs, how fast you can bring a cold adapter in from disk, why merged serving falls over at scale, and the batched adapter pattern that lets requests for different fine tunes share the same forward pass. The economics only work if the base stays put and the adapters move cheaply, so the whole design is about protecting that invariant.
Jul 28, 2026 / AI Engineering A single M5 Max is a genuinely good inference box for a steady stream of requests, and if your load were flat you would never need anything else. Real load is not flat. It has a baseline you can size hardware against and spikes you cannot, and the spike is where a local only setup fails: the queue depth climbs, time to first token blows past your latency budget, and users wait behind a machine that is already at full decode. The fix is not a bigger Mac, because you would be buying capacity for a peak that shows up a few hours a week and sitting idle the rest of the time. The fix is a hybrid: keep the baseline on local hardware where each token is nearly free and no data leaves your network, and burst only the overflow to a cloud API that you pay for by the token and only when you actually need it. The design problem is the router. A router that spills too early throws away the cost advantage you built the local tier for, and one that spills too late lets the queue hurt users before it reacts. This post covers how to set the spill threshold against queue depth rather than raw request rate, how to keep the cloud path from leaking data you meant to keep in house, what the blended cost actually looks like, and when a hybrid is worth its extra failure modes versus when one tier is the honest answer.
Jul 27, 2026 / AI Engineering If you run a single large model locally for everything, you are paying the full decode cost of that model on questions a much smaller one could have answered in a fifth of the time. A cascade fixes that: a small fast model takes the first pass, a cheap gate decides whether the answer is good enough, and only the queries that fail the gate get escalated to the big model. On an M5 Max the economics are compelling because the small model and the large model share the same unified memory, so there is no extra hardware and no network hop, just a routing decision. The catch is that a cascade only pays off if the escalation rate is low and the gate is cheap and accurate, because a gate that sends everything upstream anyway just adds latency on top of the big model you were already running. This post walks through how to size the two tiers, how to build a gate that is cheaper than the model it protects, what the measured latency and throughput look like on an M5 Max, and when the pattern is worth the extra moving parts versus when a single model is the honest answer.
Jul 26, 2026 / AI Engineering You bought a Mac with 128GB of unified memory to run a big model locally, and then the model that should fit refuses to stay resident and your tokens per second collapse. The reason is that macOS does not hand the whole pool to the GPU. There is a wired limit, the amount of memory the graphics side is allowed to pin, and by default it sits well below the installed capacity so the CPU, the window server, and the rest of the system keep breathing room. When the weights plus the KV cache push past that ceiling, the runtime spills to slower paths and the decode rate falls off a cliff, which people misread as a bandwidth or quantization problem when it is really an allocation problem. This post measures the default headroom on M4 and M5 machines, shows exactly where common model sizes cross the line, and gives the one sysctl knob that raises the ceiling safely, along with how far you can push it before you starve the OS. If you are speccing or tuning a machine to hold a large model resident, this is the number that actually decides whether it fits.
Jul 25, 2026 / AI Engineering When people shop for a Mac to run models locally, they read the GPU core count and the neural engine numbers. Those are the wrong numbers. Single stream token generation is bound almost entirely by memory bandwidth, because producing each token means reading every active weight out of unified memory, and the arithmetic per byte is tiny. That is why a chip with twice the bandwidth gives you roughly twice the tokens per second on the same model, and why adding GPU cores without adding bandwidth barely moves the decode figure. We measured llama style decode across M5 Pro, M5 Max, and M5 Ultra, put the tokens per second next to the rated bandwidth, and show where the picture flips: batched serving and prefill are compute bound and reward the cores, while everyday single user chat is bandwidth bound and rewards GB/s. If you are speccing a machine or trying to explain why your throughput is what it is, this is the model to reason with.
Jul 24, 2026 / AI Engineering There is a real reason to run OCR on your own hardware rather than a cloud document API: invoices, purchase orders, and contracts are exactly the documents you do not want leaving your network, and per page cloud pricing turns a back office batch job into a recurring bill. The question is whether a Mac can do it fast and accurately enough to replace the API. We put Surya, docTR, and PaddleOCR on an M5 Max, measured pages per second and character error rate on real business documents, and compared them against a local vision language model doing the same job, so you can see which engine fits which document and where the accuracy actually falls apart.
Jul 23, 2026 / AI Engineering Speculative decoding is the standard way to make local generation faster, but it costs you a second model resident in the same unified memory you are already fighting over. Prompt lookup decoding is the draft model free version of the same trick. Instead of a small model guessing the next tokens, it guesses them by finding a matching n-gram already sitting in the prompt and proposing the text that followed. On an M5 Max we measured how much it helps on RAG, summarization, and code editing, and why it is close to useless for open ended chat.
Jul 20, 2026 / AI Engineering Two GGUF files at the same bit width and the same file size can behave very differently, and the reason is often whether an importance matrix guided the quantization. imatrix does not shrink the model or speed it up. It spends the error budget more carefully, protecting the weights a calibration corpus proves matter most. We ran calibrated and plain quants of the same model on a 64GB M5 Max and measured perplexity, KL divergence against the full precision logits, and where the calibration step is worth running versus where it buys you nothing.
Jul 19, 2026 / AI Engineering Local transcription on Apple Silicon is not one problem with one answer. Whisper large-v3 and NVIDIA Parakeet solve it with opposite architectures, and the difference decides whether you get streaming captions or accurate multilingual batch transcripts. Whisper is an autoregressive decoder that reads a full 30 second window and is strong across 99 languages. Parakeet is a transducer that emits tokens as audio arrives and posts a lower word error rate on clean English at a fraction of the compute. We ran both on a 64GB M5 Max under MLX and mapped where each one wins.
Jul 18, 2026 / AI Engineering Mixture of Experts models are sold on the active parameter count. A 109B model that only fires 17B parameters per token sounds like it should run in the memory a 17B dense model needs. It does not. Every expert has to be resident because any token can route to any expert, so total parameters set the memory floor while active parameters set the speed. We measured both on a 64GB M5 Max, and the gap between the two numbers is exactly where teams size the machine wrong and end up swapping.
Jul 14, 2026 / AI Engineering ARC-AGI-2 is designed to be the one benchmark you cannot pretrain your way past, a set of visual reasoning puzzles where each task needs a rule inferred from a handful of examples rather than a fact recalled from training. That makes it a cleaner test of reasoning than the saturated coding and science suites, and it also makes it brutally expensive, because the models that score well spend a lot of reasoning tokens to get there. We compare GPT-5.6 Sol, Gemini 3.5 Pro, and Grok 4.5 on both axes that matter: how many tasks they solve, and what each solved task costs.
Jul 14, 2026 / AI Engineering Unified memory is the reason Apple Silicon runs big models at all, and it is also the reason a build that ran fine yesterday hangs today. The model weights, the KV cache, and whatever else your machine is doing all draw from one pool, and once the total crosses the wired limit the system starts swapping to disk. When that happens decode does not degrade gracefully, it falls off a cliff. We measured where the cliff sits on a 64GB M5 Max, why the KV cache is usually the thing that pushes you over, and how to size a model so you never reach it.