Oct 4, 2026 / AI Engineering Security researchers at OX Security found that the Model Context Protocol's STDIO transport interface runs any process command passed to it on the host system before it ever confirms the subprocess is a valid MCP server, an execute first, validate never pattern that completes an injected command even when the caller eventually gets an error message back. The scope is not small: 150 million downloads across package registries, more than 7,000 publicly accessible MCP servers, and up to 200,000 vulnerable instances across every officially supported SDK, Python, TypeScript, Java, and Rust. Anthropic's response, after OX disclosed in January 2026, was to update documentation and call the behavior intentional. Here is what that actually means if your team runs MCP servers over STDIO.
Sep 30, 2026 / AI Engineering Buried inside OpenAI's DevDay 2026 announcement list alongside GPT-6.1 Sol and Dots agents was a smaller, more technically specific release: a Decisions API built for real time decision making against predefined answer sets. Where the Agents API and a general purpose agent loop are built for open ended reasoning across an unbounded task, the Decisions API is built for the opposite case, a small number of valid outcomes and a requirement that the choice happen fast. This post works through what that distinction actually means in practice and where a bounded decision primitive fits into an existing agentic workflow.
Aug 8, 2026 / AI Engineering Most writing about local inference assumes a human is on the other end, watching tokens appear and judging the model by how fast the first one lands. Batch work is the opposite situation, and it is more common than the interactive framing suggests: you have a backlog of tens or hundreds of thousands of prompts, nobody is waiting on any single one, and the only thing that matters is how much of the pile you can clear before morning. Product descriptions for a catalog, classification over a support archive, embeddings and summaries for a corpus, synthetic data for a fine tune. In that regime latency is irrelevant and throughput is everything, and the settings that make a local server feel responsive to a person actively work against you, because they optimize for the wrong number. Running one prompt at a time on an M5 Max leaves most of the machine idle, since a single decode stream cannot saturate the memory bandwidth the chip has to offer. The job is to keep many sequences in flight so the expensive weight reads are amortized across all of them, and to arrange the backlog so the GPU never waits on padding, scheduling, or a slow tail. This post is about doing that deliberately: what actually bounds batch throughput on Apple Silicon, how far batching takes you before memory stops you, and the operational scaffolding that turns a fragile overnight run into one you can resume when it dies at 3am.
Aug 7, 2026 / AI Engineering Prompt caching skips the prefill on a repeated prefix, but it does nothing for two users who ask the same thing in different words. On a single Apple Silicon box, where every generation competes for the same unified memory, the cheapest token is the one you never generate. Semantic response caching sits in front of the model, embeds the incoming prompt, and if a past prompt was close enough in meaning it returns the stored answer without touching the GPU. The hard part is not the cache. It is deciding when two questions are actually the same question, because a threshold set too loose will confidently serve the wrong answer. This post builds the gateway, measures the hit rate on a real support workload, and sets the similarity threshold where a wrong answer costs more than a cache miss.
Aug 6, 2026 / AI Engineering Streaming is the default for anything a person reads, because watching tokens appear at reading speed feels immediate in a way that waiting nine seconds for a finished paragraph never will. So teams turn on server sent events without a second thought, and then a nagging question surfaces the first time they profile a local server under load: is the streaming itself costing throughput? Every token becomes a separate write to the socket, a separate flush, a separate trip through the HTTP stack, and it is reasonable to worry that all that per token overhead is stealing cycles the GPU could be spending on decode. On a hosted API you would never notice, because the provider absorbs it. On a single M5 Max where you own the whole box, the question is fair and worth measuring rather than guessing. We ran the same prompts through a local llama-server in streamed and buffered modes at one request and at eight concurrent, watched aggregate tokens per second and time to first token, and the answer has a useful shape: the decode loop barely cares, but streaming changes two things that matter more than raw throughput, and knowing which one you are optimizing for tells you when to stream and when not to.
Aug 5, 2026 / AI Engineering The first time a local inference service meets real traffic, it does not slow down gracefully, it falls off a cliff. A single request against a 14B model on an M5 Max streams at eighty tokens a second and feels like a hosted API. Then a second user arrives, and a third, and a burst of eight lands at once, and suddenly latency has tripled, the machine is paging to disk, and one unlucky request gets an out of memory kill mid generation. Nothing in the model changed. What changed is that concurrency on a single Apple Silicon box is bounded by unified memory, not by raw compute, because every in flight request holds a slice of KV cache that lives in the same pool the model weights and the operating system are already using. Once the concurrent working set crosses the wired memory limit, the machine does the worst possible thing under load, which is to keep accepting work it cannot serve. The fix is not a bigger model server flag. It is admission control: decide how many requests the box can actually hold, queue a bounded number behind them, and reject the rest with honest backpressure instead of pretending. This post measures where the ceiling sits on a 128GB M5 Max, explains why it is a memory ceiling and not a throughput one, and lays out the gateway pattern that keeps a local service predictable when the traffic is not.
Aug 4, 2026 / AI Engineering The instinct is reasonable: you built a local inference service, it runs fast on your M5 Max, and now you want to ship it the way you ship everything else, in a container. So you write a Dockerfile, install your runner, point it at the model, and watch throughput fall off a cliff. A model that streamed at ninety tokens per second natively is suddenly doing eight, and the fans are not even spinning, because the container is running the whole thing on CPU. This is not a misconfiguration you can fix with a flag. Docker Desktop on a Mac runs your containers inside a lightweight Linux virtual machine, and that virtual machine has no access to the Apple GPU. There is no equivalent of the NVIDIA container toolkit for Metal, no device to pass through, because the hypervisor never exposes the GPU to the guest in the first place. The result is a hard architectural boundary that catches teams by surprise when they try to treat an Apple Silicon inference box like a Linux one with a CUDA card. This post explains exactly where the boundary sits, measures what containerization actually costs on an M5 Max, and lays out the deployment pattern that lets you keep the operational tidiness of containers without paying the GPU tax.
Aug 3, 2026 / AI Engineering Everyone benchmarks decode, the tokens per second you watch stream back, because it is the number that feels like speed. Prefill is the part nobody measures until it hurts, and at long context it is where the wait actually lives. Prefill is the pass that reads the entire prompt before the model emits a single token, and unlike decode it is compute bound rather than memory bandwidth bound, so it stresses a completely different part of the machine. On an M5 Max that distinction is the whole story: decode stays comfortable well past 100K tokens because the memory bandwidth is generous, but prefill time climbs faster than the prompt does, because attention carries an order n squared term that only shows its teeth once the context is large. A prompt that takes a few seconds to process at 8K can take minutes at 128K, and if that prompt is a RAG context, a long document, or an agent history that grows every turn, the user is staring at a blank screen the entire time. This post measures how time to first token scales across the 8K to 128K range on an M5 Max, separates the linear part of the cost from the superlinear part, and covers the levers that actually move it: chunked prefill, prompt caching, and knowing when a smaller model that fits the job beats a larger one you keep re-reading the prompt into.
Aug 2, 2026 / AI Engineering The model produces a probability distribution over the whole vocabulary at every step, and the sampler is the piece that turns that distribution into a single token. This is the part of the local inference stack teams tune last and understand least, which is odd because it is the setting that most directly decides whether output is repetitive, whether it hallucinates under pressure, and whether a JSON response parses. The defaults matter more than people think: most runners ship a temperature and a top-p tuned for open ended chat, and those same values are actively wrong for code generation, structured extraction, or an agent loop that needs deterministic tool calls. Worse, the parameters interact. Temperature reshapes the distribution before top-p or min-p truncate it, so changing one silently changes what the others do, and a repetition penalty stacked on top can push a model off a correct but repeated token into a wrong one. On Apple Silicon this is all essentially free to change: sampling happens on a handful of logits after the expensive forward pass, so it costs no extra memory and negligible time, which means there is no throughput reason not to tune it. This post walks through what each knob does to the distribution, how they interact, and what settings hold up for the three jobs most local deployments actually run, using an M5 Max as the test bench.
Aug 1, 2026 / AI Engineering The context length printed on a model card is a training decision, not a hard ceiling. Most open weight models trained at 32K can be run at 64K or 128K by rescaling the rotary position embeddings at inference time, and this is exactly what happens when you set a rope scaling factor or hand a model to YaRN. The mechanism is real and it works, which is precisely why it is dangerous. There is no assertion, no truncation warning, no crash. You set the factor, the model accepts a longer prompt, and it generates fluent text. What changes is subtler: the model's ability to locate and reason over specific tokens degrades as the effective positions get stretched further from what it saw in training. On a needle in a haystack test the retrieval accuracy sags in the middle of the extended window. On real work the answers get plausibly wrong, citing the right document but the wrong number, summarizing the first half of a contract and forgetting the clause on page forty. This post is about that trade specifically on Apple Silicon, where unified memory makes a 128K context physically possible on a single machine and therefore makes the temptation to just crank the factor very real. We cover what RoPE scaling actually does to positions, why YaRN degrades more gracefully than naive linear scaling, how to measure the degradation instead of hoping it is not there, and where the honest ceiling sits for a model you did not train yourself.
Jul 31, 2026 / AI Engineering In memory prefix caching is the well known win for local inference: hold a stable system prompt in the KV cache and never prefill it twice. The part nobody plans for is what happens when the process exits. The cache lives in RAM, so it dies with the server, and the very next request after a restart, a crash, or a fresh deploy pays the full prefill cost again as if the cache never existed. On a long context that is not a rounding error. A 60K token document or repository context can cost several seconds of prefill on an M5 Max, and if your workload restarts often, ships on a schedule, or runs as a short lived batch job, you are paying that tax over and over for a prefix that never actually changed. MLX has a direct answer that most local setups leave switched off: the prompt cache is serializable. You can precompute the KV cache for a fixed prefix once, write it to a safetensors file on disk, and load it back on startup so the first token comes out at warm cache speed even though the process is brand new. This post is about that mechanism specifically, the disk persistence layer rather than the in memory hit. We look at how the cache file is produced and consumed, what the load actually costs versus a cold prefill, the constraints that make a saved cache silently wrong if you ignore them, and the workloads where persisting to disk is the difference between a viable local service and one that stutters on every boot.
Jul 30, 2026 / AI Engineering The mental model most people bring to a vision language model is wrong in a way that costs them latency. They picture an image as a single input, roughly one unit of work, the way a text token is one unit of work. What actually happens is that the vision encoder chops the image into patches, and a high resolution photo can turn into hundreds or thousands of visual tokens that the language model must then prefill through before it generates anything at all. On a cloud endpoint you never feel this, because someone else eats the prefill and bills you a flat per image rate. Run the same model on your own M5 Max and the cost becomes visible immediately: the model loads fine, it holds in unified memory with room to spare, and then a single screenshot at native resolution takes several seconds to first token because you just asked the machine to prefill four thousand tokens it manufactured out of one picture. This post is about that gap. We look at where the memory actually goes when you load a VLM locally, why image tokens and not model weights are usually your latency problem, how the resolution setting is the one knob that moves both accuracy and speed, and when a local VLM is the right call versus a cloud vision API. The economics of local vision are real, but only if you understand that the image, not the prompt, is the expensive part.