Aug 6, 2026 / AI Engineering Streaming is the default for anything a person reads, because watching tokens appear at reading speed feels immediate in a way that waiting nine seconds for a finished paragraph never will. So teams turn on server sent events without a second thought, and then a nagging question surfaces the first time they profile a local server under load: is the streaming itself costing throughput? Every token becomes a separate write to the socket, a separate flush, a separate trip through the HTTP stack, and it is reasonable to worry that all that per token overhead is stealing cycles the GPU could be spending on decode. On a hosted API you would never notice, because the provider absorbs it. On a single M5 Max where you own the whole box, the question is fair and worth measuring rather than guessing. We ran the same prompts through a local llama-server in streamed and buffered modes at one request and at eight concurrent, watched aggregate tokens per second and time to first token, and the answer has a useful shape: the decode loop barely cares, but streaming changes two things that matter more than raw throughput, and knowing which one you are optimizing for tells you when to stream and when not to.
Aug 5, 2026 / AI Models
Most benchmark leaderboards have a contamination problem: the test questions leak into training data, scores drift upward, and a number that once meant something quietly stops predicting anything. LiveBench was built to sidestep that by refreshing its question set on a monthly cadence and drawing from recent sources a model could not have trained on, which makes a strong LiveBench score harder to fake than a strong score on a static, well circulated test. It is also broad where LiveCodeBench is narrow, spanning coding, reasoning, mathematics, data analysis, language, and instruction following rather than algorithmic puzzles alone, so it reads more like a general capability profile than a single skill. We ran Claude Opus 5 and Grok 4.5 across the LiveBench categories with a fixed harness and one attempt per item, and the interesting result is not the overall average, which is close, but the shape of the split underneath it: the two models are strong in different places, and which one is right for your product depends entirely on which of those categories your workload actually lives in.
Aug 5, 2026 / AI Engineering The first time a local inference service meets real traffic, it does not slow down gracefully, it falls off a cliff. A single request against a 14B model on an M5 Max streams at eighty tokens a second and feels like a hosted API. Then a second user arrives, and a third, and a burst of eight lands at once, and suddenly latency has tripled, the machine is paging to disk, and one unlucky request gets an out of memory kill mid generation. Nothing in the model changed. What changed is that concurrency on a single Apple Silicon box is bounded by unified memory, not by raw compute, because every in flight request holds a slice of KV cache that lives in the same pool the model weights and the operating system are already using. Once the concurrent working set crosses the wired memory limit, the machine does the worst possible thing under load, which is to keep accepting work it cannot serve. The fix is not a bigger model server flag. It is admission control: decide how many requests the box can actually hold, queue a bounded number behind them, and reject the rest with honest backpressure instead of pretending. This post measures where the ceiling sits on a 128GB M5 Max, explains why it is a memory ceiling and not a throughput one, and lays out the gateway pattern that keeps a local service predictable when the traffic is not.
Aug 5, 2026 / Headless Commerce
Most teams add a second payment processor for the wrong reason and discover the right one too late. The wrong reason is pricing leverage. The right one is that a single processor is a single point of failure on the one system that turns traffic into revenue, and the day it has an incident, or declines a wave of legitimate cards in a region, or raises rates at renewal, you want the ability to route around it without a migration. That ability is what a payment orchestration layer sells: a unified API in front of many processors, a vault that holds card data independent of any one of them, routing rules that pick a processor per transaction, and failover when one blinks. Primer, Gr4vy, and Spreedly all offer this, and they are not interchangeable. They disagree on where the vault lives, how much routing logic you write versus configure, and how easy it is to leave, which for a layer whose entire pitch is escaping lock in is the question that should decide the purchase. This is how the three compare for a headless commerce stack, and how to adopt one without quietly recreating the dependency you were trying to break.
Aug 4, 2026 / AI Models
A model that scores in the ninetieth percentile on a coding benchmark can still wreck an agent, because writing correct code and calling a tool correctly are different skills, and the second one is where production agents actually fall over. The failure is rarely dramatic. The model calls a function that does not exist, or it fills a required parameter with a plausible guess, or it emits malformed arguments that your parser rejects, or it fires four calls in sequence when the situation called for one, and each of those quietly derails a workflow in a way that never shows up on a reasoning test. The Berkeley Function Calling Leaderboard, BFCL, is the benchmark built to measure this specific competence: not whether a model can reason, but whether it selects the right tool, formats the call correctly, knows when to call nothing, and handles parallel and multi turn calling without falling apart. We ran GPT-5.6 Sol, Claude Opus 5, and Gemini 3.5 Pro across the BFCL categories with attention to the parts that predict real agent reliability, separated raw accuracy from the more telling error types, and looked at what each model does when the honest answer is to not call a function at all. The headline is that all three are strong on simple single calls and separate sharply on the harder categories, which is exactly where agents live.
Aug 4, 2026 / AI Engineering The instinct is reasonable: you built a local inference service, it runs fast on your M5 Max, and now you want to ship it the way you ship everything else, in a container. So you write a Dockerfile, install your runner, point it at the model, and watch throughput fall off a cliff. A model that streamed at ninety tokens per second natively is suddenly doing eight, and the fans are not even spinning, because the container is running the whole thing on CPU. This is not a misconfiguration you can fix with a flag. Docker Desktop on a Mac runs your containers inside a lightweight Linux virtual machine, and that virtual machine has no access to the Apple GPU. There is no equivalent of the NVIDIA container toolkit for Metal, no device to pass through, because the hypervisor never exposes the GPU to the guest in the first place. The result is a hard architectural boundary that catches teams by surprise when they try to treat an Apple Silicon inference box like a Linux one with a CUDA card. This post explains exactly where the boundary sits, measures what containerization actually costs on an M5 Max, and lays out the deployment pattern that lets you keep the operational tidiness of containers without paying the GPU tax.
Aug 4, 2026 / Headless Commerce
Translation looks like a solved problem until you try to do it inside a headless commerce stack, at which point it splits into three problems that live in three different systems and refuse to be handled the same way. Your marketing content sits in a headless CMS, your product titles and descriptions sit in a commerce platform or a PIM, and your interface strings, the buttons and errors and empty states, sit in your codebase. A translation tool that is excellent at one of these is often useless at the others, and the mistake teams make is picking a single vendor for the whole job and discovering too late that it was built for a slice of it. Weglot, Crowdin, and Lokalise are the three names that come up most, and they are not really competitors so much as tools aimed at different layers of the same stack. Weglot proxies and translates a rendered site with almost no engineering. Crowdin and Lokalise are translation management systems built around developer workflows and structured content, and they differ from each other in ways that matter once you are past the demo. This post maps each tool to the layer it actually fits, lays out the integration shape for a headless storefront, and is honest about where each one leaves a gap you will have to fill yourself.
Aug 3, 2026 / AI Models
Most math benchmarks are already solved in the sense that models score in the high nineties, which tells you the ceiling has been hit and the number no longer separates one frontier model from another. FrontierMath was built to keep separating them. Its problems are original, unpublished, and reviewed by working mathematicians, which strips out the memorization that inflates scores on AIME or MATH, and its hardest band, Tier 4, is research grade: problems that take a specialist hours or days and that no model reliably solves. That makes it the rare benchmark where the scores are low on purpose and the gaps between models are real signal rather than noise near a saturated ceiling. We ran Claude Opus 5, GPT-5.6 Sol, and Gemini 3.5 Pro across the FrontierMath tiers with attention to the Tier 4 tail, separated the problems a model genuinely reasoned through from the ones it fluked with a lucky final answer, and looked at how much compute each spent to get there. The headline is not that any of them is good at Tier 4, because none of them is. It is that they fail differently, and the way a model fails on problems past its ability tells you more about whether to trust it on your hard problems than a saturated benchmark ever could.
Aug 3, 2026 / Headless Commerce
In a headless stack the data you want to report on is scattered across systems that were never meant to be queried together: Shopify Plus holds orders and customers, NetSuite holds the financial truth and inventory, a payment processor holds settlement, and a marketing platform holds engagement. Nobody makes decisions from four separate admin panels, so all of it has to land in one warehouse where you can join it, and the pipeline that moves it is the unglamorous piece that decides whether your analytics are trustworthy or a source of arguments. The choice usually comes down to three shapes: a managed connector service like Fivetran that you pay to make the problem disappear, an open platform like Airbyte that trades some of that convenience for control and a different cost curve, or a custom pipeline you build and own outright. The right answer is not a matter of taste; it turns on specific things, how many sources you have, how weird NetSuite's API is going to be for your configuration, how much schema drift you can tolerate, and whether your data volume makes a usage priced service cheap or ruinous. This post compares the three on the dimensions that actually determine the outcome, with the Shopify Plus and NetSuite pairing as the concrete case, because that pairing is where most headless commerce analytics projects get hard.
Aug 3, 2026 / AI Engineering Everyone benchmarks decode, the tokens per second you watch stream back, because it is the number that feels like speed. Prefill is the part nobody measures until it hurts, and at long context it is where the wait actually lives. Prefill is the pass that reads the entire prompt before the model emits a single token, and unlike decode it is compute bound rather than memory bandwidth bound, so it stresses a completely different part of the machine. On an M5 Max that distinction is the whole story: decode stays comfortable well past 100K tokens because the memory bandwidth is generous, but prefill time climbs faster than the prompt does, because attention carries an order n squared term that only shows its teeth once the context is large. A prompt that takes a few seconds to process at 8K can take minutes at 128K, and if that prompt is a RAG context, a long document, or an agent history that grows every turn, the user is staring at a blank screen the entire time. This post measures how time to first token scales across the 8K to 128K range on an M5 Max, separates the linear part of the cost from the superlinear part, and covers the levers that actually move it: chunked prefill, prompt caching, and knowing when a smaller model that fits the job beats a larger one you keep re-reading the prompt into.
Aug 2, 2026 / AI Models
Most coding leaderboards have a quiet contamination problem: the benchmark problems, or ones nearly identical to them, sit somewhere in the training data, so a high score partly measures recall rather than reasoning. LiveCodeBench was built to remove that confound by tagging every problem with its publication date and scoring models only on problems that appeared after their training cutoff. That single design choice changes what the number means. A model that looks dominant on a static benchmark can lose its edge on the fresh window, because the fresh window is the closest thing we have to problems the model has genuinely never seen. We ran Gemini 3.5 Pro and Grok 4.5 across the LiveCodeBench problem set, filtered to the post-cutoff window for each model so neither gets credit for memorization, and separated pass rate from the harder question of whether the passing solutions were efficient enough to clear the time limits. The two models are close on the raw self-report and comparison-style leaderboards. On the contamination-controlled window the picture sharpens: one holds its accuracy on unseen hard problems better, and the other trades some accuracy for markedly faster, cheaper solutions. Which of those you want depends entirely on whether you are shipping the code or shipping the answer.
Aug 2, 2026 / AI Engineering The model produces a probability distribution over the whole vocabulary at every step, and the sampler is the piece that turns that distribution into a single token. This is the part of the local inference stack teams tune last and understand least, which is odd because it is the setting that most directly decides whether output is repetitive, whether it hallucinates under pressure, and whether a JSON response parses. The defaults matter more than people think: most runners ship a temperature and a top-p tuned for open ended chat, and those same values are actively wrong for code generation, structured extraction, or an agent loop that needs deterministic tool calls. Worse, the parameters interact. Temperature reshapes the distribution before top-p or min-p truncate it, so changing one silently changes what the others do, and a repetition penalty stacked on top can push a model off a correct but repeated token into a wrong one. On Apple Silicon this is all essentially free to change: sampling happens on a handful of logits after the expensive forward pass, so it costs no extra memory and negligible time, which means there is no throughput reason not to tune it. This post walks through what each knob does to the distribution, how they interact, and what settings hold up for the three jobs most local deployments actually run, using an M5 Max as the test bench.