Self Hosting Open Weight LLMs in 2026: GPU Serving Cost vs the API for Kimi K3 and GLM-5.3
The August open weight releases changed what self hosting means. Kimi K3 ships as a 2.8 trillion parameter Mixture of Experts model and GLM-5.3 ties it as the strongest open weight option on the aggregate intelligence index, so for the first time the open models are genuinely competitive with the closed frontier on real work. That makes the old question new again: should you self host, or should you keep paying an API? The weights are free. The serving is not. A trillion parameter class MoE has to be resident across a lot of GPU memory to serve at low latency, and the fixed cost of that hardware does not care whether you send it one request or a million. This post does the break even math without hand waving: what it actually costs to keep an open weight model warm on GPUs, how that fixed cost compares to per token API pricing, where the crossover point sits, and why utilization, not the model, is the number that decides the whole thing. The short answer is that self hosting wins at high, steady utilization and loses badly at low or spiky load, and most teams overestimate their utilization.
Self Hosting Open Weight LLMs in 2026: GPU Serving Cost vs the API for Kimi K3 and GLM-5.3
The weights are free. That sentence is why the self hosting conversation restarted this month, and it is also the trap. Kimi K3 shipped as a 2.8 trillion parameter open Mixture of Experts model, GLM-5.3 tied it as the top open weight model on the aggregate intelligence index, and suddenly you can download a model that competes with the closed frontier. Downloading it is the cheap part. Serving it at production latency is where the money goes, and the math is not intuitive. This post does it honestly.
Free weights, expensive residency
A Mixture of Experts model splits its parameters across many experts and activates only a few per token, which is why Kimi K3 can have 2.8 trillion total parameters but only route through 16 of its 896 experts on any given token. That sparsity makes it cheap to compute per token. It does not make it cheap to host, because the router can select any expert at any step, so all of them have to be resident in fast memory. Total parameters set your memory bill; active parameters set your compute bill. We walked through that exact split in the context of local hardware in our analysis of running DeepSeek-V4-Pro locally, and it governs GPU serving just as strictly.
Do the residency math for a trillion parameter class model. At 4-bit quantization a rough rule is half a byte per parameter plus overhead. A 2.8T model lands somewhere near 1.4TB of weights before you add any KV cache. No single accelerator holds that, so you shard across many GPUs, and the GPUs have to stay powered and allocated whether or not traffic is flowing. That standing allocation is the fixed cost that dominates everything else.
| Model class | Rough 4-bit weight size | Minimum serving footprint | Standing cost driver |
|---|---|---|---|
| 2.8T MoE (Kimi K3 class) | ~1.4TB | large multi GPU node or cluster | many GPUs kept warm |
| 200B to 400B class | ~100GB to 200GB | one dense GPU node | a full node kept warm |
| 30B to 70B class | ~15GB to 40GB | single high memory GPU | one GPU kept warm |
GLM-5.3 sits well below Kimi K3 in serving cost because it is a smaller model, which is part of why it is the more practical self hosting target for most teams even though the two tie on the intelligence index. The right open model to self host is usually the smallest one that clears your quality bar, not the biggest one you can find.
The break even is about utilization, not the model
Here is the calculation that actually decides it. An API charges you per token, so your API bill scales linearly with usage: send nothing, pay nothing. Self hosting charges you per hour of GPU time whether or not you use it, so your cost is nearly fixed once the hardware is allocated. The two lines cross at a specific utilization, and below that point the API is cheaper, above it self hosting is cheaper.
# break-even sketch (fill in your own numbers)
gpu_node_cost_per_hour = sum(gpu_hourly_rate * gpu_count) # standing cost
node_throughput_tok_s = measured tokens/sec at your batch size
self_host_cost_per_1M_tok = gpu_node_cost_per_hour / (node_throughput_tok_s * 3600 / 1e6)
# compare against:
api_cost_per_1M_tok = provider list price
# self hosting wins only when:
# self_host_cost_per_1M_tok < api_cost_per_1M_tok
# which requires high, sustained utilization of node_throughput
The variable that swings this is utilization. A GPU node kept warm and running at 80 percent of its throughput amortizes its fixed cost across a huge number of tokens, and the self hosted cost per million tokens drops below API pricing. The same node at 5 percent utilization, which is what most internal tools actually run at, spreads that fixed cost across almost nothing, and the effective cost per token can be many times the API price. The model does not change this. Utilization does.
Where self hosting clearly wins
Self hosting is the right call in three situations, and they share a shape. The first is high steady throughput: a workload that keeps the GPUs busy around the clock, like a high volume classification or extraction pipeline, a coding agent running continuously across a large team, or batch processing that you can pack tightly. At that load the fixed cost is spread thin and the per token economics beat any API.
The second is data control. If your workload cannot send data to an external provider for regulatory or contractual reasons, self hosting is not an optimization, it is a requirement, and the cost comparison is against not shipping the feature at all. The third is version stability: hosting your own weights means a provider update never silently changes model behavior in the middle of your quarter, which for some production systems is worth real money to guarantee. We covered the commerce framing of this same tradeoff in our cost analysis of self hosting versus API for ecommerce, and the logic carries across domains.
Where the API clearly wins
The API wins whenever load is low, spiky, or unpredictable, which describes most applications honestly assessed. If your traffic has big peaks and long quiet stretches, you would have to provision GPUs for the peak and pay for them through the quiet, and that idle time destroys the economics. The API also wins when you want the newest frontier capability without an infrastructure project, when your team does not have the operational depth to run distributed inference reliably, and when your volume is simply too low to ever cross the break even line.
There is an operational cost that never shows up in the token math and consistently gets underestimated: the engineering time to run a sharded inference cluster well. Distributed serving, autoscaling, failover, KV cache management, and keeping throughput high under real traffic are a real system, not a script. We described the moving parts of a distributed setup in our multi Mac cluster piece, and a production GPU cluster is that problem at a larger scale. If you do not have the team for it, the API's higher per token price is buying you an entire operational capability, and that is usually a good trade.
A decision rule
Estimate your sustained utilization honestly, then let it decide. If you can keep serving hardware busy above roughly half its throughput around the clock, model the self hosting cost seriously, because it will likely win. If your utilization is low or spiky, stay on the API until a specific, measured workload justifies a dedicated node. And when data control or version stability is a hard requirement, self host regardless of the token math, because the comparison is no longer purely about cost. Prefer the smallest open model that clears your quality bar, which today often means GLM-5.3 rather than Kimi K3, and reserve the largest models for the tasks that genuinely need them.
When this applies to your stack
The open weight frontier is now good enough that self hosting is a real option rather than a compromise, but the decision is an infrastructure decision, not a model decision. It turns on utilization, data requirements, and whether you have the team to run distributed inference well. Get those three right and self hosting can cut your inference bill substantially; get them wrong and you will pay more than the API while carrying an operational burden you did not need.
Sizing that break even for a specific workload, choosing the right open model and quantization, and deciding between a self hosted cluster and a managed API is exactly the kind of AI infrastructure work we do with teams putting models into production. The weights being free is the start of the analysis, not the end of it.
FAQ
Is self hosting an open weight model cheaper than an API? Only at high, sustained utilization. Self hosting is a fixed cost per GPU hour, so it wins when the hardware stays busy and loses when it sits idle. Most low or spiky workloads are cheaper on an API.
Can I self host Kimi K3? Technically yes, the weights are public, but at 2.8 trillion parameters it needs a large multi GPU footprint kept warm. For most teams a smaller open model like GLM-5.3 is the more practical self hosting target.
What is the biggest hidden cost of self hosting? Idle GPU time and the engineering effort to run distributed inference reliably. Neither shows up in per token pricing, and both routinely make self hosting more expensive than teams expect.
When should I self host regardless of cost? When data cannot leave your network for compliance reasons, or when you need guaranteed version stability so a provider update cannot change model behavior underneath your product.
More from the lab.
vLLM vs SGLang: Choosing a Production LLM Inference Engine in Late 2026
vLLM and SGLang are both open source, both built to serve LLMs at production scale, and both frequently recommended for the exact same job, which is exactly why teams keep picking the wrong one. The real difference is not a benchmark number. It is which attention and caching architecture matches your actual traffic pattern, and most teams never check that before they commit infrastructure budget to one or the other.
Ollama's MLX Backend Beats Metal by 93 Percent: Apple Silicon Local Inference Tested (October 2026)
Ollama's 0.19 release notes say it plainly: on Apple Silicon, Ollama is now built on top of Apple's MLX framework to take advantage of its unified memory architecture, rather than the Metal backed llama.cpp path it shipped for years. On an M4 Max running Qwen3.5, that swap took decode speed from 57.8 tokens per second to 111.4, a 93 percent improvement, and prefill from 1,154 to 1,810 tokens per second. The catch is in the fine print: the MLX runner is in preview, it needs 32GB or more of unified memory to activate at all, and as of this release it only accelerates one model family. Here is what actually changed, what it costs to turn on, and where it still falls back silently to the slower path.
DGX Spark vs Mac Studio M5 Ultra: Local LLM Inference Benchmarked (October 2026)
The M5 Ultra Mac Studio shipping in September 2026 is a quad die chip with 1.2 terabytes per second of memory bandwidth and up to 256GB of unified memory, with a 512GB configuration arriving in late October. Against a single NVIDIA DGX Spark, it is not close: on a 27B dense model, the Mac decodes at roughly four times the speed. Cluster two DGX Spark units together over NVIDIA's GX10 networking instead, and the comparison flips for a different workload entirely, concurrent multi user serving. Neither machine is the right answer for every workload, and the decision comes down to whether you are serving one user or several at once.