Prompt Caching in Production: When It Actually Cuts Your LLM Bill

Prompt caching is the cost optimization most teams know exists and few implement well. The reason is that the pitch, up to 90 percent off cached input, sounds like a free discount, and the mechanics that decide whether you actually get it are unglamorous. Here is the landscape in one table before the arithmetic that matters, drawn from provider documentation as of 2026.

Provider	Cached input discount	Write cost	Storage model
Anthropic	0.10x base input (90 percent off)	1.25x for 5 minute cache, 2.0x for 1 hour	tied to TTL, no separate storage fee
OpenAI	automatic, up to 90 percent on current models	free, automatic above 1024 tokens	ephemeral, no explicit storage fee
Google Gemini	up to 90 percent on cached tokens	explicit caching adds setup	billed by stored token time, per hour

The single most important fact in that table is not the discount. It is the line every provider shares and none advertises: caching only ever touches input tokens. Output token cost is untouched by any caching scheme. That one rule determines whether caching is a large lever for your workload or a rounding error.

The rule that decides everything: input heavy or not

Because caching discounts only input, its value scales with how input heavy your calls are. A workload with a huge stable prompt and a short completion is the ideal case: most of what you pay for is the input, most of the input repeats, and the discount lands on the majority of your bill. A workload that sends a tiny prompt and generates a long answer is the opposite: the input is already cheap, the output dominates the cost, and caching saves you almost nothing no matter how high the hit ratio.

So the first question is not "should I cache" but "what fraction of my token spend is input, and how much of that input repeats." Retrieval augmented generation with a large fixed instruction block and document context, agent loops that resend a growing but mostly stable transcript, and classification pipelines with a long few shot preamble are all input heavy with a reusable prefix, and they are where caching earns real money. Chatbots with short prompts and long generated replies are not, and chasing caching there is effort spent on the wrong lever. The cost per task discipline from the Kimi K3 and Opus 5 cost comparison applies here too: measure where your tokens actually go before optimizing.

The write premium and the break even hit ratio

The discount is not free on the first call. Anthropic charges 1.25x the base input rate to write a five minute cache entry and 2.0x for a one hour entry, and Google adds a storage fee for explicit caching billed by stored token time. OpenAI is the exception where writes are automatic and free above the token threshold, which makes its caching close to pure upside for eligible prompts. Where a write premium exists, caching is a bet: you pay more on the write in exchange for cheap reads, and the bet pays off only if the entry is read enough times before it expires.

Work the break even. If a cache write costs 1.25x base input and each read costs 0.10x, the write premium over a normal call is 0.25x, and each cache hit saves you 0.90x versus paying full input again. So a single subsequent hit more than repays the premium. The real risk is not the write cost; it is caching something that is never reused, or reused after the TTL has expired. A five minute window is short. If your reuse pattern is spread over minutes with gaps, entries expire between hits and you pay the write premium repeatedly for one time reads. Match the TTL to your actual reuse cadence, and where the one hour window costs 2.0x, only take it when the traffic genuinely sustains reuse across the hour.

Prefix ordering is the implementation that makes or breaks it

Caching keys on the prefix of the prompt, which means the structure of your prompt is the whole game. Providers cache the longest stable leading span of tokens; the moment the content diverges, the cache stops applying. So anything that changes per call belongs at the end, and everything stable belongs at the front, in a fixed order.

The failure mode is subtle. If you put a timestamp, a request id, a per user variable, or a freshly ordered set of retrieved documents near the top of the prompt, you invalidate the cache for everything after it, and your beautiful ten thousand token instruction block gets billed at full price on every call. The fix is to order the prompt as a stack of stability: system instructions and few shot examples first because they never change, then the semi stable context, then the per request variables and the user turn last. Retrieved chunks should be ordered deterministically so the same query produces the same prefix. This is unglamorous prompt plumbing, and it is the difference between a 90 percent discount and a zero percent discount on the same nominal feature.

Where caching does not help, and what does instead

Caching is not a universal cost lever, and it is worth naming where it fails so you reach for the right tool. It does nothing for output heavy generation, where the fix is a cheaper or faster model and tighter output limits. It does nothing when every prompt is genuinely unique with no reusable prefix. And it does not change the fundamental economics when your real constraint is data control or sustained high volume, where the relevant question is the one from the self hosting versus API analysis: at what volume does running your own open weight model on your own hardware beat paying per token at all.

Caching composes with the other levers rather than replacing them. Route the cheap tier work to a cheap model, cache the stable prefixes, batch what can be batched, and cap output length, and the compounding is where the real savings live. The tokenizer and throughput differences that drive production cost across models still matter; caching sits on top of that choice, it does not override it.

When this applies to your stack

Reach for prompt caching when your calls are input heavy and carry a large, stable prefix that repeats within a short window: retrieval augmented generation, agent loops with a growing transcript, and few shot classification pipelines are the clear wins. Structure the prompt as a stack of stability with the immutable content first and the per request variables last, match the cache TTL to your real reuse cadence, and remember that no caching scheme ever discounts output tokens.

Building the prompt architecture that actually earns the cached input discount, measuring the input to output ratio and hit rate that decide whether caching is a large lever or a rounding error, and combining it with routing, batching, and output limits into a cost model that tracks your real workload is the AI infrastructure work Contra Collective takes on when a team's inference bill has outgrown guesswork. The 90 percent discount is real. Whether you capture it depends entirely on how your prompts are built.

FAQ

How much can prompt caching save on my LLM bill? Up to 90 percent on the cached input portion of your spend, since all three major providers now discount cached input heavily. The realized saving depends on how input heavy your calls are and how often the cached prefix is reused, because caching never discounts output tokens.

Does prompt caching reduce output token costs? No. Every provider's caching scheme discounts only input tokens. If your workload is dominated by long generated outputs, caching saves little, and the better lever is a cheaper or faster model with tighter output limits.

What is the write premium on prompt caching? Anthropic charges 1.25x the base input rate for a five minute cache write and 2.0x for a one hour write, and Google adds a storage fee for explicit caching. OpenAI writes are automatic and free above the token threshold. Where a premium exists, caching pays off once the entry is read enough times before it expires.

Why is prompt prefix order important for caching? Providers cache the longest stable leading span of the prompt, so anything that changes per call must go at the end. Put a timestamp, request id, or freshly ordered retrieval near the top and you invalidate the cache for everything after it, losing the discount on your entire instruction block.

When should I self host instead of relying on caching? When your real constraint is data control or sustained very high volume. Caching lowers per call cost on the hosted API, but at large scale or under strict data requirements the more fundamental question is whether running your own open weight model on your own hardware beats paying per token at all.

Caching discounts, write premiums, and storage models reflect provider documentation as of 2026 and vary by model and change over time; confirm current rates for your specific models and measure your input to output ratio and cache hit rate before committing to a caching architecture.