AKV Cuts LLM Agent Inference Costs 28 Percent: Inside Tsinghua's New KV Cache Engine

Researchers at Tsinghua University and CUHK open sourced AKV in late October 2026, a serving system and policy library for KV cache engineering in LLM agents, published as the paper Cache Engineering for Agents on arXiv and released under the Apache 2.0 license at github.com/thu-nics/AKV. The pitch is specific rather than aspirational: across three models and four agent benchmarks, AKV runs at an average cost 27.7 percent lower and an average score 1.7 points higher than the common alternative of summarizing old context, and on SWE-bench Verified with GPT-OSS-120B, cost per task falls from 20.0 to 9.7 cents while the score actually rises from 40.0 to 43.2. It ships no model weights, it requires NVIDIA GPUs, and it is built as an extension of SGLang rather than a standalone server.

Dropping cache entries instead of summarizing them

The standard way to control a long running agent's context cost is text engineering: once the conversation crosses a token threshold, summarize the older portions into a shorter block and keep going. AKV does something structurally different. Instead of summarizing history into new text, it drops selected KV cache entries from future attention while leaving the actual text trajectory completely untouched. The model's conversation history, as text, never changes. What changes is which cached key and value pairs the attention mechanism is still allowed to see. AKV defines three operations to manage this: Append, the standard update for new tokens; Drop, which removes selected entries from future attention without shifting the positions of what remains; and Repos, which rewrites the positions of retained keys using a RoPE rotation while leaving the underlying values unchanged. In ablations, adding Repos alone raised GPT-OSS-120B's pass rate on the BrowseComp-Plus benchmark from 55 percent to 62 percent.

The API, and what a request actually looks like

AKV exposes a stateless, OpenAI compatible interface. A client sends the full message history as usual, plus two optional fields: drop_message, which marks specific prior turns for cache eviction, and reposition, which triggers the Repos operation on what remains. A simplified request shape looks like this:

POST /v1/chat/completions
{
  "model": "gpt-oss-120b",
  "messages": [ ...full conversation history... ],
  "drop_message": ["msg_14", "msg_15", "msg_16"],
  "reposition": true
}

Built in policies handle the decision of what to drop automatically: turn based, token based, and model based policies all ship with the system, and a CacheTool lets the model decide to call drop_context on itself mid conversation rather than waiting for an external policy to trigger. The main policy used in AKV's own benchmarking is a tool response rolling drop that keeps only the latest 12 tool responses in active cache, compared against a Standard baseline that summarizes at 96,000 tokens, 192,000 for MiroThinker, and a Text Engineering baseline that summarizes at 48,000 tokens.

The actual numbers, model by model

Averaged across three models and four agent benchmarks, AKV runs 27.7 percent cheaper than the Standard baseline and 21.9 percent cheaper than Text Engineering, while scoring 1.7 points higher on average than Text Engineering. Broken out: MiniMax-M2.7 scores 52.4 percent at 5.0 cents under Standard, 52.8 percent at 5.7 cents under Text Engineering, and 53.7 percent at 4.4 cents under AKV. GPT-OSS-120B scores 51.0 percent at 11.2 cents under Standard, 48.3 percent at 10.2 cents under Text Engineering, and 53.1 percent at 7.3 cents under AKV. MiroThinker-1.7-Mini scores 44.1 percent at 20.4 cents under Standard, 37.1 percent at 18.1 cents under Text Engineering, and 45.8 percent at 14.8 cents under AKV. On the serving side, tested with GPT-OSS-120B across 80 long context SWE-bench Verified tasks, AKV delivers 1.18 times the output throughput, 1.53 times the input throughput, and 1.51 times faster P95 latency at peak settings, with the gap widening sharply under load: at concurrency 16, P95 latency runs 229.90 seconds under Standard versus 30.63 seconds under AKV.

Where it does not help, and what it requires

AKV is not a universal win. On BrowseComp-Plus specifically, MiniMax-M2.7 scores 58.9 under AKV against 62.3 under Text Engineering, a 3.4 point regression, which the researchers report plainly rather than smoothing over. Letting the model control its own cache eviction through the CacheTool also has mixed effects: it raised GPT-OSS-120B's score from 52.1 to 61.0 on a 100 task subset, but raised MiniMax-M2.7's cost from 5.4 to 5.9 cents, trading model autonomy for a modest cost increase on at least one model. On infrastructure, AKV extends SGLang's radix tree with new drop and reposition node types, frees dropped KV cache only once no active request still needs it, and restores evicted state with a technique called segment batch attention that handles the restoration in a single prefill pass. It runs on NVIDIA GPUs exclusively, with no CPU path, no MLX support, and no GGUF format support, which rules it out for an Apple Silicon serving stack entirely. The published benchmarks used 8 A800-80GB GPUs in BF16, and the quick start configuration serves GPT-OSS-120B on just 2 GPUs with tensor parallelism of 2.

FAQ

Does AKV work with quantized models in GGUF format? No. AKV builds on SGLang and runs on NVIDIA GPUs only, with no CPU, MLX, or GGUF path described in the released documentation.

What models does AKV currently support? The released policies and benchmarks cover Qwen, GPT-OSS, and MiniMax-M2 text models.

Is AKV a drop in replacement for conversation summarization? Not quite a replacement so much as an alternative mechanism. It keeps the conversation text completely unchanged and instead controls what the attention mechanism can still see in cache, which is a different tradeoff than rewriting history into a shorter summary.

Where does AKV actually lose to simpler approaches? The paper reports one clear case: MiniMax-M2.7 on BrowseComp-Plus scores lower under AKV than under Text Engineering, a result the researchers disclose directly rather than omitting.

When this applies to your stack

Cache engineering of this kind only matters once an agent pipeline is running at enough volume that cost per task is a real budget line rather than a rounding error, which is exactly the point where a self hosted NVIDIA deployment starts to make more economic sense than per token API billing. Contra Collective helps engineering teams design and operate self hosted inference stacks where this class of optimization actually pays for the engineering effort it costs to implement.