Cloudflare's GLM-5.3 KV Cache Leak: What 155 Cross Tenant Responses Mean for Shared AI Infrastructure

Cloudflare's coordinated disclosure, published October 26, 2026 alongside a detailed SGLang engineering postmortem, describes a cross request KV cache exposure in GLM-5.3 on Workers AI. Between August 27 and September 9, 2026, 155 responses to very long prompts returned content from other customers' sessions. Every affected request exceeded 681,280 tokens. Cloudflare reports no evidence of deliberate exploitation once its own testing traffic and the reporting researcher's requests were excluded. The bug lived at the intersection of a capacity misconfiguration and a serving engine defect, and it is a clean case study in exactly where multi tenant LLM infrastructure actually breaks.

Two separate problems, stacked on top of each other

The first problem was a straightforward capacity mismatch. GLM-5.3 was deployed on Workers AI with a configured context window of 1,048,576 tokens, but the prefill KV cache pool backing that configuration only held 681,280 tokens. A request could fit comfortably inside the advertised context window while exceeding the actual memory available to process it correctly, and Cloudflare's own long context testing checked HTTP response codes and performance, not output correctness, so the mismatch shipped unnoticed. The second problem was a bug in SGLang itself, filed as issue 30233 back on July 6, 2026: in disaggregated prefill and decode mode, a request that exceeded capacity and should have been rejected was instead swapped for a one token placeholder. Prefill still generated KV cache for it and signaled completion anyway, so the decode node proceeded to attend over stale cache pages left behind by earlier, unrelated requests, returning an HTTP 200 success instead of the HTTP 400 error the single node serving path would have correctly returned for the same situation.

The nine hours that mattered

Cloudflare's own incident timeline, measured in UTC on September 9, 2026, runs tight. At 08:00, Cloudflare began verifying the researcher's report. By 13:22, the team had reproduced the issue using two independent accounts and formally declared a security incident. At 14:00 the model was set to private, and by 14:35 the root cause was identified. Prefill nodes were moved to a corrected configuration at 14:52, a full redeployment with a Cloudflare authored engine patch went out at 19:04, and the model was restored to public access at 19:40. Cloudflare's own status page frames the same window slightly differently in Eastern time: incident began at 10:25 AM EDT, identified at 11:02 AM EDT, resolved at 4:09 PM EDT. Either way, from confirmed report to public restoration took under 12 hours, with the active incident response itself running roughly 9 hours.

The fix, and the version number that actually matters

Cloudflare's immediate mitigation expanded the prefill KV cache pool to about 2.04 million tokens, roughly double the configured context length and matching the setup already used for GLM-5.2, and deployed an engine patch that rejects over capacity requests at admission time, before any KV transfer begins, closing the exact gap the SGLang bug had opened. Cloudflare validated both changes against the original researcher's reproduction script and its own canary marker tests before restoring public access. Upstream, Cloudflare's proposed fix landed as SGLang pull request 38704, followed by the merged fix in PR 38935, with a related change, PR 36988, merged September 6. The combined fixes shipped in SGLang v0.5.20, with follow up hardening in v0.5.21 on October 2 and v0.5.22 on October 19. Cloudflare's own recommendation is to run v0.5.22 or later; the fix was not backported to earlier release branches, which means anyone still running an older SGLang build with disaggregated prefill and decode serving is carrying the same exposure today. SGLang reports the fix costs essentially nothing in throughput, within plus or minus 2 percent across 21 H100 runs, with deferred KV release measured at a 0.38 percent cost across 16 H200 runs, a difference SGLang itself calls not significant.

What this means if you run shared inference infrastructure

The uncomfortable lesson here generalizes well beyond Cloudflare. KV cache pages are not zeroed between requests by default in high throughput serving engines, which means page ownership tracking is not an optional nicety, it is the actual security boundary between tenants sharing the same GPU memory. A health check built around HTTP status codes and raw performance numbers will miss this class of bug completely, because the system was responding fast and returning 200 the entire time it was leaking data, which means output correctness checks, not just availability checks, belong in the monitoring stack for anyone serving multiple tenants off shared inference capacity. Cloudflare's own longer term fix, a prototype that tags every physical KV page and verifies its mapping before a decode read, measured under 1 percent throughput and latency cost in testing, but was disabled shortly after an August trial on GLM-5.2 due to reliability issues, which tells you this is still an open engineering problem industry wide, not a solved one.

FAQ

Which specific SGLang version fixes this? Cloudflare recommends v0.5.22 or later. The fixes shipped across v0.5.20, v0.5.21, and v0.5.22, and were not backported to earlier branches.

Was this exploited by an attacker? Cloudflare reports no evidence of deliberate exploitation once its own testing traffic and the reporting researcher's requests are excluded from the 155 affected responses.

Does this affect every model on Cloudflare Workers AI? No. Cloudflare states its investigation found no other Workers AI model affected, only @cf/zai-org/glm-5.3 during the specific window from August 27 to September 9, 2026.

What should a team running its own disaggregated SGLang deployment check first? Measure prefill KV cache pool size against the configured context window on every node after any change to weights, quantization, parallelism, or memory fraction, and confirm the deployed SGLang version includes both admission path fixes described in Cloudflare's disclosure.

When this applies to your stack

Any shared inference deployment, whether it is a public cloud AI product or an internal multi tenant platform serving several business units off the same GPU pool, carries this exact class of risk until KV cache ownership is verified rather than assumed. Contra Collective helps engineering teams audit and harden self hosted and shared LLM serving infrastructure against isolation failures like this one before they become an incident report.