Terminal-Bench 2.0 vs 3.0: Why the Hardest Coding Agent Benchmark Keeps Getting Harder

Terminal-Bench 2.0 shipped January 17, 2026 as a set of 89 genuinely difficult terminal tasks, run inside real containers against human written solutions and strict automated tests, covering work like configuring legacy systems, reimplementing research papers from scratch, and solving multi-step software engineering problems that do not reduce to a single file edit. It is, by design, closer to a full day of a senior engineer's actual terminal work than a toy coding puzzle. And within months of launch, the top models were already clearing it at a rate that should make anyone who tracks benchmarks suspicious: a self-reported 82.0 percent for Claude Mythos Preview, 73.2 percent for GPT-5.5 on high effort settings, 70.3 percent self-reported for Qwen3.7 Plus.

Those numbers matter less on their own than in contrast to what we could find for Terminal-Bench 3.0, where the best public score we located is Z.ai's GLM-5.3 at 28.3 percent, trailing OpenAI's GPT-5.6 Sol at 34.6 percent. Going from models clearing 70 to 82 percent of a benchmark to the field's best struggling to clear 35 percent is not a story about models getting worse. It is a story about what a benchmark author does, deliberately, when a benchmark starts answering the wrong question.

The saturation problem benchmark authors keep running into

A benchmark exists to separate strong systems from weak ones. The moment most frontier models clear 70 to 80 percent of it, that separation collapses, everyone looks similarly good, and the benchmark stops telling buyers or researchers anything useful about which system is actually stronger. This is not a new problem specific to Terminal-Bench; it is the same arc that played out with earlier versions of SWE-bench and most other agentic coding benchmarks that eventually got a harder successor. The honest response from a benchmark author facing this is to raise the difficulty bar substantially for the next version, accepting that scores will drop across the board, rather than let the old version keep running past its useful life while every model claims a mid-80s score that stopped meaning much.

That appears to be exactly what happened between Terminal-Bench 2.0 and 3.0. The task count and construction methodology details for 3.0 were not available in the sources we could verify directly, which is itself a reminder to treat third party reporting of benchmark internals cautiously, but the score collapse across models that attempted both versions, GLM-5.3 improving from a prior generation's 4.6 percent to 28.3 percent on Terminal-Bench 3.0 while comfortably clearing 70-plus percent territory on easier agentic tasks, is consistent with a genuinely harder task set rather than a regression in model capability.

What this means if you are choosing a model based on a benchmark score

The practical lesson for anyone using Terminal-Bench, or any single agentic benchmark, to inform a model selection decision is to check which version of the benchmark a score refers to before comparing it to anything else. A Terminal-Bench 2.0 score of 73 percent and a Terminal-Bench 3.0 score of 28 percent are not comparable numbers even though they share a benchmark name, and a vendor citing a Terminal-Bench score without the version number attached is giving you a number that cannot be checked against anyone else's. This is the same discipline we apply when comparing coding agents on SWE-bench and related benchmarks: the version, the exact task set, and the self-reported versus independently verified distinction all matter more than the headline percentage.

It is also worth treating self-reported scores, Claude Mythos Preview's 82.0 percent and Qwen3.7 Plus's 70.3 percent on Terminal-Bench 2.0 among them, with a different level of confidence than independently verified leaderboard entries like GPT-5.5's 73.2 percent. A self-reported number is not automatically wrong, but it has not been run against the same adversarial conditions an independent leaderboard applies, and the gap between self-reported and independently verified scores on the same benchmark is itself a useful signal about how much daylight to expect between a vendor's launch claims and what you will see running the model yourself.

When this applies to your stack

If your team is making an agentic coding tool or model selection decision partly on benchmark scores, which is reasonable as one input among several, build the habit of checking benchmark version and reporting methodology before treating any single percentage as decisive, the same way you would check which dataset split a traditional ML metric was measured against. For clients where we are advising on agentic coding tool and model choice as part of a broader engineering infrastructure engagement, benchmark scores are one input into a broader evaluation that includes your actual codebase, your team's workflow, and real task completion on your own repository, not a substitute for running the candidates against your own work.

FAQ

Why did scores drop so much between Terminal-Bench 2.0 and 3.0? The most likely explanation, consistent with how other agentic benchmarks have evolved, is that version 3.0 raised the task difficulty specifically because 2.0 was becoming saturated, with top models clearing 70 to 82 percent of it.

Does a lower Terminal-Bench 3.0 score mean today's models are worse than the ones that scored well on 2.0? No. The models attempting 3.0 are generally newer than the ones that scored highest on 2.0, and the lower percentage reflects task difficulty, not capability regression.

Should I trust self-reported benchmark scores? Treat them as a starting point rather than a verified result, and prefer independently run leaderboard entries when comparing across vendors, since self-reported numbers have not been checked under the same conditions.

Is Terminal-Bench a good proxy for how a model will perform on my codebase? It is a useful signal about general terminal and systems level task competence, but it is not a substitute for testing a model against your own repository and workflows, which is where most of the variance we see with clients actually shows up.