Cognition's SWE-2 vs Claude Opus 5.5 vs Gemini 4 Argon: What the Benchmark Numbers Actually Mean

Cognition built SWE-2 by post training Kimi K3, a 2.8 trillion parameter base model, with reinforcement learning that trains the medium, high, and max effort levels in a single run rather than as separate fine tuning passes. It ships a 1M token context window and explicit reasoning traces, and it has five published benchmark results, though the publishing site notes it does not qualify for a public overall rank because of how those scores were produced: provider run evaluations using Cognition's own Devin CLI, at the model's best reasoning effort setting, with every row flagged as harness specific. That last caveat is not boilerplate. It is the reason the four numbers above tell a coherent story only once you stop reading them as one capability score and start reading them as four separate, narrowly scoped measurements.

Why the same model looks strong and weak at once

Terminal-Bench 2.1 and Terminal-Bench 4.0 sound like the same benchmark at two version numbers, incrementally harder. They are not close to that in practice, which is why Terminal-Bench 4.0's frontier saturation problem matters here directly: 4.0 was built specifically because 2.1 had stopped separating frontier models from each other, with too many agents clearing most of its tasks. A 92.8 percent on 2.1 and a 27.3 percent on the same family's 4.0 revision is not really one model regressing by 65 points. It is closer to one model acing an easier, more saturated test and then meeting a genuinely harder one for the first time, the same pattern that pushed Epoch AI to build FrontierSWE v2 with a 20 hour task budget once shorter benchmarks stopped producing useful separation at the top of the field.

The DeepSWE and FrontierCode results sit in between, both close losses rather than routs, 4.9 points behind Gemini 4 Argon on DeepSWE and 4.4 points behind Claude Opus 5.5 on FrontierCode. Gemini 4 Argon's own benchmark profile, built to sustain deep reasoning across long horizon workflows in Google's own framing, is a plausible explanation for why it edges out SWE-2 specifically on DeepSWE rather than the other benchmarks in this set: DeepSWE rewards the kind of sustained reasoning Argon was explicitly tuned for. None of these gaps, 4.9 points, 4.4 points, are the kind of number that should drive a tool selection decision on their own. A 43.3 point gap on Terminal-Bench 4.0 is a different category of signal entirely, and conflating it with the smaller gaps elsewhere in the same table is the single most common way these comparisons get misread.

The harness problem nobody puts in the headline

Every one of SWE-2's published scores comes from Cognition's own Devin CLI, run at the model's best reasoning effort setting, by the provider itself rather than an independent evaluator. That is explicitly flagged on the benchmark catalog entry as a reason the model does not qualify for a public overall rank, which is a more honest disclosure than most vendor benchmark pages offer, and it should set your confidence level accordingly. A provider run score at best effort tells you the ceiling of what the model can do under ideal conditions with its own tooling. It does not tell you what you will get running the same model through a different harness, at a default effort setting, inside your own CI pipeline. The comparison we ran against Opus 4.8, GPT-5.5, and Gemini 3.1 Pro earlier this year made the same point from the other direction: harness choice alone can move a score by a wider margin than most of the model to model gaps these tables report.

What to actually take from this comparison

Do not rank SWE-2 against Claude Opus 5.5 and Gemini 4 Argon as a single ordered list. Read each benchmark gap independently, weighted by how close it is to your actual workload. If your work looks like the long, sustained reasoning DeepSWE and Terminal-Bench 4.0 test for, the 4.9 point and 43.3 point gaps both point away from SWE-2 toward Argon or Sonnet 5.5 for that specific job. If your work looks like the shorter, more contained tasks Terminal-Bench 2.1 still measures reasonably well, SWE-2's 92.8 percent is a genuinely strong result worth testing against your own CI. Either way, treat every number in that table as Cognition's best case, provider run, best effort figure, and validate it against your own pipeline with your own default settings before it drives a procurement decision.

When this applies to your stack

If you are evaluating coding agents for a workload that involves long running, multi hour engineering tasks rather than short, well scoped functions, weight Terminal-Bench 4.0 and FrontierSWE v2 style results far more heavily than Terminal-Bench 2.1 or a provider's own best effort Devin CLI number, because that is where SWE-2's gaps actually show up. Running your own harness level evaluation against the specific agents your team is choosing between, rather than trusting a vendor's leaderboard entry directly, is evaluation work we help clients run at Contra Collective when standing up agentic engineering workflows.

FAQ

What model is SWE-2 built on? Kimi K3, a 2.8 trillion parameter base model, post trained by Cognition with reinforcement learning across three effort levels in a single run.

Why does SWE-2 score so differently on Terminal-Bench 2.1 versus Terminal-Bench 4.0? Terminal-Bench 4.0 was built specifically because 2.1 had become saturated at the frontier, with too many models clearing most of its tasks. The two versions test meaningfully different levels of difficulty despite the similar name.

Are SWE-2's benchmark scores independently verified? No. The published scores are provider run evaluations using Cognition's own Devin CLI at best reasoning effort, which is why the benchmark catalog does not assign SWE-2 a public overall rank.

Does SWE-2 beat Claude Opus 5.5 or Gemini 4 Argon overall? There is no single answer. SWE-2 leads on Terminal-Bench 2.1, trails Gemini 4 Argon by 4.9 points on DeepSWE, trails Claude Opus 5.5 by 4.4 points on FrontierCode, and trails Claude Sonnet 5.5 by 43.3 points on Terminal-Bench 4.0. Which model wins depends entirely on which benchmark best matches your actual workload.