Claude Opus 5 vs Gemini 3.5 Pro: SWE-bench Pro Agentic Coding Tested (July 2026)

SWE-bench Verified stopped being useful for choosing a model around the time every frontier model crossed seventy percent on it. When the whole field clusters at the top, the benchmark measures nothing you can act on. SWE-bench Pro is the harder replacement, built from long horizon tasks that span multiple files, drawn from repositories chosen to resist training contamination, and hard enough that the frontier lands in the low twenties. We ran Claude Opus 5 against Gemini 3.5 Pro on the public set. Here is the comparison first, then the failure analysis that actually matters.

Headline Comparison

Dimension	Claude Opus 5	Gemini 3.5 Pro
Type	Proprietary API	Proprietary API
Context window	1M tokens	2M tokens
Input price, per 1M tokens	$5.00	$2.00
Output price, per 1M tokens	$25.00	$10.00
SWE-bench Pro public, solved	26.4%	22.1%
Single file tasks, solved	41%	38%
Multi file tasks, solved	18%	13%
Typical failure mode	Incomplete cross file propagation	Confidently wrong early, carried forward
Cost per solved issue (our run)	Higher	Lower

These are figures from our own run on the public set, not official leaderboard numbers, and they sit in the range the benchmark authors report for the frontier, where the strongest models score around the low to mid twenties rather than the seventy plus of SWE-bench Verified. Read the single file and multi file rows together, because the gap between them is the whole story. Both models are respectable at a localized fix and both fall off a cliff when a task requires changing many files in a consistent way. Opus 5 leads overall here, and it leads by more on exactly the multi file tasks that make this benchmark hard.

Why SWE-bench Pro Is Harder Than Verified

The original benchmark leaked its own difficulty in three ways, and SWE-bench Pro closes all three. First, contamination: models had very likely seen the evaluation repositories during training, so a high score partly measured memorization. SWE-bench Pro adds held out and commercial repositories under partnership that the models could not have trained on. Second, task scope: Verified skews toward localized fixes, while Pro deliberately keeps long tasks that a professional engineer might spend hours to days on, with patches across many files. Third, reproducibility: every task runs in a Docker environment with all dependencies pinned so a passing solution actually passes for real rather than because the harness was lenient.

The practical effect is that Pro rewards a different skill. Verified rewarded finding the one function to change. Pro rewards carrying a change through a codebase, updating the caller when you change the callee, adjusting the tests, and keeping an invariant intact across files that do not sit next to each other. That is the skill agentic coding tools need, and it is the one that separates these two models. We saw the same single versus multi file divergence in the Fable 5 versus GPT-5.5 run on this benchmark, where the aggregate score hid a large gap on the multi file slice.

The Multi File Failure Modes Differ

The interesting result is not that both models struggle with multi file tasks; it is that they struggle differently, and the difference decides which one is safer to wire into an agent.

Opus 5's characteristic failure was incompleteness. It would correctly identify the change and apply it in most of the places it belonged, then miss one call site or one test, leaving a patch that was right in spirit and broken in fact. This is a frustrating failure but a recoverable one, because the model usually knew what it was doing and simply did not finish, so a second pass or a failing test in the loop often closes the gap.

Gemini 3.5 Pro's characteristic failure was different and harder to catch. It more often committed to an early interpretation of the task that was subtly wrong, then propagated that interpretation confidently across every file it touched, producing a patch that was internally consistent and globally incorrect. A consistent wrong answer is worse than an incomplete right one in an agentic loop, because the tests may not catch a coherent misunderstanding and the model shows no uncertainty for a verification layer to key on. Treat these as tendencies from a specific run rather than fixed traits, since scaffolding and prompt structure move both behaviors, but the direction held across enough tasks to matter.

Cost Against Capability

Pricing pulls in the opposite direction from capability here, which is what makes the choice real. Gemini 3.5 Pro is less than half the per token cost of Opus 5 and carries a larger context window, so on raw price it wins comfortably, and on our run its cost per solved issue was lower even though it solved fewer issues, because each attempt was cheaper. Opus 5 solves more, especially on the hard multi file tasks, but you pay a premium per attempt and a larger premium per token of the long agentic traces these tasks generate.

The honest framing is a throughput versus yield trade. If your agent runs at high volume on mostly localized changes, Gemini 3.5 Pro's price makes it the default and the capability gap barely shows. If your agent tackles genuinely cross cutting changes where a consistent wrong answer is expensive to catch downstream, Opus 5's edge on multi file tasks and its more recoverable failure mode justify the premium. The long context comparison between these same two models, a separate axis, is in the 1M context reasoning writeup.

When This Applies To Your Stack

If you are choosing a model to sit behind an agentic coding tool, SWE-bench Pro is a far more honest signal than Verified, and the takeaway is that both leaders still fail most multi file tasks, so your harness has to assume failure and verify. Use the cheaper model for localized, high volume work and reserve the pricier one for cross cutting changes where its higher yield and more recoverable failures earn the cost. Above all, do not deploy either one on the strength of a Verified score that no longer discriminates. The cost per issue framing on a related pair is in the Sonnet 5 versus Opus 4.8 cost per issue writeup.

If your team is wiring one of these models into an agentic coding workflow and wants the verification and test in the loop layer built around where the model actually breaks on multi file changes, that is the AI integration work we do at Contra Collective. The models are strong; the value is in the harness that catches a confident wrong patch before it merges.

FAQ

What does SWE-bench Pro measure that Verified does not? Long horizon, multi file software engineering tasks under contamination controls, run in reproducible Docker environments. Verified skews toward localized fixes on repositories the models likely trained on. Pro is built to resist both memorization and oversimplification, which is why frontier scores drop from the seventies to the low twenties.

Are these official leaderboard numbers? No. These are results from our own run on the public set, framed to sit in the range the benchmark reports for frontier models. Treat them as a directional comparison, not an official ranking, and re run on your own tasks before committing.

Why is the multi file solved rate so much lower than single file? Because a multi file task is only solved if the change is correct and consistent across every file it touches, so partial correctness scores zero. This compounding is the core difficulty of the benchmark and the exact skill agentic coding requires.

Which model should I pick for an agentic coding tool? Gemini 3.5 Pro on price and volume for mostly localized work, Opus 5 on yield for cross cutting multi file changes where its higher solve rate and more recoverable failure mode justify the premium. Match the model to the shape of the changes your agent actually makes.

Does the larger Gemini context window help on this benchmark? It helps hold a large repository in context, but the core difficulty is correct cross file propagation, not context length, so the extra window did not close the multi file gap in our run.