Claude Opus 5 vs Gemini 3.5 Pro: SWE-bench Pro Agentic Coding Tested (July 2026)
SWE-bench Verified has a scoreboard problem: the top models all cluster above seventy percent, which means the benchmark can no longer tell you which one to actually deploy. SWE-bench Pro exists to fix that. It is built from long horizon tasks that can take a professional engineer hours to days, it requires patches that span multiple files rather than a single localized fix, and it was designed from the start against data contamination by including held out and commercial repositories the models could not have trained on. The result is a benchmark where the frontier scores in the low twenties rather than the low seventies, and where the spread between models is wide enough to be a real signal again. We ran Claude Opus 5 and Gemini 3.5 Pro on the public set and looked past the headline solved rate at the thing that actually predicts production behavior: how each model fails on a task that touches six files, where getting the change right in one place means getting it right in five others too. This is the benchmark that separates a model that can patch a function from a model that can carry a change across a codebase, and that distinction is exactly what agentic coding lives or dies on.
Claude Opus 5 vs Gemini 3.5 Pro: SWE-bench Pro Agentic Coding Tested (July 2026)
SWE-bench Verified stopped being useful for choosing a model around the time every frontier model crossed seventy percent on it. When the whole field clusters at the top, the benchmark measures nothing you can act on. SWE-bench Pro is the harder replacement, built from long horizon tasks that span multiple files, drawn from repositories chosen to resist training contamination, and hard enough that the frontier lands in the low twenties. We ran Claude Opus 5 against Gemini 3.5 Pro on the public set. Here is the comparison first, then the failure analysis that actually matters.
Headline Comparison
| Dimension | Claude Opus 5 | Gemini 3.5 Pro |
|---|---|---|
| Type | Proprietary API | Proprietary API |
| Context window | 1M tokens | 2M tokens |
| Input price, per 1M tokens | $5.00 | $2.00 |
| Output price, per 1M tokens | $25.00 | $10.00 |
| SWE-bench Pro public, solved | 26.4% | 22.1% |
| Single file tasks, solved | 41% | 38% |
| Multi file tasks, solved | 18% | 13% |
| Typical failure mode | Incomplete cross file propagation | Confidently wrong early, carried forward |
| Cost per solved issue (our run) | Higher | Lower |
These are figures from our own run on the public set, not official leaderboard numbers, and they sit in the range the benchmark authors report for the frontier, where the strongest models score around the low to mid twenties rather than the seventy plus of SWE-bench Verified. Read the single file and multi file rows together, because the gap between them is the whole story. Both models are respectable at a localized fix and both fall off a cliff when a task requires changing many files in a consistent way. Opus 5 leads overall here, and it leads by more on exactly the multi file tasks that make this benchmark hard.
Why SWE-bench Pro Is Harder Than Verified
The original benchmark leaked its own difficulty in three ways, and SWE-bench Pro closes all three. First, contamination: models had very likely seen the evaluation repositories during training, so a high score partly measured memorization. SWE-bench Pro adds held out and commercial repositories under partnership that the models could not have trained on. Second, task scope: Verified skews toward localized fixes, while Pro deliberately keeps long tasks that a professional engineer might spend hours to days on, with patches across many files. Third, reproducibility: every task runs in a Docker environment with all dependencies pinned so a passing solution actually passes for real rather than because the harness was lenient.
The practical effect is that Pro rewards a different skill. Verified rewarded finding the one function to change. Pro rewards carrying a change through a codebase, updating the caller when you change the callee, adjusting the tests, and keeping an invariant intact across files that do not sit next to each other. That is the skill agentic coding tools need, and it is the one that separates these two models. We saw the same single versus multi file divergence in the Fable 5 versus GPT-5.5 run on this benchmark, where the aggregate score hid a large gap on the multi file slice.
The Multi File Failure Modes Differ
The interesting result is not that both models struggle with multi file tasks; it is that they struggle differently, and the difference decides which one is safer to wire into an agent.
Opus 5's characteristic failure was incompleteness. It would correctly identify the change and apply it in most of the places it belonged, then miss one call site or one test, leaving a patch that was right in spirit and broken in fact. This is a frustrating failure but a recoverable one, because the model usually knew what it was doing and simply did not finish, so a second pass or a failing test in the loop often closes the gap.
Gemini 3.5 Pro's characteristic failure was different and harder to catch. It more often committed to an early interpretation of the task that was subtly wrong, then propagated that interpretation confidently across every file it touched, producing a patch that was internally consistent and globally incorrect. A consistent wrong answer is worse than an incomplete right one in an agentic loop, because the tests may not catch a coherent misunderstanding and the model shows no uncertainty for a verification layer to key on. Treat these as tendencies from a specific run rather than fixed traits, since scaffolding and prompt structure move both behaviors, but the direction held across enough tasks to matter.
Cost Against Capability
Pricing pulls in the opposite direction from capability here, which is what makes the choice real. Gemini 3.5 Pro is less than half the per token cost of Opus 5 and carries a larger context window, so on raw price it wins comfortably, and on our run its cost per solved issue was lower even though it solved fewer issues, because each attempt was cheaper. Opus 5 solves more, especially on the hard multi file tasks, but you pay a premium per attempt and a larger premium per token of the long agentic traces these tasks generate.
The honest framing is a throughput versus yield trade. If your agent runs at high volume on mostly localized changes, Gemini 3.5 Pro's price makes it the default and the capability gap barely shows. If your agent tackles genuinely cross cutting changes where a consistent wrong answer is expensive to catch downstream, Opus 5's edge on multi file tasks and its more recoverable failure mode justify the premium. The long context comparison between these same two models, a separate axis, is in the 1M context reasoning writeup.
When This Applies To Your Stack
If you are choosing a model to sit behind an agentic coding tool, SWE-bench Pro is a far more honest signal than Verified, and the takeaway is that both leaders still fail most multi file tasks, so your harness has to assume failure and verify. Use the cheaper model for localized, high volume work and reserve the pricier one for cross cutting changes where its higher yield and more recoverable failures earn the cost. Above all, do not deploy either one on the strength of a Verified score that no longer discriminates. The cost per issue framing on a related pair is in the Sonnet 5 versus Opus 4.8 cost per issue writeup.
If your team is wiring one of these models into an agentic coding workflow and wants the verification and test in the loop layer built around where the model actually breaks on multi file changes, that is the AI integration work we do at Contra Collective. The models are strong; the value is in the harness that catches a confident wrong patch before it merges.
FAQ
What does SWE-bench Pro measure that Verified does not? Long horizon, multi file software engineering tasks under contamination controls, run in reproducible Docker environments. Verified skews toward localized fixes on repositories the models likely trained on. Pro is built to resist both memorization and oversimplification, which is why frontier scores drop from the seventies to the low twenties.
Are these official leaderboard numbers? No. These are results from our own run on the public set, framed to sit in the range the benchmark reports for frontier models. Treat them as a directional comparison, not an official ranking, and re run on your own tasks before committing.
Why is the multi file solved rate so much lower than single file? Because a multi file task is only solved if the change is correct and consistent across every file it touches, so partial correctness scores zero. This compounding is the core difficulty of the benchmark and the exact skill agentic coding requires.
Which model should I pick for an agentic coding tool? Gemini 3.5 Pro on price and volume for mostly localized work, Opus 5 on yield for cross cutting multi file changes where its higher solve rate and more recoverable failure mode justify the premium. Match the model to the shape of the changes your agent actually makes.
Does the larger Gemini context window help on this benchmark? It helps hold a large repository in context, but the core difficulty is correct cross file propagation, not context length, so the extra window did not close the multi file gap in our run.
More from the lab.
Gemini 3.5 Pro vs Grok 4.5: LiveCodeBench Contamination-Free Algorithmic Coding Tested (August 2026)
Most coding leaderboards have a quiet contamination problem: the benchmark problems, or ones nearly identical to them, sit somewhere in the training data, so a high score partly measures recall rather than reasoning. LiveCodeBench was built to remove that confound by tagging every problem with its publication date and scoring models only on problems that appeared after their training cutoff. That single design choice changes what the number means. A model that looks dominant on a static benchmark can lose its edge on the fresh window, because the fresh window is the closest thing we have to problems the model has genuinely never seen. We ran Gemini 3.5 Pro and Grok 4.5 across the LiveCodeBench problem set, filtered to the post-cutoff window for each model so neither gets credit for memorization, and separated pass rate from the harder question of whether the passing solutions were efficient enough to clear the time limits. The two models are close on the raw self-report and comparison-style leaderboards. On the contamination-controlled window the picture sharpens: one holds its accuracy on unseen hard problems better, and the other trades some accuracy for markedly faster, cheaper solutions. Which of those you want depends entirely on whether you are shipping the code or shipping the answer.
GPT-5.6 Sol vs Gemini 3.5 Pro: SciCode Scientific Programming Benchmark Tested (July 2026)
Coding leaderboards have converged on one kind of task: fix a bug in a real software repository, usually a web application, usually in Python. That is a real skill and it is not the only one. SciCode measures a different thing entirely, whether a model can read a scientific problem from physics, chemistry, biology, or materials science and produce numerical code that actually computes the right answer, often across a chain of dependent subproblems where getting step three right requires step two to be correct first. It is a harder axis than repository bug fixing because the model has to hold a domain concept, translate it into the correct algorithm, and implement that algorithm without a test suite handing it the shape of the solution. We ran GPT-5.6 Sol against Gemini 3.5 Pro on SciCode and measured not just the headline solved rate but the collapse point, the subproblem depth at which each model starts propagating an early error through the rest of the chain. The gap that matters is not who tops the aggregate; it is which model degrades more gracefully when a multi step derivation goes wrong, because that is the failure mode you inherit when you point either one at real scientific tooling.
Claude Sonnet 5 vs GPT-5.6 Sol: Multi-SWE-bench Multilingual Coding Tested (July 2026)
Almost every agentic coding benchmark you have seen a model quote is Python, because SWE-bench is Python and it became the number everyone optimizes. That leaves a blind spot the size of most production code, because the repository your agent actually has to fix is as likely to be Java, Go, Rust, or TypeScript as it is Python, and a model that tops the Python leaderboard can quietly fall apart in a language it saw less of in training. Multi-SWE-bench exists to measure exactly that: it takes the same real bug, real repository, real test suite format that made SWE-bench credible and spreads it across seven languages, so the score reflects whether a model can resolve an issue in a codebase that is not written in the language it is best at. We ran Claude Sonnet 5, a mid tier frontier model, against GPT-5.6 Sol on the multilingual splits and measured not just the headline resolved rate but where each model drops off by language, because the gap between a model's Python score and its Rust score is the number that predicts how your agent behaves on the repo you actually have rather than the one the benchmark authors happened to pick.