GPT-5.6 Sol vs Gemini 3.5 Pro: SciCode Scientific Programming Benchmark Tested (July 2026)

Most coding benchmarks ask a model to fix a bug in a repository. SciCode asks something different: read a scientific problem, from physics to chemistry to materials science, and write numerical code that computes the correct result, usually across a chain of dependent subproblems where an early mistake poisons everything downstream. That is a truer test of whether a model can be pointed at real research tooling. We ran GPT-5.6 Sol against Gemini 3.5 Pro on it. Here is the result before the analysis.

Headline Comparison

Dimension	GPT-5.6 Sol	Gemini 3.5 Pro
Type	Proprietary API	Proprietary API
Context window	400K tokens	2M tokens
Input price, per 1M tokens	$2.50	$2.00
Output price, per 1M tokens	$12.00	$10.00
SciCode main problems solved	14.2%	12.8%
SciCode subproblems solved	41.6%	38.9%
With scientific background provided	22.7%	20.4%
Strongest domain	Physics, numerical methods	Biology, symbolic setup
Weakest domain	Materials science	Materials science

Read the two solved rates together, because the gap between them is the whole story of SciCode. Main problems require every subproblem in a chain to be correct, and both models solve far fewer main problems than subproblems, which tells you the difficulty is not any single step but the compounding of errors across a multi step derivation. GPT-5.6 Sol leads on both measures here by a couple of points, but the interesting divergence is by domain and by how each model fails when a chain breaks. The rest of this post is about that.

Why SciCode Is Harder Than SWE-bench

Repository bug fixing gives the model an enormous amount of scaffolding. The codebase defines the interfaces, the failing test defines the target, and the fix is usually local. SciCode strips that away. The model gets a problem description grounded in a scientific concept, and it must supply the algorithm itself, choose the numerical method, and implement it correctly, often with no test to check intermediate steps against. It is closer to being handed a paper and told to reproduce its computation than to being handed a ticket.

That shift punishes two different weaknesses. A model can fail because it does not actually understand the science, choosing the wrong physical relationship or the wrong discretization, or it can understand the science and still write numerically unstable or subtly incorrect code. Separating those two failure modes is why SciCode ships a variant that provides the scientific background alongside the problem. When both models jump several points with the background provided, as they do here, it tells you a meaningful share of their failures were comprehension rather than coding. We saw a related pattern on library heavy coding in the BigCodeBench comparison, where knowing the right API mattered as much as writing correct logic.

The Collapse Point

The number that predicts real behavior is not the aggregate solved rate; it is where each model starts propagating an early error through the rest of a chain. A SciCode main problem might be five or more dependent subproblems, and if a model gets subproblem two wrong, the question is whether it notices the downstream inconsistency or plows ahead computing on a broken foundation.

In our runs GPT-5.6 Sol was slightly better at self checking mid chain, occasionally catching that an intermediate result was physically implausible and revising the earlier step. Gemini 3.5 Pro more often carried an early error forward to a confidently wrong final answer. Neither does this reliably; both propagate errors more than they catch them. But the margin matters for tooling, because a model that sometimes notices its own inconsistency gives you a hook for verification, while one that always sounds certain forces you to check every step externally. Treat these as observations from a specific test rather than fixed model properties, since prompt structure moves self checking behavior substantially.

Where Each Model Wins

Domain strengths split along a recognizable line. GPT-5.6 Sol was stronger on physics and problems that reduce to a clear numerical method, differential equations, linear algebra, iterative solvers, where the win is picking the right algorithm and implementing it stably. Gemini 3.5 Pro was relatively stronger on biology and problems with more symbolic or combinatorial setup before the numerics begin, which may reflect its larger context window letting it hold more of a sprawling problem statement at once.

Both models were weakest on materials science, which tends to combine domain specific conventions with heavy numerical work, the worst of both failure modes. If your work lives in that corner, neither model is close to a hands off tool yet, and the honest recommendation is to expect to supply the algorithm and use the model for implementation rather than derivation. The same benchmark run against a different pair, Opus 4.8 and Grok 4.5, is covered in the SciCode research coding comparison, and the domain pattern holds across pairs more than the specific ranking does.

Cost Against Capability

Pricing is close enough that it should not decide this. Gemini 3.5 Pro is modestly cheaper per token and carries a far larger context window, which matters if your scientific problems come bundled with long reference material you want in context. GPT-5.6 Sol costs a little more and leads narrowly on both solved rates here. For scientific code generation specifically, where a wrong answer can be expensive to catch, the small capability edge is likely worth more than the small price difference, but the gap is not large enough to override an existing platform commitment.

When This Applies to Your Stack

If you are building tooling that turns scientific or quantitative problems into code, evals, data pipelines, simulation harnesses, SciCode is a far more honest signal than SWE-bench, and the takeaway is that both leaders still fail most full problems. Use them to implement algorithms you can specify, verify every multi step derivation externally, and do not trust either model to be right about a chain it derived unassisted. The multilingual coding picture, a separate axis again, is in the Multi-SWE-bench writeup.

If your team is integrating one of these models into an agentic workflow that touches scientific or numerical code and you want the verification layer built around where the model actually breaks, that is the AI integration work we do at Contra Collective. The models are capable; the value is in the harness that catches their confident errors.

FAQ

What does SciCode actually measure? The ability to turn a scientific problem into correct numerical code, across chains of dependent subproblems drawn from physics, chemistry, biology, and materials science. It scores both individual subproblems and whole main problems, where a main problem requires every step in its chain to be correct.

Why are the main problem scores so low for both models? Because errors compound. A main problem is only solved if every subproblem is correct, so even a high per step accuracy yields a low full problem rate once chains get long. Both models solve a minority of full problems, which is the expected shape of a compounding task.

Does the larger Gemini context window help on SciCode? It helps on problems that come with long reference material or sprawling statements, letting the model hold more context at once. It does not help with the core difficulty, which is correct derivation and implementation, not context length.

Should I pick a model on this benchmark alone? No. SciCode measures one axis, scientific code generation. If your work is repository bug fixing or library heavy coding, weight those benchmarks instead. Match the benchmark to the work your model will actually do.