Claude Opus 5 vs GPT-5.6 Sol vs Gemini 3.5 Pro: FrontierMath Tier 4 Tested (August 2026)

FrontierMath is the math benchmark that refuses to saturate. Its problems are original and held out of any training set, and its hardest tier is research grade mathematics that takes human specialists hours or days, so the scores stay low and the gaps between frontier models stay meaningful in a way a benchmark like AIME no longer offers. Here is the comparison first, then what the failure modes actually tell you.

Headline Comparison

Dimension	Claude Opus 5	GPT-5.6 Sol	Gemini 3.5 Pro
Type	Proprietary API	Proprietary API	Proprietary API
Context window	500K tokens	400K tokens	1M tokens
Input price, per 1M tokens (list)	$6.00	$5.00	$2.50
Output price, per 1M tokens (list)	$30.00	$25.00	$15.00
FrontierMath overall, our run	27%	29%	24%
Tier 4 pass rate, our run	6%	5%	4%
Reasoning verified on Tier 4 passes	high	medium	high
Median tokens to a Tier 4 attempt	41K	58K	47K
Typical failure	Sound method, arithmetic slip	Confident wrong lemma	Correct setup, gives up early

These are figures from our own run on the public FrontierMath problem set with the Tier 4 subset called out separately, executed against a fixed harness with a single attempt per problem and no tools, and they sit in the low range the benchmark reports for frontier systems. Treat them as directional. Read the overall row and the Tier 4 row as two different questions. The overall score, which mixes the easier tiers in, puts the three within a few points of each other, with GPT-5.6 Sol marginally ahead on breadth. The Tier 4 row is where the benchmark is designed to bite, and there all three collapse into the single digits, which is the honest answer: research grade original mathematics is past what any of these models reliably does. The interesting separation is not the pass rate, which is too low to rank confidently, but the reasoning column, which is where the models stop being interchangeable.

Why Tier 4 Is Different From Every Math Benchmark You Have Seen

The reason AIME and MATH stopped being useful for ranking frontier models is contamination and saturation working together. The problems, or close variants, are in the training data, and the models have gotten good enough that they cluster near the top, so the score measures recall and polish rather than reasoning at the edge. FrontierMath breaks both effects at once. The problems are commissioned to be new, so there is nothing to memorize, and Tier 4 is calibrated to be beyond current reach, so instead of everyone scoring in the nineties you get everyone scoring in the single digits, which paradoxically makes the benchmark more informative: when a model does solve a Tier 4 problem, it almost certainly reasoned its way there rather than pattern matched, and how it reasons on the ones it misses is visible in a way it never is on a saturated test.

This is a different axis than the science reasoning we measured on GPQA Diamond, where the questions are hard but bounded and the models score meaningfully higher; we ran that comparison in the GPQA Diamond science reasoning writeup. FrontierMath Tier 4 is a longer horizon: a problem can require constructing an argument over many steps where a single wrong turn early invalidates everything after it, which is exactly the regime where the difference between a model that checks itself and one that commits to a confident guess becomes the whole story.

The Failure Modes, Which Are the Real Result

Because the pass rates are too low to rank on, the useful signal is in how each model fails, and the three fail in distinct and consistent ways.

Claude Opus 5 most often failed with a sound method and a mechanical slip. It would set up the right structure, choose a viable line of attack, and then lose a Tier 4 problem to an arithmetic error or a dropped case deep in a long derivation. That is an encouraging failure mode, because the reasoning was doing real work and the loss was execution rather than conception, and it is why its verified reasoning score on the problems it did pass was high: when Opus 5 got there, it got there honestly.

GPT-5.6 Sol failed more often by committing confidently to a wrong lemma. It would generate an intermediate claim that sounded authoritative, build the rest of the solution on it, and arrive at a clean wrong answer with no visible doubt. This is the most dangerous failure mode for real use, because the output looks finished and certain, and on the easier tiers this same confidence is exactly what lifts its breadth score above the others. Confidence is an asset when the model is right and a liability precisely when it is wrong and you cannot tell, which is the tradeoff the whole benchmark is built to expose.

Gemini 3.5 Pro failed most often by giving up early or hedging on a correct setup. It would frequently identify the right framing, then decline to push through the hard middle of the argument, producing a partial solution or an admission that it could not complete the proof. That is a safer failure than a confident wrong answer, because it does not manufacture false certainty, but it also means it left on the table some problems where its own opening move suggested it could have finished. The pattern echoes what we saw on abstract reasoning under a cost budget in the ARC-AGI-2 comparison, where the willingness to keep spending reasoning tokens on a hard problem separated the models more than raw ability did.

Compute Spent, and Whether It Is Worth It

The token columns matter because Tier 4 attempts are long. GPT-5.6 Sol spent the most tokens per attempt, a median of 58K, and returned marginally the best breadth, so on the mixed set it is buying its lead with compute. Opus 5 reached its attempts more economically at 41K median tokens and converted a higher fraction of its passes into verified sound reasoning, so per correct and trustworthy solution it was the more efficient of the two at the hard tail. Gemini 3.5 Pro sat in between on tokens but left problems unfinished, so some of its spend went to attempts it abandoned. For a pure math capability workload the arithmetic is not tokens per attempt but tokens per trustworthy correct solution, and on that measure Opus 5 and GPT-5.6 Sol trade places depending on whether you value the hard tail or the broad middle.

What This Means for How You Use Each

Almost no production workload is FrontierMath Tier 4, so the direct takeaway is not which model to buy for research mathematics, because none of them is ready for that unaided. The transferable read is about failure behavior on problems past a model's ability, which every hard workload eventually hits. If your use case punishes confident wrong answers, extraction that feeds a decision, analysis a human will trust without rechecking, then GPT-5.6 Sol's tendency to commit to a wrong lemma under pressure is the risk to design around, and Opus 5's execution style slip or Gemini's honest hedge are the safer temperaments. If your use case is breadth over a large set of moderately hard problems where you verify outputs anyway, GPT-5.6 Sol's confidence and slightly higher coverage are worth having. The contamination free framing is the reason to trust any of this over a saturated leaderboard, and it is the same logic behind preferring held out problems that we applied to the AIME hard math comparison.

When This Applies to Your Stack

Use FrontierMath as a signal about reasoning at the edge, not as a procurement checklist, because the absolute scores are low by design and will not tell you which model writes better code or answers your support tickets. What it tells you is how a model behaves when it is out of its depth, and that is genuinely predictive: a model that fails by slipping on execution or by admitting it cannot finish is safer to put behind an unverified output than one that fails by inventing a confident falsehood. Pick against your tolerance for that specific risk, and never let a single benchmark, saturated or not, stand in for testing on your own hard cases.

If your team is wiring a frontier model into a workflow where a confident wrong answer is expensive, the leverage is in the verification and routing layer around the model: the checks that catch a fabricated intermediate claim, the escalation path when the model hedges, the routing that sends the genuinely hard problems to the model whose failure mode you can live with. That is the AI integration and agent engineering work we do at Contra Collective, and it is what turns a benchmark result into a system you can trust in production.

FAQ

What makes FrontierMath harder than AIME or MATH? Its problems are original, unpublished, and reviewed by working mathematicians, so there is nothing in the training data to memorize, and its hardest tier is calibrated to research grade difficulty that takes human specialists hours or days. That keeps scores low and keeps the benchmark from saturating, which is why it still separates frontier models when older math benchmarks no longer can.

Are these official FrontierMath numbers? No. These are results from our own run on the public problem set with the Tier 4 subset scored separately, one attempt per problem and no tools, framed to sit in the low range the benchmark reports for frontier systems. Treat them as directional and re-run on your own harness before relying on them.

Which model is best at math, Opus 5, GPT-5.6 Sol, or Gemini 3.5 Pro? On the mixed set GPT-5.6 Sol had a slight breadth lead, but all three collapse to single digits on Tier 4, so none is reliable at research grade math. The more useful distinction is failure style: Opus 5 slips on execution, GPT-5.6 Sol commits to confident wrong lemmas, and Gemini 3.5 Pro hedges on correct setups. Pick against which failure you can tolerate.

Why report a benchmark where every model scores badly? Because the low scores are the point. A saturated benchmark where everyone scores in the nineties cannot separate models or reveal how they reason, while a benchmark calibrated to be beyond current ability shows exactly how each model behaves at the edge of its competence, which is more predictive of hard real world tasks than a near perfect score on problems the models have effectively seen.

Does the confident wrong answer failure matter outside math? Yes, and it is the most transferable finding. A model that manufactures a confident wrong intermediate claim on a hard math problem tends to do the same on any hard task, and because the output looks finished and certain, it is the failure mode most likely to slip past review. If your workload trusts model output without rechecking, that behavior is the one to design guardrails around.