Claude Opus 5 vs GPT-5.6 Sol vs Gemini 3.5 Pro: MLE-bench Machine Learning Engineering Agents Tested (2026)

If you want to know whether an agent can actually do machine learning engineering, editing a function to pass a unit test will not tell you. MLE-bench will. It hands the agent a real Kaggle competition, the raw data and the scoring metric and nothing else, and asks it to produce a submission that would have earned a bronze, silver, or gold medal against the humans who originally competed. That means exploring the data, engineering features, choosing a model family, training it inside a fixed compute budget, iterating on validation scores, and knowing when to stop and submit. It is the closest public benchmark to the job, and it separates these three frontier models more sharply than any code editing test does.

Model	Released	Context window	Price /1M in	Price /1M out	MLE-bench medal rate	Avg cost per competition
Claude Opus 5	Jun 2026	300K	$12.00	$60.00	42%	~$14
GPT-5.6 Sol	Jul 2026	400K	$10.00	$40.00	39%	~$9
Gemini 3.5 Pro	May 2026	1M	$4.00	$24.00	34%	~$5

The medal rate is the headline, and Opus 5 leads it, but read the cost column in the same glance, because on this benchmark the two are inseparable. These are directional figures from our own harness on a fixed slice of competitions with one attempt each, not the official leaderboard, so treat the gaps as the signal and the exact percentages as approximate.

What MLE-bench Rewards That Other Benchmarks Do Not

The reason MLE-bench ranks models differently than SWE-bench does is that it rewards a skill code benchmarks barely touch: sustained, budget aware judgment over a long horizon. A SWE-bench task has a clear target, the failing test, and a mostly local fix. An MLE-bench competition has no test to pass, only a metric to maximize, and the agent has to invent its own plan, decide which of a hundred possible improvements is worth the compute, and cut itself off before it burns the budget chasing a fourth decimal place. That is a different cognitive load. It is less about writing correct code in one shot and more about running a disciplined loop of hypothesis, experiment, measurement, and revision without losing the thread across dozens of steps.

This is why a model can be excellent at Aider style code editing and merely good at MLE-bench. We saw Opus 5 and GPT-5.6 Sol trade blows on pure code editing in the Aider Polyglot comparison, where the task is well specified and the horizon is short. MLE-bench stretches the horizon until planning quality dominates raw coding accuracy, and that stretch is exactly where the three models start to diverge in character rather than in a single number.

Claude Opus 5: Methodical and Expensive

Opus 5 wins MLE-bench the way a careful engineer wins a Kaggle competition, by not wasting moves. Its runs show the most disciplined loop of the three: a thorough data exploration pass before it commits to an approach, a sensible first model rather than an exotic one, and validation driven iteration that stops when the gains flatten instead of chasing noise. It is the model least likely to blow its compute budget on a doomed idea, and most likely to recover gracefully when an approach underperforms, backing out and trying something adjacent rather than doubling down.

That discipline is why it tops the medal rate, and its larger practical strength shows up in the hardest competitions, the ones where a naive pipeline scores nothing and only a thoughtful feature engineering step earns a medal. The cost is literal. Opus 5 is the most expensive model per token and it does not skimp on tokens, so it also posts the highest average cost per competition. When the downstream value of a correct model is high and you would rather pay for a better shot at a medal than save on tokens, that premium is defensible, and it lines up with the reasoning lead it showed on longer agentic tasks in the SWE-bench Pro comparison.

GPT-5.6 Sol: Fast and Streaky

GPT-5.6 Sol is the fastest to a first submission and the most willing to try an aggressive approach early, which makes it exciting to watch and a little less predictable. On competitions where its early bet pays off, it reaches a medal in fewer steps than Opus 5 and at lower cost, because it wastes less time exploring before it acts. On competitions where the early bet is wrong, it sometimes commits too hard, spending budget refining a fundamentally weaker pipeline instead of stepping back to reconsider, and that is where it drops the three points that separate it from Opus 5 on medal rate.

The net is a model that is genuinely competitive and meaningfully cheaper, landing within a few points of the top on medal rate while costing roughly a third less per competition. Its function calling reliability, which matters a great deal when an agent is orchestrating training runs and reading back metrics, held up well and matched what we saw in the BFCL function calling comparison. If your workload can absorb a little more variance in exchange for lower cost and faster iteration, GPT-5.6 Sol is the pragmatic default, and the gap to Opus 5 is small enough that on many competitions you would not notice it.

Gemini 3.5 Pro: The Value Pick

Gemini 3.5 Pro posts the lowest medal rate of the three and by a wide margin the lowest cost, and those two facts are the whole story. At a quarter of Opus 5's input price and a large discount to GPT-5.6 Sol, it is the value option, and its million token context window is a real asset on competitions with sprawling data dictionaries and long documentation the agent has to hold in view while it works. Its weakness is consistency over the long horizon. It is more prone to losing the plot on the longest competitions, repeating an experiment it already ran or drifting from its own plan, and that is what costs it medals rather than any deficiency in the individual steps.

The tradeoff is clean. For a high volume setting where you are running many competitions or many experiments and the aggregate cost dominates, Gemini 3.5 Pro's price lets you buy far more attempts per dollar, and more attempts can outweigh a lower per attempt success rate. For a setting where each run is precious and you want the best single shot at a medal, its lower reliability is the wrong place to economize. It is the model to reach for when volume is the strategy and to pass on when each attempt has to count.

How to Read These Numbers for a Real Build

The mistake would be to pick the top medal rate and stop. MLE-bench is a proxy, and the right way to use it is to match the profile to how you would actually deploy an ML engineering agent. If you are building an agent that occasionally tackles a high stakes modeling problem where a good result is worth real money, Opus 5's discipline and top medal rate justify its cost, because the token savings from a cheaper model do not offset a missed medal. If you are building a system that runs many modeling tasks continuously and cost scales with volume, the calculus flips toward GPT-5.6 Sol or Gemini 3.5 Pro, where more attempts per dollar can beat a higher success rate per attempt.

The one number that travels across all three is that none of them is close to solving the benchmark. A 42% medal rate means the best model here fails to medal on most competitions it attempts, so an ML engineering agent in production still needs a human in the loop for the hard cases, guardrails on compute spend, and validation of what it ships. Treat these models as capable assistants that can carry a routine modeling task a long way, not as autonomous data scientists, and size the human oversight to the medal rate rather than to the marketing.

When This Applies to Your Stack

If you are choosing a model to power an agent that does real machine learning work, benchmark averages will mislead you and MLE-bench's profile will not. Weight medal rate and long horizon discipline if each run is high value and you want the best single shot, and Opus 5 is your model despite the cost. Weight cost and iteration speed if you run many tasks and volume is the strategy, and GPT-5.6 Sol is the balanced pick while Gemini 3.5 Pro is the value floor with a context window advantage on data heavy competitions. In every case, budget for a human reviewer sized to a medal rate that tops out in the low forties, because none of these agents is autonomous yet.

If your team is building an agent whose value depends on it running a real ML or data workflow end to end, matching the model to your cost and reliability constraints and validating it against your own tasks is the model selection and agent engineering work we do at Contra Collective, where the goal is an evidence based build decision rather than a leaderboard bet.

FAQ

What does MLE-bench measure that SWE-bench does not? MLE-bench tests whether an agent can run a full machine learning engineering workflow on a real Kaggle competition: explore data, engineer features, train and tune a model, and submit a medal worthy result under a fixed compute and time budget. SWE-bench measures code editing against a failing test, which is a shorter, better specified task. MLE-bench rewards long horizon planning and budget aware judgment that code editing benchmarks barely exercise.

Which model scored highest on MLE-bench? In our harness Claude Opus 5 posted the highest medal rate at roughly 42%, ahead of GPT-5.6 Sol at about 39% and Gemini 3.5 Pro at about 34%. These are directional figures from a fixed slice of competitions with one attempt each, not the official leaderboard, so treat the gaps as the signal rather than the exact percentages.

Is Claude Opus 5 worth the higher cost on MLE-bench? It depends on how you deploy the agent. Opus 5 leads medal rate and is the most disciplined at not wasting compute, but it is also the most expensive per competition. If each run is high value and you want the best single shot at a medal, the premium is defensible. If you run many tasks and cost scales with volume, a cheaper model that buys more attempts per dollar can win overall.

Can these models replace a data scientist? Not yet. The best medal rate here is in the low forties, which means the top model fails to earn a medal on most competitions it attempts. An ML engineering agent in production still needs a human in the loop for hard cases, guardrails on compute spend, and validation of what it ships. Treat these models as capable assistants for routine modeling work, not autonomous data scientists.

Why does Gemini 3.5 Pro score lower despite a larger context window? Its million token window helps on data heavy competitions with long documentation, but its weakness is consistency over the longest horizons, where it is more likely to repeat an experiment or drift from its plan. That reliability gap, not the individual steps, is what costs it medals. Its very low price still makes it the value pick when running many attempts is the strategy.