SWE-Bench Pro vs SWE-Bench Verified: Why the Two Scores Disagree (2026)
A model can post a 95 on SWE-Bench Verified and land near half that on SWE-Bench Pro, and both numbers are real. Verified is a 500 task human validated subset of open GitHub issues, and the frontier has largely saturated it in 2026 with scores in the mid 90s. SWE-Bench Pro, built by Scale AI, runs harder tasks across more languages and, crucially, includes private codebases that model trainers cannot legally access, so contamination is prevented by construction. The result is a benchmark that separates models Verified no longer can. If you are picking a coding model on benchmark numbers, reading these two scores as if they measure the same thing is how you end up disappointed in production. This post explains what each one measures, why the gap is structural rather than noise, and how to weight them.
SWE-Bench Pro vs SWE-Bench Verified: Why the Two Scores Disagree (2026)
A frontier model can score in the mid 90s on SWE-Bench Verified and then land close to half that on SWE-Bench Pro, and neither number is wrong. They measure different things, and in 2026 the difference is the whole point. Verified has been largely saturated by the top models, so a leaderboard where everyone clusters between 94 and 97 tells you almost nothing about which model to pick. SWE-Bench Pro, built by Scale AI, was designed to be the harder, contamination resistant test that Verified can no longer be, and it reintroduces the spread that lets you actually rank models on real world coding. If you are choosing a coding model on public benchmarks, understanding why the two disagree is the difference between a defensible choice and a surprise in production.
| Attribute | SWE-Bench Verified | SWE-Bench Pro |
|---|---|---|
| Maintainer | OpenAI curated subset | Scale AI |
| Task count | 500 human validated | roughly 1,865 |
| Repositories | open source Python | 41 repos, Python, Go, TypeScript, JavaScript |
| Private code | none, all public | includes private proprietary codebases |
| Contamination risk | high, tasks are public | structurally prevented |
| Frontier scores 2026 | mid 90s, saturated | sharply lower, wide spread |
Score ranges above describe the 2026 pattern rather than any single frozen leaderboard, because both boards update as new models ship. The shape is the stable part: Verified is compressed at the top, Pro is spread out.
What each benchmark actually measures
SWE-Bench Verified is a 500 task subset of the original SWE-Bench where human annotators confirmed that each issue is genuinely solvable and that the tests reliably judge a fix. That curation was a real improvement, because the original benchmark contained tasks that were underspecified or had broken tests. Verified became the standard coding number for good reason. The problem is success: by 2026 the frontier models resolve the mid 90s of it, so the benchmark has stopped discriminating. When Opus 5, GPT-5.6 Sol, and the Fable 5 line all sit within a couple of points near the top, the ranking is inside the noise and the benchmark has done its job and aged out.
SWE-Bench Pro was built to pick up where Verified leaves off. It runs roughly 1,865 tasks across 41 actively maintained repositories spanning Python, Go, TypeScript, and JavaScript, so it tests multi language competence rather than a single ecosystem. The design choice that matters most is the inclusion of private, proprietary codebases that have never been publicly available, alongside a public set drawn from repositories under strong copyleft licenses like the GPL. Both choices target the same failure of older benchmarks: contamination.
The contamination problem is why the gap exists
Here is the mechanism. SWE-Bench Verified is built from public GitHub issues. Those issues, their discussions, and often their merged fixes are exactly the kind of text that goes into a model's training data. A model can score well on a public benchmark partly because it has seen the answer, or something close to it, during training. Nobody has to cheat for this to happen. It is a structural property of testing models on public code they were trained on.
SWE-Bench Pro attacks that directly. The private codebases are legally inaccessible to model trainers, so those tasks cannot have leaked into training data. The public portion leans on strong copyleft licensing as a legal deterrent against inclusion in training corpora. The net effect is a benchmark that is contamination resistant by construction, and when you remove the contamination advantage, scores fall. That is why a model in the mid 90s on Verified can drop to roughly half that on Pro. The drop is not the model getting dumber. It is the benchmark measuring problem solving on unseen code instead of partly measuring recall of seen code. The Scale AI research write up and the accompanying paper lay out the construction in detail, and the leaderboard updated through early September 2026 keeps showing the same pattern: a compressed top on Verified, a wide and moving spread on Pro.
How to read them when picking a model
The practical rule is to stop treating Verified as a ranking tool and start treating it as a floor. In 2026, a frontier model clearing the mid 90s on Verified is table stakes, not a differentiator. If a model cannot clear it, that is a red flag. If it can, you have learned only that it meets the baseline. The discrimination now lives on the harder, contamination resistant tests, so weight SWE-Bench Pro and other unseen code evaluations more heavily when you are choosing between models that all pass Verified.
The second rule is to distrust any single benchmark for your specific work. Pro tests Python, Go, TypeScript, and JavaScript across 41 repos, which is broad, but it is still not your codebase, your conventions, or your test suite. The right move for a team making a real decision is to build a small internal evaluation from your own resolved issues, run the candidate models against it, and measure resolution rate and cost per resolved issue on tasks that look like your actual work. That is more effort than reading a leaderboard, and it is the only thing that predicts production behavior. We keep the public numbers current in our SWE-Bench Verified frontier leaderboard and work through a head to head on SWE-Bench Pro between Opus 5 and Gemini 3.5 Pro, but both are starting points for your own evaluation, not substitutes for it.
Cost per resolved issue is the number that survives
One more reason not to pick on raw benchmark rank: the model that resolves two more points of a benchmark can cost several times more per issue to run in an agent loop. On real work the metric that decides your bill is cost per resolved issue, benchmark score divided into what it costs to get there. A slightly lower scoring model that costs a third as much per resolved task is often the correct production choice, especially at volume. We break down that trade in our cost per issue comparison across the Opus and Sonnet line, and it is the frame we bring to every model selection: not which model is smartest, but which one clears your quality bar at the lowest cost per resolved issue.
FAQ
Which benchmark should I trust more in 2026? For discrimination between frontier models, SWE-Bench Pro, because Verified is saturated. Use Verified as a minimum bar a serious model must clear, not as a ranking.
Why do scores drop so much from Verified to Pro? Two reasons: Pro's tasks are harder and multi language, and Pro is contamination resistant by construction, so models cannot lean on having seen the answers in training. Remove that advantage and scores fall.
Is a high Verified score meaningless now? Not meaningless, just necessary rather than sufficient. Clearing the mid 90s is expected. It no longer tells you which of several strong models is best for your work.
Does SWE-Bench Pro test my language? It covers Python, Go, TypeScript, and JavaScript across 41 repositories. That is broad, but it still is not your codebase, so pair it with an internal evaluation.
What number should actually drive the decision? Cost per resolved issue on tasks that resemble your own, measured on a small internal benchmark. Public boards narrow the field; your own evaluation makes the call.
When this applies to your stack
If you are selecting a coding model or an agent for real engineering work, the benchmark literacy here is the point: read Verified as a floor, weight contamination resistant tests like SWE-Bench Pro for real ranking, and then build a small evaluation from your own resolved issues to measure resolution rate and cost per resolved issue on work that looks like yours. Teams that skip that last step tend to buy the top of a saturated leaderboard and pay frontier prices for a gap that disappears on their actual codebase.
Standing up that evaluation harness, the internal benchmark, the cost per issue measurement, and the model routing that follows from it, is work we do for engineering organizations choosing where to spend their AI budget. If you want an evaluation pipeline that tells you which model actually earns its cost on your code, Contra Collective can build it so your model choice rests on your numbers rather than a public board.
More from the lab.
vLLM vs SGLang: Choosing a Production LLM Inference Engine in Late 2026
vLLM and SGLang are both open source, both built to serve LLMs at production scale, and both frequently recommended for the exact same job, which is exactly why teams keep picking the wrong one. The real difference is not a benchmark number. It is which attention and caching architecture matches your actual traffic pattern, and most teams never check that before they commit infrastructure budget to one or the other.
Ollama's MLX Backend Beats Metal by 93 Percent: Apple Silicon Local Inference Tested (October 2026)
Ollama's 0.19 release notes say it plainly: on Apple Silicon, Ollama is now built on top of Apple's MLX framework to take advantage of its unified memory architecture, rather than the Metal backed llama.cpp path it shipped for years. On an M4 Max running Qwen3.5, that swap took decode speed from 57.8 tokens per second to 111.4, a 93 percent improvement, and prefill from 1,154 to 1,810 tokens per second. The catch is in the fine print: the MLX runner is in preview, it needs 32GB or more of unified memory to activate at all, and as of this release it only accelerates one model family. Here is what actually changed, what it costs to turn on, and where it still falls back silently to the slower path.
DGX Spark vs Mac Studio M5 Ultra: Local LLM Inference Benchmarked (October 2026)
The M5 Ultra Mac Studio shipping in September 2026 is a quad die chip with 1.2 terabytes per second of memory bandwidth and up to 256GB of unified memory, with a 512GB configuration arriving in late October. Against a single NVIDIA DGX Spark, it is not close: on a 27B dense model, the Mac decodes at roughly four times the speed. Cluster two DGX Spark units together over NVIDIA's GX10 networking instead, and the comparison flips for a different workload entirely, concurrent multi user serving. Neither machine is the right answer for every workload, and the decision comes down to whether you are serving one user or several at once.