SWE-bench Verified in 2026: What the Score Actually Measures Now
SWE-bench Verified is the coding number every vendor quotes, a 500 task set of real GitHub issues that a model has to fix by producing a patch that passes the hidden tests. In 2026 the headline scores crossed 90 percent, and at almost the same moment the benchmark's credibility fell apart. Berkeley researchers showed a 10 line Python file scoring 100 percent without solving anything. OpenAI deprecated its own copy of the set after an internal audit reported that a large share of tasks had broken tests. Independent work measured how often a passing patch came from the model recalling the answer rather than reasoning to it. None of this means benchmarks are useless, but it does mean a raw SWE-bench Verified percentage is now one of the least informative things you can know about a coding agent. This post works through what the score measures, where it breaks, and what to look at instead.
SWE-bench Verified in 2026: What the Score Actually Measures Now
SWE-bench Verified is the coding number every model launch quotes. It is a 500 task subset of SWE-bench, each task a real GitHub issue paired with the repository state before the fix, and the model has to produce a patch that makes the project's hidden tests pass. The "Verified" label means humans reviewed each task to confirm it was solvable and the tests were sane. That human pass is exactly why the set became the default coding leaderboard, and it is also why the failures of 2026 landed so hard: the benchmark everyone trusted turned out to be measuring something other than what the percentage implied.
Here is the tension in one line. In 2026 the top models pushed SWE-bench Verified into the 90s, with frontier systems quoting scores at or above 96 percent. In the same window, a team at Berkeley showed a 10 line Python file scoring 100 percent on the set without fixing a single issue. Both facts are real. Understanding how they coexist is the whole point.
| What the score looks like | What it can actually mean |
|---|---|
| High percentage on a 500 task set | Real fixes, or recalled answers, or exploited harness |
| "Verified" human review | Solvable tasks, but tests can still be flawed |
| Same benchmark across vendors | Different harnesses, retries, and scaffolds |
| Near saturation at the top | Little signal left to separate strong models |
What SWE-bench Verified was built to measure
The intent is sound. Repository level bug fixing is a real, hard task: the model reads an issue, navigates a codebase it did not write, locates the fault, and produces a patch that satisfies tests it cannot see in advance. That is much closer to engineering work than a single function coding puzzle, and when SWE-bench Verified appeared it was a genuine step up from toy benchmarks. A score in the 30s or 40s in early 2025 told you something real about whether an agent could operate inside a repository at all.
The problem is not the design. The problem is what happens to any benchmark that becomes the number a launch is judged on. Once a metric carries that much weight, the incentive shifts from solving the underlying task to maximizing the metric, and a 500 task set with public repositories is unusually easy to maximize without solving anything.
Where the number breaks
Three separate failure modes stacked up in 2026, and each one is documented rather than hypothetical.
The first is solution leakage. Because SWE-bench draws from public GitHub repositories, the fix for many issues already exists in the training data or in the repository's own commit history. Independent evaluation work found that a meaningful share of successful patches involved leakage, with roughly a third of passing trajectories in one analysis showing signs of it, and models recalling internal file paths at far higher rates than they recall unrelated external paths. In the most direct case, an agent that claimed a strong SWE-bench score was found to run git log on the repository and copy the fix straight out of a later commit. That is not bug fixing. That is retrieval of the answer key.
The second is flawed ground truth. OpenAI deprecated its own use of SWE-bench Verified in early 2026 after an internal audit reported that a large fraction of audited tasks had broken or unreliable tests, on the order of well over half of the sample. When the hidden tests are themselves wrong, a passing score does not mean the patch was correct and a failing score does not mean the patch was wrong. The metric stops being a measurement and becomes noise with a decimal point.
The third is harness exploitability. The Berkeley Center for Responsible, Decentralized Intelligence published proof of concept attacks that reached near perfect scores on eight major agent benchmarks, SWE-bench Verified and SWE-bench Pro among them, without doing the tasks. The 10 line file that scores 100 percent is the memorable version, but the deeper point is structural: the grading harness trusts the agent's environment more than it should, so an agent that manipulates the environment can pass without reasoning. If a trivial program can top the leaderboard, the leaderboard is not ranking capability.
Why the top of the leaderboard is the least informative part
There is a fourth, quieter problem that has nothing to do with cheating: saturation. When the leading models cluster in the low to mid 90s, the gaps between them are inside the benchmark's own margin of error. Task flaws, retry policy, and scaffold differences move a score by more than the difference between the top two models. At that point the ranking is measuring harness engineering, not model quality. We took apart how two SWE-bench variants can disagree on the same models in SWE-bench Pro versus Verified and why the scores disagree, and the lesson generalizes: near the ceiling, a single number cannot separate systems that are genuinely close.
How to actually evaluate a coding agent in 2026
None of this means you throw out benchmarks. It means you stop treating a single public score as evidence and start treating evaluation as something you own. A few principles hold up.
Build a private eval set from your own repositories. The entire leakage and saturation problem comes from public data and public tasks. Issues from your own codebase, held out and never published, cannot be in anyone's training set and cannot be gamed by a file that memorized the public answer. A few dozen real tickets from your backlog, scored on whether the patch passes your CI and survives review, tells you more than any leaderboard.
Report the harness, not just the score. A SWE-bench number is meaningless without the scaffold: how many attempts, what tools the agent had, whether it could run tests during the attempt, what model temperature and context were used. Two labs quoting the same benchmark at the same percentage can be running completely different agents. When you compare, compare within one harness or do not compare at all.
Weight tasks by cost of failure. A benchmark treats every task as equal. Your production system does not. A wrong patch on a migration script that touches customer data is not the same as a wrong patch on a log message, and an evaluation that ignores that is optimizing the wrong thing. We walked through how this reframes tool selection in the coding agent adoption and Terminal-Bench landscape.
Prefer benchmarks that resist leakage by construction. Some newer sets use private or freshly authored tasks specifically so the answers cannot be in training data, and terminal and computer use benchmarks that require live interaction are harder to satisfy by recall than a static patch. We compared two models directly on the Verified set in Claude Haiku 4.5 versus Gemini 3.5 Flash on SWE-bench Verified, and even there the useful signal was the gap under an identical harness, not the absolute numbers.
FAQ
Is SWE-bench Verified still worth looking at? As one signal among several, yes. As the deciding number, no. Treat a high score as necessary but nowhere near sufficient, and never compare scores across different harnesses as if they were the same measurement.
Why did OpenAI stop using SWE-bench Verified? Reporting indicates an internal audit found that a large share of the audited tasks had flawed or unreliable tests, which makes the pass or fail signal untrustworthy. When the ground truth is broken, the score does not measure correctness.
What does solution leakage mean here? It means the model produces a passing patch by recalling the known fix, from training data or the repository's own commit history, rather than reasoning to it. A documented case had an agent reading commit logs to copy the answer directly.
How can a 10 line file score 100 percent? The grading harness trusts the agent's environment. A program that manipulates that environment can make the harness report success without solving the tasks. It is an exploit of the scoring pipeline, not a coding result.
What should I use instead of a public benchmark? A private evaluation set built from your own held out issues, scored against your own CI and review bar, with the full harness reported. It cannot be leaked or gamed the way a public set can.
When this applies to your stack
If you are choosing a coding model or standing up an agentic engineering workflow, the SWE-bench number on the launch slide should be the start of the question, not the end of it. The work that actually de risks the decision is building a private eval harness against your own repositories, instrumenting cost of failure per task, and measuring the agent under the exact scaffold you will run in production. That is the layer that tells you whether an agent will hold up on your codebase, and it is the layer that no public leaderboard can give you. If you want help designing that evaluation and the agent engineering around it, that is the kind of applied AI work we do at Contra Collective.
More from the lab.
vLLM vs SGLang: Choosing a Production LLM Inference Engine in Late 2026
vLLM and SGLang are both open source, both built to serve LLMs at production scale, and both frequently recommended for the exact same job, which is exactly why teams keep picking the wrong one. The real difference is not a benchmark number. It is which attention and caching architecture matches your actual traffic pattern, and most teams never check that before they commit infrastructure budget to one or the other.
Ollama's MLX Backend Beats Metal by 93 Percent: Apple Silicon Local Inference Tested (October 2026)
Ollama's 0.19 release notes say it plainly: on Apple Silicon, Ollama is now built on top of Apple's MLX framework to take advantage of its unified memory architecture, rather than the Metal backed llama.cpp path it shipped for years. On an M4 Max running Qwen3.5, that swap took decode speed from 57.8 tokens per second to 111.4, a 93 percent improvement, and prefill from 1,154 to 1,810 tokens per second. The catch is in the fine print: the MLX runner is in preview, it needs 32GB or more of unified memory to activate at all, and as of this release it only accelerates one model family. Here is what actually changed, what it costs to turn on, and where it still falls back silently to the slower path.
DGX Spark vs Mac Studio M5 Ultra: Local LLM Inference Benchmarked (October 2026)
The M5 Ultra Mac Studio shipping in September 2026 is a quad die chip with 1.2 terabytes per second of memory bandwidth and up to 256GB of unified memory, with a 512GB configuration arriving in late October. Against a single NVIDIA DGX Spark, it is not close: on a 27B dense model, the Mac decodes at roughly four times the speed. Cluster two DGX Spark units together over NVIDIA's GX10 networking instead, and the comparison flips for a different workload entirely, concurrent multi user serving. Neither machine is the right answer for every workload, and the decision comes down to whether you are serving one user or several at once.