SWE-Bench Verified Leaderboard: Frontier Models Tested (May 2026)

SWE-Bench Verified is the benchmark that actually correlates with shipping working code. It is a human-validated subset of 500 real GitHub issues from popular Python repositories where the test cases reliably distinguish correct fixes from incorrect ones. Unlike HumanEval, it is hard to memorize. Unlike Aider's polyglot benchmark, it covers full-issue resolution rather than diff application. If a frontier model claims coding ability and does not have a credible SWE-Bench Verified number, treat the claim with skepticism.

As of May 2026, five models sit at the top of the leaderboard, all within ten percentage points of each other. The scores matter less than the cost per resolved task, the failure modes, and which agent scaffolding each one prefers.

SWE-Bench Verified: May 2026 Leaderboard

Scores reflect single-shot agent runs on the full 500-issue Verified split, no test-time multi-sample voting, with each model's preferred scaffold. Cost figures are averaged across resolved and unresolved tasks (including failed attempts), per provider list pricing as of May 2026. Latency is wall-clock median, single run.

The headline: Claude Opus 4.7 leads, but Grok 4.3 is the unit-economics winner among hosted frontier models and Qwen 3.6 is the unit-economics winner overall if you can run it locally.

What SWE-Bench Verified Actually Measures

SWE-Bench Verified takes a real GitHub issue, gives the model the repository at the parent commit, and asks it to produce a patch that passes the issue's hidden test suite. Tasks span Django, scikit-learn, sympy, matplotlib, requests, and other widely used Python repos. The model has to:

1. Read enough of the repository to understand the change required
2. Locate the right file or files
3. Produce a minimal correct patch
4. Not break unrelated tests

This is closer to the actual experience of an engineer landing a PR in a codebase they did not write. It rewards models that can do read-heavy navigation efficiently and produce small, surgical changes. It punishes models that over-edit, that hallucinate file paths, or that fail to read enough context before writing.

The "Verified" qualifier matters. The original SWE-Bench dataset had ambiguous test cases and broken environment setups for a meaningful fraction of issues. SWE-Bench Verified is the human-validated subset where the test suite reliably distinguishes correct from incorrect fixes. Numbers on the original SWE-Bench are not comparable to numbers on Verified.

Cost per Resolved Task: The Real Comparison

Score percentage is the wrong unit. What you actually care about is cost per resolved task. That is total spend divided by number of issues solved, including failed attempts.

For a 500-issue benchmark run:

Grok 4.3 resolves about 95 percent as many issues as Claude Opus 4.7 at 45 percent of the cost. For an agentic coding loop that runs thousands of times per day across CI bots, code review automation, and dev tools, that ratio decides which API you ship.

Qwen 3.6 self-hosted is the dark horse. Self-hosting math (cost of GPU hardware amortized over expected utilization, electricity, ops time) puts the cost per resolved task in the $0.04 to $0.08 range. For high-volume internal automation where data residency matters, that is hard to beat.

Failure Modes by Model

The aggregate scores hide that models fail differently on the same problems.

Claude Opus 4.7 fails most often on tasks that require reading large portions of the repository before writing. It is the best at producing minimal correct patches when it has the right context. It is also the most likely to over-explain in its scratch reasoning, which inflates token cost without improving outcomes. Prefix caching helps but does not fully close the gap with GPT 5.5.

GPT 5.5 fails most often on tasks that require multi-file edits across module boundaries. It is the best at quickly producing a candidate patch but sometimes commits to the wrong file before exploring alternatives. Tool use is more reliable than in 5.4.

Grok 4.3 fails most often on tasks involving subtle semantics in numerical libraries (numpy, sympy, scikit-learn). It is the fastest at navigation and the cheapest per token. Its failures are usually quick failures, which keeps the cost-per-task ratio favorable even when accuracy lags.

Gemini 3.1 Pro fails most often on tasks requiring deep type inference in Python (Django ORM internals, descriptor protocol edge cases). Its strength is long-context reasoning across files; its weakness is the same thing other Google models have: occasionally confident wrong answers without hedging.

Qwen 3.6 (32B) fails most often on tasks requiring multi-step refactoring. As a self-hostable model, it punches well above its weight, but the gap to frontier hosted models on the hardest tasks is real.

Scaffolding Matters More Than the Score Difference

The top three models on the leaderboard differ by four percentage points. The difference between a model running under OpenHands versus running under a naive prompt-and-respond loop is often eight to twelve percentage points. Scaffolding is doing more of the work than benchmark coverage suggests.

OpenHands (formerly OpenDevin) is the most consistent open-source agent harness for SWE-Bench evaluations. It handles file navigation, command execution, and test running with reasonable defaults. Aider's --auto-test mode is a lighter scaffold that works well for Grok 4.3 and Gemini. Anthropic's native computer use harness is best for Opus 4.7.

If you are evaluating frontier coding models for internal use, run them through the same scaffold. Comparing Opus 4.7 under Anthropic's harness to Grok 4.3 under a custom loop tells you about scaffolds, not models.

When This Applies to Your Stack

For most product teams shipping AI-assisted coding features, the SWE-Bench Verified ranking matters less than the cost curve at your specific traffic. If you are running thousands of agentic coding turns per day (CI bots, automated PR review, internal dev tooling), the difference between $0.25 and $0.55 per resolved task compounds into a meaningful AWS bill within a quarter.

If you are running a smaller number of high-stakes coding turns (architectural changes, security-sensitive patches), Opus 4.7's quality lead is worth the premium.

If data residency or air-gapping is non-negotiable, Qwen 3.6 self-hosted is the only credible answer in this list, and it is closer to frontier than self-hosted models have ever been.

Contra Collective builds AI-augmented engineering tooling for enterprise teams, including agent harnesses, internal model routing layers, and SWE-Bench-style evaluation pipelines for model selection. If your team is choosing between frontier coding APIs and self-hosted alternatives, the answer is usually the one that maps best to your traffic pattern, not the one at the top of the leaderboard.

FAQ

Q: What is Grok 4.3's SWE-Bench Verified score? 72.4% on the 500-issue Verified split using OpenHands scaffolding, as of May 2026.

Q: Is SWE-Bench Verified harder than the original SWE-Bench? It is more reliable, not harder. Verified is the human-validated subset where test cases reliably distinguish correct fixes. Scores on Verified are not directly comparable to scores on the original full dataset.

Q: Which model has the best cost per resolved task? Among hosted models, Grok 4.3 at roughly $0.25 per resolved task. Qwen 3.6 self-hosted comes in around $0.06 if you amortize hardware properly.

Q: Does scaffolding affect SWE-Bench scores significantly? Yes, often by 8 to 12 percentage points. Comparing models across different scaffolds tells you about the scaffolds. Standardize on one (OpenHands is the safe default) before comparing model scores.

Q: Can a 32B open-source model really compete on SWE-Bench Verified? Qwen 3.6 (32B) reaches 66.3% with OpenHands. That is below frontier hosted models but above where Claude 3.5 Sonnet sat a year ago. The gap is closing.