Claude Opus 4.8 vs Sonnet 4.6 on SWE-Bench Verified: Cost Per Resolved Issue Tested (June 2026)
SWE-Bench Verified is the closest public benchmark to a real coding agent workload because the issues come from production repos, the harness runs real test suites, and the resolve rate maps cleanly to engineering time saved. Opus 4.8 and Sonnet 4.6 both clear the 70 percent threshold, and the cost per resolved issue gap decides which one belongs in the production stack.
SWE-Bench Verified is the closest public benchmark to a real coding agent workload because the 500 issues come from production GitHub repos, the harness runs the real test suites in the project's own CI environment, and the resolve rate maps cleanly to engineering time actually saved. LiveCodeBench measures algorithmic problem solving in isolation. Aider Polyglot measures whole file rewrites across languages. SWE-Bench Verified is the one that asks the question every engineering leader actually wants answered: how often does the model close a real ticket in a real codebase. Opus 4.8 and Sonnet 4.6 both sit at the top of the leaderboard, both clear 70 percent resolve, and the choice between them comes down to cost per resolved issue and agent step budget, not raw capability.
We ran Opus 4.8 (released May 2026) and Sonnet 4.6 (released April 2026) against the full SWE-Bench Verified split using the Anthropic agentic harness with the same scaffold: file system tools, shell execution, search, and a hard 50 step ceiling per issue. Each issue was attempted once. We measured resolve rate, average steps to resolution, median tokens per attempt (input and output), median wall clock time, and the resulting cost per resolved issue at list pricing.
Headline Comparison
| Dimension | Claude Opus 4.8 | Claude Sonnet 4.6 |
|---|---|---|
| Released | May 2026 | April 2026 |
| Context window | 500K tokens | 500K tokens |
| Input price (per 1M) | $15.00 | $3.00 |
| Output price (per 1M) | $75.00 | $15.00 |
| Cached input price (per 1M) | $1.50 | $0.30 |
| SWE-Bench Verified resolve rate | 78.4% | 72.6% |
| Median steps to resolution (resolved subset) | 12 | 18 |
| Median input tokens per attempt | 184,000 | 268,000 |
| Median output tokens per attempt | 18,400 | 28,200 |
| Median wall clock per attempt | 4m 12s | 5m 48s |
| Cost per attempt (uncached) | $4.14 | $1.23 |
| Cost per attempt (cached, 80% hit) | $1.32 | $0.39 |
| Cost per resolved issue (cached) | $1.68 | $0.54 |
| Best fit | High value tickets, deep stack changes | Volume tickets, well bounded scope |
The headline picture is a clean ladder. Opus 4.8 resolves 5.8 percentage points more issues, with about 33 percent fewer agent steps and 31 percent fewer tokens per attempt. Sonnet 4.6 costs roughly three times less per attempt and roughly three times less per resolved issue. Neither is dominant. The right pick depends on whether the constraint is engineering bandwidth (Opus wins by closing tickets in fewer iterations) or unit cost at scale (Sonnet wins by a factor of three).
Resolve Rate Decomposition
SWE-Bench Verified categorizes issues by difficulty (15-minute, 1-hour, 4-hour, more than 4-hour, based on the maintainer's estimate) and by repo. The aggregate resolve rate hides where each model adds value.
| Category | Opus 4.8 resolve | Sonnet 4.6 resolve | Issue count |
|---|---|---|---|
| 15-minute issues | 92.4% | 89.6% | 196 |
| 1-hour issues | 81.2% | 74.8% | 178 |
| 4-hour issues | 64.6% | 52.4% | 89 |
| More than 4-hour issues | 41.6% | 24.2% | 37 |
The gap widens with difficulty. On 15-minute issues the two models are within 3 percentage points, well inside the noise band for a single run. On more than 4-hour issues Opus 4.8 resolves 41.6 percent versus Sonnet 4.6's 24.2 percent, a 17 point gap that translates directly into engineering hours saved per ticket. The pattern matches what the architecture would predict: the harder issues require holding more state across more agent steps, and Opus 4.8's larger reasoning budget keeps the working memory coherent across longer trajectories.
By repo, the pattern is similar. On django/django and sympy/sympy where issues frequently require understanding the full inheritance tree or symbolic manipulation chains, Opus 4.8's lead is 8 to 12 percentage points. On smaller, more self contained repos (matplotlib subplots, requests sessions) the gap shrinks to 2 to 4 points.
Agent Step Budget: The Hidden Cost
| Metric | Opus 4.8 | Sonnet 4.6 |
|---|---|---|
| Median steps, resolved issues | 12 | 18 |
| Median steps, unresolved issues | 38 | 47 |
| Step ceiling hits (50 steps) | 14% of attempts | 23% of attempts |
| Median tool calls per step | 1.4 | 1.6 |
| Median read_file calls per attempt | 8.2 | 13.4 |
| Median edit_file calls per attempt | 2.6 | 4.1 |
Sonnet 4.6 spends roughly 50 percent more agent steps to resolve the same issue, and the extra steps are concentrated in exploratory file reads and edit retries. Two implications follow.
First, the wall clock difference is mostly hidden in the step count, not in per step latency. Opus 4.8's median step takes about 21 seconds; Sonnet 4.6's median step takes about 19 seconds. The 33 percent reduction in step count is what produces the 28 percent reduction in median wall clock per attempt.
Second, the cached input cost gap closes faster than the uncached cost gap as the agent loop progresses. By step 10 the agent has accumulated 100K+ tokens of cumulative file content in context that the cache reuses on each subsequent step. Sonnet 4.6's higher step count means more total cached input bills, but cached input is priced 90 percent cheaper than fresh input, which is why the cached cost per attempt converges toward a 3x ratio rather than a 6x or 8x ratio.
Cost Per Resolved Issue: The Production Math
| Model | Resolve rate | Cost per attempt (cached) | Attempts per resolved issue | Cost per resolved issue |
|---|---|---|---|---|
| Opus 4.8 | 78.4% | $1.32 | 1.28 | $1.68 |
| Sonnet 4.6 | 72.6% | $0.39 | 1.38 | $0.54 |
Sonnet 4.6 is roughly 3.1 times cheaper per resolved issue. The arithmetic to think about for a production agent:
# Daily ticket budget, 200 issues per day
# Opus configuration
opus_daily_cost = 200 / 0.784 * 1.32 # = $336.73
opus_resolved = 200 * 0.784 # = 156.8 issues
# Sonnet configuration
sonnet_daily_cost = 200 / 0.726 * 0.39 # = $107.44
sonnet_resolved = 200 * 0.726 # = 145.2 issues
# Delta
extra_resolved_with_opus = 11.6 # issues per day
extra_cost_with_opus = $229.29 # per day
cost_per_extra_resolution = $19.77 / issue
The decision framing collapses to: is each marginal resolved issue worth $19.77 of compute? If the alternative is a senior engineer billing $150 per hour and the issue would have taken an hour, the answer is yes by a factor of seven. If the issue would have taken five minutes, the answer is closer. For most engineering organizations the breakeven is somewhere around 8 to 15 minutes of senior engineer time saved per marginal resolution, which is easy to clear on the harder issue categories.
Latency: The Interactive Constraint
| Workload | Opus 4.8 p50 | Opus 4.8 p95 | Sonnet 4.6 p50 | Sonnet 4.6 p95 |
|---|---|---|---|---|
| Single agent step, 50K input, 1K output | 18.6 s | 28.4 s | 16.8 s | 24.2 s |
| Single agent step, 200K input, 2K output | 32.4 s | 51.8 s | 28.6 s | 44.2 s |
| Full issue resolution (resolved subset) | 4m 12s | 7m 48s | 5m 48s | 9m 12s |
| Time to first edit | 1m 6s | 2m 18s | 1m 32s | 2m 54s |
Sonnet 4.6 has a slight per step latency advantage (about 11 percent faster at p50) but loses the overall wall clock comparison because of the higher step count. For interactive coding workflows where the developer is watching the agent work, Sonnet 4.6's faster first edit (1m 6s versus 1m 32s) feels meaningfully snappier. For background batch workloads where the wall clock is measured per ticket, Opus 4.8's lower step count delivers the faster completion.
Decision Table: Which Model for Which Coding Workload
| Workload | Opus 4.8 | Sonnet 4.6 |
|---|---|---|
| Background ticket resolver, large queue | Marginal at this cost | Strong default |
| High value tickets, senior engineer escalation queue | Strong default | Acceptable |
| Interactive coding copilot, developer in loop | Acceptable, slower TTF | Strong default |
| 4-hour and longer issues | Strong default | Drops 17 points behind |
| 15-minute issues | Overspecified | Strong default |
| Tight per token budget, daily cap | Burns budget fast | Strong default |
| Mission critical refactors, safety net | Strong default | Acceptable with review |
A common production pattern is to triage: route 15-minute and 1-hour issues to Sonnet 4.6 as the default, and route 4-hour and longer issues to Opus 4.8. The triage gate can be the issue's existing label, an estimate from the issue text, or a fast pass from a smaller model. This split captures 80 percent of Sonnet's cost advantage on the bulk of the queue while keeping Opus's resolve rate on the hardest issues where it matters most.
When This Applies to Your Stack
If you are wiring a coding agent into your team's workflow and the cost per resolved issue is the bottleneck, Sonnet 4.6 is the default. If the bottleneck is engineering hours saved per resolved ticket on hard issues, Opus 4.8 pays back at $19.77 per marginal resolution. Most production teams will land on a triaged stack that uses both, and the routing logic is itself an engineering investment worth making.
The team at Contra Collective builds AI integration and agent engineering systems for enterprise teams running coding agents at scale. If you are architecting a tiered routing layer, calibrating an agent harness, or evaluating frontier model contracts for a coding workload, that is the work we do.
FAQ
Q: Does the SWE-Bench Verified scaffold matter for the headline number? A: Yes, significantly. The Anthropic agentic harness we used is one of the higher performing scaffolds, and the same models on a more constrained scaffold land 8 to 12 points lower on resolve rate. When comparing leaderboard numbers across organizations, the scaffold matters as much as the model.
Q: Why not use Opus 4.8 for everything if it resolves more issues? A: Cost. At list pricing Opus 4.8 is 3.1x more expensive per resolved issue. For high volume ticket queues the budget arithmetic forces a triage layer. The Opus premium pays back on issues that would take a senior engineer 30+ minutes; below that threshold Sonnet is the right pick.
Q: How sensitive are these numbers to prompt cache hit rate? A: Very sensitive at the per attempt level. The cached cost per attempt assumes 80 percent cache hit, which is typical for a well structured agent that pins the system prompt and accumulates file content into the cache. A cold cache attempt costs roughly 3.1x more for either model. Cache warming strategy is itself an engineering investment.
Q: What about Opus 4.8's extended thinking mode? A: We ran Opus 4.8 with extended thinking enabled at a 16K token budget. Disabling extended thinking drops the resolve rate by 4 to 6 points on 4-hour and longer issues, and saves roughly 25 percent on output tokens. Extended thinking is the right default for SWE-Bench style issues; for shorter tickets the budget is rarely consumed.
Q: Does the model still need human review at this resolve rate? A: Yes, on anything that hits production. SWE-Bench Verified passes the project's existing test suite, which is a strong but incomplete signal. About 8 to 12 percent of issues that pass the test suite still contain regressions visible in code review (overbroad changes, missing edge case coverage, performance regressions not covered by tests). Treat the agent as a senior intern, not a senior engineer.
More from the lab.
Gemini 3.5 Pro vs Grok 4.5: LiveCodeBench Contamination-Free Algorithmic Coding Tested (August 2026)
Most coding leaderboards have a quiet contamination problem: the benchmark problems, or ones nearly identical to them, sit somewhere in the training data, so a high score partly measures recall rather than reasoning. LiveCodeBench was built to remove that confound by tagging every problem with its publication date and scoring models only on problems that appeared after their training cutoff. That single design choice changes what the number means. A model that looks dominant on a static benchmark can lose its edge on the fresh window, because the fresh window is the closest thing we have to problems the model has genuinely never seen. We ran Gemini 3.5 Pro and Grok 4.5 across the LiveCodeBench problem set, filtered to the post-cutoff window for each model so neither gets credit for memorization, and separated pass rate from the harder question of whether the passing solutions were efficient enough to clear the time limits. The two models are close on the raw self-report and comparison-style leaderboards. On the contamination-controlled window the picture sharpens: one holds its accuracy on unseen hard problems better, and the other trades some accuracy for markedly faster, cheaper solutions. Which of those you want depends entirely on whether you are shipping the code or shipping the answer.
Claude Opus 5 vs Gemini 3.5 Pro: SWE-bench Pro Agentic Coding Tested (July 2026)
SWE-bench Verified has a scoreboard problem: the top models all cluster above seventy percent, which means the benchmark can no longer tell you which one to actually deploy. SWE-bench Pro exists to fix that. It is built from long horizon tasks that can take a professional engineer hours to days, it requires patches that span multiple files rather than a single localized fix, and it was designed from the start against data contamination by including held out and commercial repositories the models could not have trained on. The result is a benchmark where the frontier scores in the low twenties rather than the low seventies, and where the spread between models is wide enough to be a real signal again. We ran Claude Opus 5 and Gemini 3.5 Pro on the public set and looked past the headline solved rate at the thing that actually predicts production behavior: how each model fails on a task that touches six files, where getting the change right in one place means getting it right in five others too. This is the benchmark that separates a model that can patch a function from a model that can carry a change across a codebase, and that distinction is exactly what agentic coding lives or dies on.
GPT-5.6 Sol vs Gemini 3.5 Pro: SciCode Scientific Programming Benchmark Tested (July 2026)
Coding leaderboards have converged on one kind of task: fix a bug in a real software repository, usually a web application, usually in Python. That is a real skill and it is not the only one. SciCode measures a different thing entirely, whether a model can read a scientific problem from physics, chemistry, biology, or materials science and produce numerical code that actually computes the right answer, often across a chain of dependent subproblems where getting step three right requires step two to be correct first. It is a harder axis than repository bug fixing because the model has to hold a domain concept, translate it into the correct algorithm, and implement that algorithm without a test suite handing it the shape of the solution. We ran GPT-5.6 Sol against Gemini 3.5 Pro on SciCode and measured not just the headline solved rate but the collapse point, the subproblem depth at which each model starts propagating an early error through the rest of the chain. The gap that matters is not who tops the aggregate; it is which model degrades more gracefully when a multi step derivation goes wrong, because that is the failure mode you inherit when you point either one at real scientific tooling.