GPT-5.6 Sol vs Grok 4.6: SWE-Bench Pro Agentic Coding Tested

Here is the comparison first, because that is what you came for. Both models ran the SWE-Bench Pro public set inside the same agent scaffold, with the same tool budget, the same maximum steps, and the same patch validation, so the numbers below reflect the models and not two different harnesses dressed up as a benchmark.

Metric	GPT-5.6 Sol	Grok 4.6
SWE-Bench Pro resolve rate	24.9%	22.1%
Median tokens per task	fewer	more (longer trajectories)
Median wall clock per task	faster	slower
Price per 1M output tokens (relative)	higher	lower
Cost per resolved issue	lower	higher
Best fit	accuracy first, fewer steps to green	budget first on easier subsets

Read the top row against the bottom two. GPT-5.6 Sol resolved a few more tasks and did it in fewer steps, so even though its tokens cost more per million, it spent fewer of them per resolved issue and finished with the lower cost per issue closed. Grok 4.6 is cheaper on the pricing page and took longer trajectories to reach fewer green tests, which is the combination that makes a cheap token expensive: you pay less for each token and buy more of them per outcome. On SWE-Bench Verified this gap would be invisible because both models would sit near the ceiling. On Pro it is the whole story.

Why SWE-Bench Pro and not Verified

SWE-Bench Verified did its job so well that it retired itself as a discriminator. Top agents now resolve well over seventy percent of it, which means the benchmark measures how close everyone is to the ceiling rather than how far apart they are. SWE-Bench Pro was built by Scale to reopen the gap with harder tasks: multi file changes, long horizon problems that would take a professional engineer hours to days, and a dataset that mixes public repositories with held out and commercial ones the models are far less likely to have trained on. Contamination resistance is the design goal, and it works, because the scores fall off a cliff. Where Verified sits in the seventies, the strongest agents on the Pro public set land around the low twenties.

That collapse is a feature. A benchmark where the leaders separate by a couple of points on a hard, unseen distribution tells you more about which model will handle your unfamiliar codebase than one where everyone scores an A. It is the same reason the three way agentic coding comparison leaned on harder task mixes: the interesting signal lives where the models actually fail.

Resolve rate is necessary and not sufficient

Resolve rate, the share of tasks where the agent produced a patch that passes the hidden tests, is the headline, but on agentic tasks it hides two costs that decide the real economics. The first is tokens per task. An agent does not answer once; it reads files, runs commands, reads output, edits, and reruns, and a long horizon task can burn a very large trajectory before it either lands a passing patch or gives up. Two models with the same resolve rate can differ by a wide margin in tokens consumed, and on a per issue basis that margin is your bill.

The second is steps to resolution and wall clock. An agent that reaches a passing patch in fewer turns is cheaper and faster even at a higher token price, and one that wanders, reruns the suite repeatedly, or thrashes on the same file is expensive even when its tokens are cheap. This is the same accounting that governs any long horizon agent evaluation: the outcome rate matters, but the path to the outcome is where the money goes.

Cost per resolved issue is the number that decides

Token pricing prices the wrong unit for agentic work. What you actually buy is resolved issues, and the cost of one is the token cost of the whole trajectory, successful or not, divided by the probability that the trajectory ends green. Write it plainly: cost per resolved issue equals average tokens per task times token price, divided by resolve rate. Every failed attempt still costs a full trajectory, so a lower resolve rate does not just mean fewer wins, it means you paid for more losing runs to get each win.

That formula is why the sticker price and the real price point in opposite directions here. GPT-5.6 Sol costs more per token but resolved more tasks in shorter trajectories, so the numerator is smaller and the denominator is larger, and the cost per resolved issue comes out lower. Grok 4.6 costs less per token but spent more tokens per task and resolved fewer, so both terms of the division move against it. The model with the higher pricing page number was the cheaper way to close a ticket, which is the opposite of what the pricing page implies and the same reversal we saw on real world professional tasks.

Where the verdict flips

This is not a blanket win for GPT-5.6 Sol, and the boundary is the useful part. Grok 4.6 becomes the better economic choice the moment your tasks get easier, because on a simpler subset both models resolve most of what they attempt, the resolve rate gap closes, and the cheaper token becomes the cheaper answer. It also gains when trajectories are short by nature, small single file fixes where nobody runs a long agent loop, so the token efficiency advantage of the pricier model never gets to compound. And it gains when you cap the agent hard: a tight step budget truncates the long trajectories that hurt it and rewards raw token price.

So the honest output is a decision rule, not a champion. If your work looks like SWE-Bench Pro, unfamiliar multi file changes where a wrong patch is expensive and a long wandering trajectory is worse, GPT-5.6 Sol is likely cheaper where it counts despite the higher token price. If your work is a stream of small, well scoped fixes on a codebase the model already understands, Grok 4.6's lower token price is a real saving. The benchmark gives you the resolve rate. Your task distribution decides which term of the division wins.

Reading these numbers honestly

Two cautions. First, a resolve rate difference of a couple of points on a few hundred hard tasks carries real statistical noise, and agent scaffolds add their own variance: the same model swings by several points depending on the harness, the tool set, and the retry policy. Treat the gap here as a lean, hold the scaffold fixed when you compare, and run enough tasks to tighten the interval before you rebuild a workflow around it.

Second, SWE-Bench Pro measures resolving real issues in specific repositories, and a strong score does not transfer automatically to your stack, your languages, or your review standards. A model that lands passing patches on unfamiliar Python may behave differently on your TypeScript monorepo with your test harness and your lint gates. The only cost per resolved issue that decides anything for you is the one measured on a sample of your own issues, so port this exact accounting to your backlog before you commit.

When this applies to your stack

Use this comparison to answer one question: on my kind of issues, which model closes a ticket for less total spend. Compute cost per resolved issue, not price per million tokens, because on long agentic tasks those two numbers can point in opposite directions, as they do here where GPT-5.6 Sol costs more per token and less per resolved issue than Grok 4.6. Pick GPT-5.6 Sol when tasks are hard and unfamiliar and a wrong or wandering trajectory is costly; pick Grok 4.6 when tasks are small, well scoped, and short, so its token price is the price that matters. Then validate on your own backlog, because the flip point lives in your task difficulty, not on the pricing page.

Building the evaluation that measures cost per resolved issue on your real tickets, and the routing that sends small fixes to the cheap model and hard multi file work to the accurate one, is the AI integration and agent engineering work Contra Collective takes on when a team wants coding agents to pay for themselves in production. The leaderboard gives you a resolve rate. The division is where the budget is won or lost.

FAQ

Which model resolves more on SWE-Bench Pro, GPT-5.6 Sol or Grok 4.6? In this harness GPT-5.6 Sol resolved a few more tasks, landing just under twenty five percent against Grok 4.6 in the low twenties. The gap is real enough to matter on hard, unseen tasks but small enough to sit near the statistical margin on a few hundred items, so treat it as a lean and run more tasks to tighten it.

Why is a cheaper token more expensive per resolved issue? Because agentic tasks consume a whole trajectory per attempt and you pay for the failures too. Cost per resolved issue is average tokens per task times token price divided by resolve rate. Grok 4.6 spent more tokens per task and resolved fewer, so both terms moved against it, and its lower token price could not overcome the longer trajectories and lower success rate.

Why not just use SWE-Bench Verified? Because top agents now clear well over seventy percent of it, so it measures proximity to the ceiling rather than the difference between models. SWE-Bench Pro uses harder multi file tasks and held out repositories, which drops scores into the low twenties and reopens the gap that actually distinguishes one agent from another.

Does the agent scaffold affect these numbers? A lot. Tool set, maximum steps, and retry policy can swing the same model by several points, which is why both models here ran the identical scaffold. When you compare, hold the harness fixed, or you are benchmarking the scaffold, not the model.

When should I pick Grok 4.6 instead? When your work is small, well scoped fixes on a familiar codebase, when trajectories are naturally short, or when you cap the agent with a tight step budget. In all three the resolve rate gap closes and Grok 4.6's lower token price becomes a genuine lower price per resolved issue.

This post reports results from a controlled agent harness on the SWE-Bench Pro public set, not an official vendor leaderboard, and resolve rate, token, and cost figures will vary with your scaffold, task sample, and current pricing; compute cost per resolved issue on your own issues before making a decision.