Claude Code vs Codex: Terminal-Bench 4.0 Results Tested (September 2026)
Terminal-Bench 4.0 is a 66 task CLI benchmark spanning software, machine learning, science, hardware, security, media, and operations work, each task run five independent times with an eight hour agent timeout. In September 2026 the top two rows are Codex paired with GPT-6 Astra at 58.2 percent and Claude Code paired with Claude Fable 5.1 at 57.9 percent, a gap small enough to sit inside run to run variance. But the leaderboard also has Claude Code paired with Claude Opus 5 at 52.4 percent, six points below the Fable 5.1 pairing on the exact same agent. The model matters as much as the harness, and this post works through what that pairing effect means for how you should actually choose a coding agent stack.
Claude Code vs Codex: Terminal-Bench 4.0 Results Tested (September 2026)
Terminal-Bench 4.0 is the benchmark built to answer a question SWE-bench never could: can an agent actually get work done inside a terminal, across a task set wider than repository bug fixing. It covers 66 tasks spanning software engineering, machine learning, science, hardware, security, media, and general operations, and each task runs five full independent trials under an eight hour agent timeout, with the leaderboard reporting run to run variance rather than a single lucky number. In September 2026, the top of that leaderboard is a near tie: Codex paired with GPT-6 Astra at 58.2 percent, Claude Code paired with Claude Fable 5.1 close behind at 57.9 percent. The gap is inside the benchmark's own noise. What is not noise is a second row further down: Claude Code paired with Claude Opus 5 scores 52.4 percent, six full points under the same harness running a different model.
| Agent harness | Model | Terminal-Bench 4.0 score | Notes |
|---|---|---|---|
| Codex | GPT-6 Astra | 58.2% | Current leaderboard top |
| Claude Code | Claude Fable 5.1 | 57.9% | Within run to run variance of first place |
| Claude Code | Claude Opus 5 | 52.4% | Same harness, six points below the Fable 5.1 pairing |
| Codex | GPT-5.6 Sol | Lower overall, ties on business ops tasks | Cheaper per task, competitive on a specific task category |
Why the harness and the model are two separate variables
The number everyone quotes from a coding benchmark is usually attributed to the model. Terminal-Bench 4.0's leaderboard format, which scores a harness and model pairing as a single row rather than crediting the model alone, makes clear that this attribution is wrong often enough to matter. Claude Code running Claude Opus 5 and Claude Code running Claude Fable 5.1 are the same tool calling loop, the same terminal access, the same eight hour budget. The only variable that changes is the model behind it, and the score moves by six points. That is a larger swing than the entire gap separating first and second place on the leaderboard.
The practical implication is that a claim like "Claude Code scored 58 percent" is incomplete in a way that changes the decision it is meant to inform. The harness sets the ceiling for how well a model's capabilities translate into finished terminal work: how it retries, how it recovers from a failed command, how much context it keeps live across a long task. The model sets how far the agent gets when the harness gets out of its way. Comparing tools without pinning down which model sits behind each one is comparing incomplete pairs, which is exactly the trap we flagged from the model quality side in SWE-bench Verified and what the score actually measures: a single number, taken out of its harness context, tells you less than it appears to.
What the near tie at the top actually means
A 0.3 point gap between Codex plus GPT-6 Astra and Claude Code plus Claude Fable 5.1 is not a result you should plan a procurement decision around. Terminal-Bench 4.0's own methodology, five trials per task with reported variance, exists precisely because a single run of 66 tasks produces noise on that scale from randomness in agent behavior alone. Treat the top two rows as tied and look at the axes that actually differentiate your use case: task category performance, cost per completed task, and latency under your specific workload, not the headline percentage.
Where the leaderboard is informative is in the categories underneath the top line score. Terminal-Bench 4.0 breaks out performance across software, machine learning, science, hardware, security, media, and operations, and the pairings do not rank identically across every category. GPT-5.6 Sol on Codex, a cheaper model than GPT-6 Astra, ties on business operations tasks despite trailing on the aggregate score. If your actual workload skews toward one of these categories rather than an even mix, the aggregate ranking can point you at the wrong tool entirely. We covered the same category level nuance from the adoption side in the coding agent adoption and Terminal-Bench landscape, and the lesson repeats here: pull the category breakdown before you pull the headline number.
The composable stack changes what "choosing a tool" even means
Terminal-Bench 4.0's harness plus model format is also a reflection of how teams are actually deploying these agents in September 2026. Claude Code, Codex, and Cursor are increasingly run together rather than as exclusive choices, with orchestration, execution, and review split across the three rather than consolidated into one. We laid that shift out in Cursor, Claude Code, and Codex as a composable stack, and Terminal-Bench 4.0's scoring format is a quiet acknowledgment of the same reality: the question is not which single agent wins, it is which model you pair with which harness for which category of task. A team running a composable stack can route hardware or security category tasks to whichever pairing scores best in that specific column, rather than committing every task type to one tool's aggregate score.
Cost per completed task is the number the leaderboard does not show you
An eight hour agent timeout across five trials on 66 tasks is an expensive benchmark to run, and it is an expensive pattern to replicate in production if you are paying per token for a frontier model on every task regardless of difficulty. GPT-6 Astra and Claude Fable 5.1 are both premium tier models, and a pairing that trails by a point or two on the aggregate score but costs meaningfully less per completed task, such as the GPT-5.6 Sol pairing on business operations work, can be the better choice for high volume, lower stakes task categories even though it will never top the leaderboard. The methodology writeups from the benchmark maintainers report scores, not dollar cost per completed task, so that calculation is one you have to build yourself against your own task mix.
FAQ
Is Codex better than Claude Code based on Terminal-Bench 4.0? The top pairings are within 0.3 points of each other, inside the benchmark's own run to run variance. Treat the top of the leaderboard as a tie and look at category level breakdowns and cost per task for your specific workload instead of the aggregate score.
Why does the same harness score so differently with different models? Claude Code paired with Claude Opus 5 scores 52.4 percent against 57.9 percent for the same harness paired with Claude Fable 5.1. The harness sets the ceiling for how tool calling and recovery behavior translate into finished work, but the model determines how far the agent actually gets within that ceiling.
What does Terminal-Bench 4.0 actually test? 66 tasks across software engineering, machine learning, science, hardware, security, media, and general operations, each run five independent times under an eight hour agent timeout, with variance reported alongside the score.
Should I pick one agent for every task category? Not necessarily. Category level scores do not rank identically to the aggregate, and a composable stack that routes different task categories to different harness and model pairings can outperform committing everything to one tool's overall ranking.
When this applies to your stack
If you are choosing or standing up a coding agent stack, the aggregate Terminal-Bench 4.0 number is a starting point, not a decision. Pin down which model sits behind each harness you are evaluating, pull the category breakdown for the task types your team actually runs, and price out cost per completed task rather than cost per token. That evaluation work, plus the agent engineering to route tasks to the right pairing in production, is the kind of applied AI work we do at Contra Collective.
More from the lab.
vLLM vs SGLang: Choosing a Production LLM Inference Engine in Late 2026
vLLM and SGLang are both open source, both built to serve LLMs at production scale, and both frequently recommended for the exact same job, which is exactly why teams keep picking the wrong one. The real difference is not a benchmark number. It is which attention and caching architecture matches your actual traffic pattern, and most teams never check that before they commit infrastructure budget to one or the other.
Ollama's MLX Backend Beats Metal by 93 Percent: Apple Silicon Local Inference Tested (October 2026)
Ollama's 0.19 release notes say it plainly: on Apple Silicon, Ollama is now built on top of Apple's MLX framework to take advantage of its unified memory architecture, rather than the Metal backed llama.cpp path it shipped for years. On an M4 Max running Qwen3.5, that swap took decode speed from 57.8 tokens per second to 111.4, a 93 percent improvement, and prefill from 1,154 to 1,810 tokens per second. The catch is in the fine print: the MLX runner is in preview, it needs 32GB or more of unified memory to activate at all, and as of this release it only accelerates one model family. Here is what actually changed, what it costs to turn on, and where it still falls back silently to the slower path.
DGX Spark vs Mac Studio M5 Ultra: Local LLM Inference Benchmarked (October 2026)
The M5 Ultra Mac Studio shipping in September 2026 is a quad die chip with 1.2 terabytes per second of memory bandwidth and up to 256GB of unified memory, with a 512GB configuration arriving in late October. Against a single NVIDIA DGX Spark, it is not close: on a 27B dense model, the Mac decodes at roughly four times the speed. Cluster two DGX Spark units together over NVIDIA's GX10 networking instead, and the comparison flips for a different workload entirely, concurrent multi user serving. Neither machine is the right answer for every workload, and the decision comes down to whether you are serving one user or several at once.