Claude Opus 5 vs Grok 4.6: Vending-Bench Long-Horizon Agent Tested

Here is the comparison first, because that is why you clicked. Both models ran the same Vending-Bench harness under one protocol: the model acts as the manager of a simulated vending machine business, with tools to check its balance, order inventory from suppliers, set prices, and collect revenue, over a long sequence of steps with a fixed daily operating fee. The numbers below come from repeated runs on that controlled setup, not a vendor leaderboard, and they are test results, not a claim about any real business.

Metric	Claude Opus 5	Grok 4.6
Mean ending net worth (relative)	higher	moderate
Median run outcome	profitable	profitable
Run to run variance	lower	higher
Bankruptcy rate across runs	rare	occasional
Typical coherence horizon before drift	long	medium
Recovery after a bad decision	frequent	inconsistent
Best single run	strong	strong
Worst single run	contained loss	full collapse

Read that table for the shape, not decimal places. On this benchmark the mean is the least informative column. The variance and the bankruptcy rate are the story, because they tell you whether the model is a reliable operator or a gifted gambler.

Why a vending machine is a serious benchmark

The premise sounds like a toy and the design is anything but. Running the business is trivial as a business. What is hard is doing it for thousands of steps without losing track of state. The model has to remember that it placed an order three days ago and stock has not arrived yet, so it should not reorder. It has to reconcile the cash it thinks it has against the fee it owes today. It has to notice that a price is too high because units are not moving, or too low because it is selling out and leaving money on the table. None of these are hard decisions in isolation. The difficulty is that they accumulate, and a single misremembered fact early can compound into a death spiral hundreds of steps later.

That is exactly the capability that single turn benchmarks cannot measure. A model can ace a coding benchmark or a reasoning exam and still be unfit to run unattended for an afternoon, because those tests reset context after every problem. Vending-Bench never resets. It is the closest common benchmark to the thing you actually want from an agent: stay coherent, remember what you did, and do not fall apart when the run gets long. This is the same reason a tool use agent benchmark and a long horizon coding benchmark disagree so often; they are measuring different muscles.

Where each model actually breaks

The averages hide the interesting part, so here is what the runs looked like under the hood.

Claude Opus 5 was the steadier operator. Its runs clustered: most ended profitable, the spread between a good run and a middling one was narrow, and when it made a bad call, such as overordering a slow item, it usually noticed within a handful of steps and corrected. Its failures were expensive but contained; it rarely drove all the way to zero. The characteristic Opus 5 behavior was conservative bookkeeping. It re checked its balance before committing to orders and it kept a mental ledger that mostly survived long runs. Where it left money on the table was ambition: it under experimented with pricing, converging on a safe price and holding it rather than probing for a better one.

Grok 4.6 had a higher ceiling and a lower floor. Its best runs were as strong as anything, and it was more willing to experiment with price and stock levels, which occasionally found a better operating point than Opus settled for. But its variance was the problem. In a meaningful fraction of runs it lost the thread: it misread its own cash position, reordered stock that was already in transit, or fell into a repetitive loop of near identical actions that bled the daily fee until the business failed. When Grok 4.6 was good it was very good, and when it drifted it drifted all the way to collapse. The failure was not a lack of intelligence in any single step. It was a loss of coherence across many steps, which is precisely what the benchmark is built to catch.

The number that actually matters

If you are choosing a model to run an unattended long horizon workflow, the mean ending net worth is a trap. Two models with the same average can be completely different products: one that reliably lands near the average every time, and one that averages the same by pairing spectacular wins with total wipeouts. For anything you cannot babysit, the second profile is far more dangerous, because a single collapsed run in production is not offset by a great run last week. Opus 5 wins on this axis not because it is smarter per step but because it is more predictable across the whole run, and predictability is the property you are actually buying when you deploy an agent.

The flip side is real too. If a human is in the loop, catching drift and resetting when a run goes sideways, Grok 4.6's higher ceiling and greater willingness to experiment can be the better tool, because you are harvesting its best runs and discarding its bad ones. The right choice is not which model is better. It is whether your deployment has a human who can catch a spiral before it ends the run.

When this applies to your stack

Vending-Bench is a proxy, and the trait it measures, coherence over a long horizon, is the trait that decides whether an agent can be trusted to run unattended: a nightly data pipeline that makes decisions, an autonomous support workflow, a multi step migration agent that operates for hours. If your use case resets context every request, this benchmark is not predictive for you and a single turn coding or reasoning eval will tell you more. If your use case runs long and unsupervised, this is one of the few numbers worth watching, and the column to watch is variance, not the mean.

Choosing and wiring up the right model for an agentic workflow, with the guardrails, checkpointing, and human in the loop escape hatches that keep a long run from spiraling, is the core of what our AI integration and agent engineering practice does. The model is one decision. The system around it that catches drift before it becomes a collapse is the rest of the job.

FAQ

What does Vending-Bench actually measure? Coherence over a long horizon. The model manages a simulated business for thousands of steps, and the benchmark tests whether it can hold state, remember its own past actions, and avoid spiraling, rather than whether it can answer one hard question well.

Which model won, Claude Opus 5 or Grok 4.6? On mean outcome they are close, but Claude Opus 5 was more reliable: lower run to run variance and a lower bankruptcy rate. Grok 4.6 had a higher ceiling on its best runs and a lower floor on its worst.

Is a higher ending net worth the right thing to optimize? Not for unattended use. Two models can share an average while one is steady and the other pairs big wins with total collapses. For an agent you cannot supervise, variance and failure rate matter more than the average.

Does strong coding or reasoning performance predict Vending-Bench performance? Weakly. Single turn coding and reasoning benchmarks reset context every problem, so they do not test the long horizon coherence Vending-Bench is built to expose. A model can top a coding leaderboard and still drift on a long run.

Which should I deploy for a long running agent? If the workflow runs unsupervised, favor the more predictable model, which here was Claude Opus 5. If a human monitors the run and can reset on drift, Grok 4.6's higher ceiling can pay off. Match the model to whether drift will be caught.