GPT-5.5 vs Opus 4.8 vs Gemini 3.5 Pro: Tau-Bench Tool-Use Agent Reliability Tested (July 2026)

Every team shipping an agent that calls tools on a user's behalf, refunding an order, rebooking a flight, updating a subscription, eventually learns that a benchmark measuring one attempt tells you almost nothing about what happens across a thousand real sessions. Tau-Bench is the benchmark built for exactly this gap. It runs a full multi turn conversation between the model, a simulated user, and a set of tools, and it checks two things a single shot score cannot: whether the final database state is correct, and whether the agent respected a written policy while getting there. Then it runs the same task eight times and reports how often the model succeeds every single time. We tested GPT-5.5, Opus 4.8, and Gemini 3.5 Pro across both Tau-Bench domains.

Metric	GPT-5.5	Opus 4.8	Gemini 3.5 Pro
Retail pass@1	74%	78%	71%
Airline pass@1	58%	64%	55%
Retail pass^8 (all 8 trials)	41%	56%	38%
Airline pass^8 (all 8 trials)	22%	34%	19%
Tool call accuracy (valid args)	93%	96%	91%
Policy violations per 100 tasks	9	4	11

Pass@1 is the chance the agent completes a task correctly on one attempt. Pass^8 is the chance it completes the same task correctly on all eight attempts, which is the number that predicts unattended reliability. The gap between those two columns is the whole story of this post. On a single attempt the three models look within a few points of each other. Run each task eight times and the field splits hard: Opus 4.8 holds far more of its pass@1 score than GPT-5.5 or Gemini 3.5 Pro, which means it is not just capable, it is consistent, and consistency is what you are actually buying when you put an agent on a live workflow.

Pass@1 Is the Demo, Pass^8 Is Production

A pass@1 score is what you see in a demo. You run the task once, it works, the room nods. Pass^8 is what your users see, because your agent runs the same class of task over and over and any run that goes wrong is a ticket, a bad refund, or a rebooking that stranded a traveler. The reason the two diverge is variance. A model with a 74 percent pass@1 and high variance is right most of the time and wrong in a different way each time, so the probability it clears eight independent runs collapses. A model with the same pass@1 and low variance clears far more of them.

That is why Opus 4.8's retail pass^8 of 56 percent against GPT-5.5's 41 percent matters more than the six point pass@1 lead that produced it. The pass@1 gap is modest; the pass^8 gap is a different regime of trust. On the airline domain, which is harder because its policy is stricter and its tasks chain more tool calls, every model drops, but the ranking holds and the spread widens in relative terms. Opus keeps a third of its tasks perfect across all eight runs where Gemini keeps under a fifth. If your agent lives in a domain that looks like airline, dense policy, long chains, expensive mistakes, that column is the one to underwrite your launch on.

Policy Adherence Is a Separate Skill From Task Success

The second thing Tau-Bench measures, and the thing most function calling benchmarks miss entirely, is whether the agent followed the rules while succeeding. A model can complete the task and still violate policy: issuing a refund it was not authorized to issue, skipping an identity check, or rebooking outside the fare rules. Tau-Bench encodes these as explicit policy documents the agent must obey, and it counts violations separately from task failures.

Here Opus 4.8 separates again, at 4 policy violations per 100 tasks against 9 for GPT-5.5 and 11 for Gemini 3.5 Pro. This is not the same axis as raw capability. A model can be highly capable and still cut corners under pressure, and in an agent that touches money or bookings the corner it cuts is your liability. The pattern mirrors what we found on strict schema conformance, where the model that produced the most valid tool arguments was not always the most raw capable one, a dynamic we broke down in the tool use reliability and strict JSON schema comparison. Valid arguments and obeyed policies are both forms of doing the boring part right, and the boring part is what breaks in production.

Where GPT-5.5 and Gemini Still Earn a Slot

None of this makes Opus the answer to every agent. GPT-5.5's pass@1 is close, its tool call accuracy is strong, and in a workflow where a human reviews every action before it commits, the pass^8 gap matters far less because the human is the consistency layer. If your agent proposes and a person approves, you are effectively living in the pass@1 column, and there GPT-5.5 is competitive and often cheaper per task depending on your token profile. Gemini 3.5 Pro trails on both reliability columns here, but it brings the largest context window of the three, which is decisive when the agent has to reason over a long policy document or a sprawling conversation history that the other models would have to summarize and risk dropping a detail from.

The engineering takeaway is to match the model to the supervision level of the workflow. Fully unattended, high stakes, long chain agents want the model with the best pass^8 and the fewest policy violations, which is Opus 4.8 in this test. Human in the loop or low stakes agents can trade some consistency for cost or context, which opens the door to GPT-5.5 or Gemini 3.5 Pro. The mistake is reading the pass@1 row, seeing a near tie, and picking on price, because the near tie evaporates the moment the agent runs unsupervised at volume.

# Route by supervision level, not by pass@1
def select_agent_model(workflow):
    if workflow.unattended and workflow.stakes == "high":
        # consistency and policy adherence dominate; underwrite on pass^8
        return "opus-4-8"
    if workflow.context_tokens > 200_000:
        # long policy or history that must stay resident
        return "gemini-3-5-pro"
    # human approves each action; you effectively live in the pass@1 column
    return "gpt-5-5"

When This Applies to Your Stack

Choose Opus 4.8 when your agent runs unattended against a strict policy and a wrong action is expensive, because its pass^8 consistency and low policy violation rate are exactly the reliability that keeps an autonomous workflow out of the incident channel. Choose GPT-5.5 when a human reviews each action or the stakes are low enough that occasional variance is cheap to catch, where its strong pass@1 and tool call accuracy make it competitive, often at a better cost per task. Choose Gemini 3.5 Pro when the agent must hold a very long policy or conversation in context, where its window advantage outweighs its lower reliability scores. The same calibration versus raw capability split shows up on factual answering, where knowing when to abstain beat knowing more, which we tested in the SimpleQA hallucination comparison.

If your team is putting an agent in front of customers and needs pass^8 reliability and policy adherence measured on your own tools and your own policy rather than a public benchmark, Contra Collective builds and evaluates these agent systems with the cost of a wrong action priced into the acceptance criteria. The leaderboard rewards getting it right once; your users need it right every time.

FAQ

What is Tau-Bench and why does it matter? Tau-Bench evaluates tool using agents by running a full multi turn conversation between the model, a simulated user, and a set of tools, then checking the final system state and whether the agent obeyed a written policy. It matters because it measures reliability across repeated runs, not a single lucky attempt, which is what predicts real agent behavior.

What is the difference between pass@1 and pass^8? Pass@1 is the chance the agent completes a task correctly on one attempt. Pass^8 is the chance it completes the same task correctly on all eight attempts. Pass^8 is the better predictor of unattended production behavior because it exposes variance that a single attempt hides.

Which model is most reliable for an unattended agent in July 2026? In this test, Opus 4.8. It held far more of its pass@1 score under the pass^8 measure and logged the fewest policy violations, which is the profile you want when no human reviews each action before it commits.

Does a higher pass@1 mean a better agent? Not on its own. Two models with similar pass@1 can behave very differently across repeated runs, and the one with lower variance is far more useful in production. Pass@1 is the demo number; pass^8 and policy adherence are the production numbers.

When is GPT-5.5 or Gemini 3.5 Pro the better pick? GPT-5.5 fits human in the loop or low stakes agents where variance is cheap to catch and cost per task matters. Gemini 3.5 Pro fits agents that must keep a very long policy document or conversation history resident in context, where its larger window outweighs its lower reliability scores here.