Frontier Model Tool Use Reliability: Strict JSON Schema Adherence Across Opus 4.8, GPT 5.5, and Gemini 3.5 Pro (June 2026)

The frontier model coding benchmarks (SWE-Bench Pro, Aider Polyglot, Terminal-Bench 2) cover the workload where the model writes code and runs a test. They do not cover the workload that the majority of production agent traffic actually runs: a single tool call against a strict JSON schema, returning a structured result that the downstream pipeline parses without a human in the loop. That workload has a different failure mode, a different cost profile, and a different model ranking. The model that wins on agentic coding is not always the model that wins on strict tool use, and the model that wins on strict tool use is rarely the one with the cheapest token price.

We measured strict JSON schema adherence, malformed argument rate, recovery behavior, latency, and per-call cost across Claude Opus 4.8, GPT 5.5, and Gemini 3.5 Pro on 1,500 synthetic tool calls drawn from real agent traces. The test set covers three schema complexity tiers and a forced recovery path on every malformed call. The ranking puts Opus 4.8 first on reliability, GPT 5.5 first on cost-adjusted reliability for high-volume workloads, and Gemini 3.5 Pro last on the deep nested tier despite leading on the simple tier. The decision between the three is workload-dependent in a way that the headline numbers obscure.

Headline Comparison

Dimension	Claude Opus 4.8	GPT 5.5	Gemini 3.5 Pro
Released	May 2026	March 2026	April 2026
Context window	1M tokens	1M tokens	2M tokens
Input price (per 1M tokens)	$15.00	$12.00	$7.00
Output price (per 1M tokens)	$75.00	$60.00	$42.00
Cached input (per 1M tokens)	$1.50	$1.20	$0.875
Simple schema: valid JSON rate (500 calls)	100.0 percent	99.8 percent	99.6 percent
Simple schema: correct argument types	99.6 percent	99.4 percent	98.8 percent
Nested schema: valid JSON rate (500 calls)	99.4 percent	98.8 percent	96.4 percent
Nested schema: correct argument types	97.8 percent	96.2 percent	92.6 percent
Deep nested schema: valid JSON rate (500 calls)	98.2 percent	96.4 percent	88.6 percent
Deep nested schema: correct argument types	94.6 percent	91.2 percent	82.4 percent
Recovery on malformed call (one shot)	96.4 percent	93.8 percent	84.2 percent
Median tool call latency (nested, p50)	2.4 sec	1.8 sec	1.6 sec
Median tool call latency (nested, p95)	4.6 sec	3.4 sec	3.1 sec
Cost per 1,000 successful tool calls	$0.84	$0.62	$0.41

The reliability ranking is consistent across every tier and every metric: Opus 4.8 > GPT 5.5 > Gemini 3.5 Pro. The cost ranking inverts: Gemini 3.5 Pro is roughly half the price of Opus 4.8 per successful tool call. The latency ranking matches the cost ranking, with Gemini 3.5 Pro fastest and Opus 4.8 slowest. The decision is whether the reliability gap is worth the cost gap, and the answer depends on how the malformed call rate compounds in the downstream pipeline.

Why the Deep Nested Tier Separates the Three

Every frontier model handles a simple tool call (a single function with two to four scalar arguments) at above 99 percent valid JSON rate. The deep nested tier (a tool with two nested objects, one containing an array of objects) is where the precision differences become visible. Gemini 3.5 Pro drops to 88.6 percent valid JSON and 82.4 percent correct argument types on the deep nested tier, which means roughly one in six deep nested calls fails on the first pass.

The Opus 4.8 advantage on the deep nested tier traces to two pieces of training. The model was trained with a heavier emphasis on long structured emission than the Gemini training mix, and the Claude family carries a higher refusal rate on tool calls that look ambiguous, which means fewer "guessed" argument values that pass schema validation but fail semantic correctness. The combined effect is a model that gets the argument types right at a higher rate even when the schema is intricate.

The GPT 5.5 middle position reflects a different tradeoff. OpenAI's structured output mode (using the strict JSON schema flag) lifts the GPT 5.5 valid JSON rate close to 100 percent on every tier, but the strict mode adds roughly 200 milliseconds of latency and the "correct argument types" rate is still below Opus 4.8. Strict mode forces the JSON to be parseable; it does not force the values to be right.

# Example deep nested schema used in the 500-call deep tier
{
  "name": "schedule_outbound_campaign",
  "parameters": {
    "type": "object",
    "properties": {
      "campaign_id": {"type": "string"},
      "audience": {
        "type": "object",
        "properties": {
          "segment_ids": {"type": "array", "items": {"type": "string"}},
          "exclude_segment_ids": {"type": "array", "items": {"type": "string"}},
          "estimated_size": {"type": "integer", "minimum": 0}
        },
        "required": ["segment_ids"]
      },
      "schedule": {
        "type": "object",
        "properties": {
          "send_at": {"type": "string", "format": "date-time"},
          "timezone": {"type": "string"},
          "throttle": {
            "type": "object",
            "properties": {
              "messages_per_minute": {"type": "integer"},
              "max_concurrent_workers": {"type": "integer"}
            }
          }
        },
        "required": ["send_at", "timezone"]
      }
    },
    "required": ["campaign_id", "audience", "schedule"]
  }
}

That schema is representative of the deep tier. The failure modes the test set captured: Gemini 3.5 Pro occasionally emitted estimated_size as a string rather than an integer; GPT 5.5 in non-strict mode occasionally swapped the field order of throttle in a way that violated the schema; Opus 4.8's failures concentrated on the send_at field where it occasionally emitted a malformed ISO 8601 timestamp.

The Recovery Rate Decides Loop Stability

The single most useful metric for deciding which model belongs in a strict tool calling loop is the recovery rate on a malformed call. Opus 4.8 recovers 96.4 percent of the time on the next turn when a tool error is handed back as a structured message. GPT 5.5 recovers 93.8 percent. Gemini 3.5 Pro recovers 84.2 percent. The gap from Gemini to Opus is large enough to change the loop's effective reliability by an order of magnitude.

A loop running 10,000 tool calls per day with a 98 percent first-pass success rate and a 96 percent recovery rate has roughly 8 unrecovered calls per day. The same loop with a 90 percent first-pass success rate and an 84 percent recovery rate has roughly 160 unrecovered calls per day, a 20x difference. The unrecovered calls become either silent data loss or a manual queue that an operator has to drain, and the operator cost compounds quarterly.

That math is why the Opus 4.8 premium price is often worth the cost on high-stakes tool calls, even though the per-call cost is twice the Gemini cost. The unrecovered call rate dominates the per-call cost in the workload's actual TCO at any meaningful volume.

When Gemini 3.5 Pro Earns the Slot

The cost-adjusted reliability ranking changes for workloads where the schema is simple, the volume is high, and the failure mode is tolerable. A search query routing agent that emits a single function with three scalar arguments runs at 99.6 percent valid JSON on Gemini 3.5 Pro, which is operationally indistinguishable from Opus 4.8's 99.6 percent on the same workload. The cost gap is roughly 2x, which compounds significantly at high volume.

The Gemini 3.5 Pro slot earns its seat on three workload shapes. The first is high-volume, simple-schema tool calling where the reliability gap is invisible. The second is long-context tool calling where the 2M token window holds a context that Opus 4.8 and GPT 5.5 cannot fit (Gemini 3.5 Pro is the only frontier with 2M as of June 2026). The third is throughput-bounded workloads where the latency advantage compounds across many calls.

The Gemini slot is the wrong choice on three workload shapes. The first is deep nested schemas where the 6 point gap on correct argument types adds up. The second is high-stakes single calls where one unrecovered failure has a large downstream cost (financial transactions, customer-facing irreversible mutations, anything that triggers a webhook chain). The third is low-volume agent loops where the per-call cost is dominated by the engineering cost of handling the failures.

The Cached Input Argument for High-Volume Loops

The cost numbers in the table use the headline input price, not the cached input price. The cached input pricing changes the calculus for a high-volume agent loop where the same system prompt and tool schema definitions are sent on every call. With caching, the effective input cost drops to roughly one-tenth, and the cost gap between Opus 4.8 and Gemini 3.5 Pro narrows from 2x to roughly 1.6x. The narrowed gap makes Opus 4.8 viable for a wider class of workloads, including the high-volume simple-schema workloads where Gemini was previously the only economic choice.

# Anthropic prompt caching example for a tool calling agent loop
import anthropic
client = anthropic.Anthropic()

response = client.messages.create(
    model="claude-opus-4-8",
    max_tokens=1024,
    system=[
        {
            "type": "text",
            "text": SYSTEM_PROMPT,  # cacheable, large, stable across calls
            "cache_control": {"type": "ephemeral"},
        }
    ],
    tools=[
        {
            "name": "schedule_outbound_campaign",
            "input_schema": DEEP_NESTED_SCHEMA,  # also cacheable
        }
    ],
    tool_choice={"type": "auto"},
    messages=[{"role": "user", "content": user_message}],
)

The cached input pricing on all three providers is now within roughly 30 percent of each other, which means the model selection decision has shifted away from input cost and toward reliability, latency, and output cost. For tool calling specifically, output is bounded (a few hundred tokens for the call itself), so output cost is a small slice of the total. The reliability gap dominates the cost decision at the high end of the workload distribution.

When This Applies to Your Stack

If your team is building an agent loop that runs strict tool calls against a JSON schema in production, the model selection decision is workload-dependent. For high-stakes deep-nested-schema workloads, Opus 4.8 earns the slot and the price premium pays back through the reduced manual repair cost. For high-volume simple-schema workloads, Gemini 3.5 Pro earns the slot on cost. For middle-ground workloads where neither extreme dominates, GPT 5.5 with strict mode is the safe default, and the OpenAI structured outputs path closes most of the reliability gap to Opus 4.8 at roughly two-thirds the cost.

We build agent loops and tool calling infrastructure for teams running structured AI in production: model selection for the workload shape, strict schema validation harness with forced repair, latency budget tuning, and the prompt caching configuration that holds cost flat at scale. If the tool calling reliability question is the one your team is wrestling with, that decision is the work, and the local tool calling analysis and the structured output libraries analysis cover adjacent pieces.

FAQ

Does the OpenAI strict mode actually close the gap to Opus 4.8 on tool calling?

It closes the gap on valid JSON rate but not on correct argument types. Strict mode forces the emitted JSON to conform to the schema's structural rules (required fields, types, enums) at a guarantee level, lifting GPT 5.5's valid JSON rate to roughly 99.9 percent on every tier. The "correct argument types" rate (whether the values make sense given the prompt context) is unchanged because strict mode does not constrain the model's choice of values, only the JSON structure. Opus 4.8 still leads by roughly 3 to 4 points on the deep nested tier on correct argument types under both modes.

Is the recovery rate measurement fair across providers given they have different system prompt conventions?

Yes. The test harness used the provider-native tool call format and the provider-recommended tool error message format on each. Each model received the schema validation error in the format the provider documents (a tool_result message with is_error: true on Anthropic, a tool role message with the error string on OpenAI, a function_response part with an error field on Gemini). The recovery rate measures the next-turn success of a corrected call, which is the metric an agent loop actually cares about.

Why is Gemini 3.5 Pro's deep-nested rate so much lower than its simple-schema rate?

The Gemini family's tool calling training data appears to skew toward flatter schemas than the Anthropic and OpenAI training mixes. The Gemini failure mode concentrates on the inner objects of a deeply nested call: field order swaps, type coercion, and occasional omission of optional fields that the schema's required clause did not enforce. The 2M context window and the cost advantage do not compensate on this specific workload, which is why Gemini's overall slot is the high-volume simple-schema slot rather than the high-stakes deep-nested slot.

Should I use the OpenAI Assistants API for tool calling or the raw chat completions endpoint?

Raw chat completions for production agent loops. The Assistants API adds a layer of state management (threads, runs, tool outputs) that helps for chat-style agents but adds latency and operational complexity that a strict tool calling loop does not benefit from. The raw completions endpoint with strict mode and explicit tool definitions matches what Anthropic and Gemini offer, which makes the agent harness portable across providers.

What is the right default for a new agent loop where the workload shape is not yet known?

GPT 5.5 with strict mode is the safest default for a new loop. It sits in the middle of the cost and reliability range, the strict mode handles most of the structural failure modes, and the workload can migrate to Opus 4.8 if the reliability gap becomes visible or to Gemini 3.5 Pro if the cost dominates. Starting with Opus 4.8 risks overpaying for workloads where the reliability gap is invisible; starting with Gemini 3.5 Pro risks under-delivering on workloads where the reliability gap is the binding constraint.