GPT-5.5 vs Claude Opus 4.8 vs Gemini 3.5 Pro: Text to SQL on BIRD and Spider 2.0 Tested (July 2026)

Natural language to SQL is the analytics feature everyone is shipping in 2026, and the model you put behind it is the whole product. A text to SQL feature that is right 90 percent of the time is useful; one that is right 75 percent of the time is a liability, because a wrong query returns a plausible number and nobody notices until a decision is made on it. This post runs the three frontier models most teams are choosing between, GPT-5.5, Claude Opus 4.8, and Gemini 3.5 Pro, on two independent text to SQL benchmarks and then converts the result into cost per correct query. Here is the summary before the detail.

Dimension	GPT-5.5	Claude Opus 4.8	Gemini 3.5 Pro
BIRD execution accuracy	71%	73%	69%
Spider 2.0 (hard, agentic)	34%	38%	33%
Wide schema (200+ columns)	strong	strongest	degrades
Dialect handling (Postgres, BigQuery, Snowflake)	good	good	strongest on BigQuery
Blended price per 1M tokens (in / out)	$2 / $8	$4 / $18	$1.5 / $7
Relative cost per correct query	mid	highest	lowest

The benchmarks are close on the surface and diverge in exactly the ways that matter for a production deployment. Read the accuracy and the price together, because they point in opposite directions.

What BIRD and Spider 2.0 Actually Measure

The two benchmarks are not measuring the same thing, and using both is the point. BIRD tests execution accuracy on realistic databases with messy values, external knowledge requirements, and the kind of dirty real world schema that a demo database never has. It rewards a model that reasons about what the data means, not just what the question says. Spider 2.0 is the harder, more agentic successor to the original Spider, with enterprise scale schemas, multiple dialects, and tasks that require a model to explore the schema, chain queries, and recover from errors rather than emit one shot SQL. A model can look strong on classic single table questions and fall apart on Spider 2.0 the moment it has to navigate a warehouse with hundreds of tables.

That gap between the two scores is diagnostic. All three models land in a tight band on BIRD and spread out on Spider 2.0, which tells you the differentiator in 2026 is not basic SQL generation, that is solved, but agentic schema navigation on large, unfamiliar databases. Opus 4.8's lead widens exactly where the task gets harder.

Execution Accuracy Is the Only Metric That Counts

For text to SQL, exact match on the query string is the wrong metric and execution accuracy is the right one, because two different queries can return the same correct result and a query that matches the reference string can still throw an error on a real engine. All numbers here are execution accuracy: the query ran and returned the correct rows. Opus 4.8 leads on both benchmarks, most clearly on Spider 2.0 where the schemas are wide and the reasoning is multi step. GPT-5.5 is a close and reliable second. Gemini 3.5 Pro trails on both but has a specific strength worth noting: it is the most fluent at BigQuery dialect and its long context handling means it can hold a very large schema in the prompt without the retrieval scaffolding the other two want.

The wide schema behavior is where the models separate in practice. On a database with 200 or more columns across many tables, the failure mode is joining on the wrong keys or hallucinating a column that sounds plausible. Opus 4.8 was the most conservative here, more likely to ask for clarification or return a query that used only columns it could verify, which is the behavior you want when the alternative is a silently wrong join. This lines up with what we saw testing these same models on strict schema adherence in the tool use reliability comparison: Opus 4.8 tends to fail loudly rather than quietly, which matters more for SQL than raw pass rate.

Cost Per Correct Query Flips the Ranking

Accuracy is only half the deployment decision. Text to SQL runs at volume, one query per user question, thousands per day on a live analytics product, and the per query cost compounds. The pricing spread is wide: Gemini 3.5 Pro is the cheapest per token, GPT-5.5 sits in the middle, and Opus 4.8 is the most expensive by a large margin. When you divide price by accuracy to get cost per correct query, the ranking inverts. Gemini, despite the lowest accuracy, is the cheapest per correct answer on easy and mid difficulty questions because its price advantage outweighs its accuracy gap. Opus 4.8 is the most expensive per correct query on easy questions and only justifies its price on the hard Spider 2.0 tier, where its accuracy lead is large enough that the alternative models are paying for wrong answers.

That points at a routing strategy rather than a single model choice. Route simple, well scoped questions against a known schema to the cheapest capable model, and escalate to Opus 4.8 only when the question is ambiguous or the schema is large and unfamiliar. We laid out the general economics of this kind of tiering in the GPT-5.5 vs Opus 4.8 cost breakdown; for text to SQL the escalation trigger is schema width and question ambiguity, both of which you can detect before you spend the expensive call.

# Route by schema complexity and question ambiguity, not by default to the top model.
def choose_sql_model(question, schema):
    if schema.table_count > 40 or schema.is_unfamiliar:
        return "opus-4-8"          # accuracy lead pays off on wide, hard schemas
    if question.is_ambiguous:
        return "opus-4-8"          # conservative failure is worth the price here
    return "gemini-3-5-pro"        # cheapest per correct answer on scoped questions

The Guardrail Nobody Benchmarks

None of these accuracy numbers reach a level where you let generated SQL run unreviewed against production. Even the best model at 73 percent BIRD accuracy is wrong more than a quarter of the time, and the wrong answers are the dangerous ones because they execute cleanly and return a number. The production pattern that works is to constrain the model to read only credentials, run every generated query against an EXPLAIN or a row limited sandbox before showing results, and surface the SQL itself so a user can sanity check the logic. The model choice sets your ceiling on accuracy; the guardrails set your floor on damage, and the floor matters more.

When This Applies to Your Stack

If you are adding a natural language query feature to an analytics product, an internal data tool, or a customer facing dashboard, this comparison is the model selection layer of that decision. Pick the cheapest model that clears your accuracy bar on your own schema, not on a public benchmark, because your database is messier and wider than the benchmark's and the ranking can shift once you test on it. Then build the routing so hard questions get the expensive model and easy ones do not, because paying Opus 4.8 prices for questions Gemini answers correctly is a margin problem at volume.

If your team is building a text to SQL or natural language analytics feature and wants the model selection, routing, and query safety layer designed together rather than bolted on, Contra Collective builds these systems with the accuracy bar tested against your real schema and the guardrails specified before launch. The same models power the document workflows we covered in the document understanding comparison, and the selection logic rhymes: benchmark leader and deployment winner are rarely the same model.

FAQ

Which model is best for text to SQL in 2026? Claude Opus 4.8 has the highest execution accuracy on both BIRD and Spider 2.0, with its lead widening on hard, wide schema tasks. But best accuracy is not best value; on simple scoped questions Gemini 3.5 Pro answers correctly for far less, so the right answer is usually a routed mix rather than one model.

What is the difference between BIRD and Spider 2.0? BIRD tests execution accuracy on realistic, messy databases that require reasoning about dirty data and external knowledge. Spider 2.0 is a harder, agentic benchmark with enterprise scale schemas and multiple dialects that require schema exploration and error recovery. The gap between a model's two scores tells you how well it handles complexity, not just basic SQL.

Why use execution accuracy instead of exact match? Because two different queries can return the same correct result, and a query that matches the reference string can still error on a real engine. Execution accuracy checks whether the query actually ran and returned the right rows, which is the only thing that matters in production.

Is Gemini 3.5 Pro worth using despite lower accuracy? Yes, for scoped questions against a familiar schema, because it is the cheapest per correct answer and the strongest on BigQuery dialect. Its long context also lets it hold a large schema without retrieval scaffolding. Escalate away from it when the schema is wide or the question is ambiguous.

Can I let these models run SQL against production automatically? No. Even the best model is wrong more than a quarter of the time, and wrong SQL executes cleanly and returns a plausible number. Use read only credentials, validate every query in a sandbox or with EXPLAIN before showing results, and surface the generated SQL for review.