llms.txt for Ecommerce: Why Most Retail Product Pages Are Still Invisible to AI Shopping Agents

Detail	Finding
What it is	A Markdown file at a domain's root giving AI systems a curated index of key pages with one line descriptions
Origin	Proposed by Jeremy Howard of Answer.AI in September 2024
Adoption	10.13 percent of 300,000 domains studied by SE Ranking
Crawler behavior	Across more than 515 million LLM bot events, requests to llms.txt from GPTBot, ClaudeBot, PerplexityBot, OAI SearchBot, and Google Extended were negligible
Official support	Google stated in July 2025 it does not support llms.txt; no major provider has publicly committed to using it
Where it does work	IDE and MCP based coding tools, including Cursor, Claude Code, and Copilot, fetch it routinely

llms.txt has a genuine adoption problem and a genuine relevance problem at the same time, and the two facts point in different directions for an ecommerce team deciding whether to bother shipping one. Only 10.13 percent of domains in a 300,000 site SE Ranking study have adopted it. Separately, an analysis of more than 515 million LLM bot events found that the crawlers most people assume the file is for, GPTBot, ClaudeBot, PerplexityBot, OAI SearchBot, and Google Extended, almost never request it. Google said outright in July 2025 that it does not support the standard. If the goal is getting picked up by a general purpose AI search crawler, llms.txt is not currently the lever.

What llms.txt actually is, and what it is not

llms.txt is a plain Markdown file at a site's root that gives an AI system a curated list of a site's most important pages, each with a short, concrete description, rather than asking the system to crawl and parse full HTML. A companion file, llms-full.txt, goes further and embeds the linked page content directly in one document. The underlying argument for the standard is simple: HTML carries a lot of markup, navigation, and styling noise that costs tokens and adds ambiguity for a language model, while a curated Markdown summary gets straight to the content that matters.

It is not a sitemap, and it is not a replacement for structured data like product schema markup. A sitemap tells a crawler everything exists; llms.txt is meant to tell an AI system what matters most and why, in plain language, which is a curation exercise rather than an indexing one.

The uncomfortable part: the crawlers that matter most mostly ignore it

The 515 million event analysis is the number that should temper expectations here. If the AI crawlers behind ChatGPT search, Claude's web tools, and Perplexity are not meaningfully requesting llms.txt today, publishing one will not move your visibility in those surfaces on its own. No major model provider, not OpenAI, not Anthropic, not Google, not Meta, not Mistral, has publicly committed to using it as an input to how their systems discover or rank content. That is a real gap between the standard's promise and its current reality, and any vendor pitching llms.txt as a guaranteed way into AI search results is overselling a file that the evidence says most of those systems are not reading yet.

Where it demonstrably does work today

The file is not useless, it is narrower than its early hype suggested. IDE and MCP based agentic tools, Cursor, Claude Code, and GitHub Copilot among them, do fetch llms.txt routinely when operating against a codebase or a connected site. If your ecommerce stack exposes any kind of developer facing API or documentation that an engineering team's coding agent might reference, that is a real, working use case today, not a speculative one. The other place it has a real foothold is the newer category of true agentic commerce, where a shopping agent is specifically instructed to transact rather than just browse, and the agent's tooling is built to look for exactly this kind of curated entry point rather than crawl a full site like a search engine does.

What a real ecommerce llms.txt actually looks like

The useful version is narrow, specific, and built around transaction readiness rather than general site navigation:

# Acme Outdoor Gear

> Direct to consumer outdoor equipment retailer. Prices in USD, free shipping over $75, 30 day returns.

## Catalog
- [Hiking Boots](/collections/hiking-boots): Trail and mountaineering boots, $89 to $340, sizes 6 to 14
- [Tents](/collections/tents): 1 to 6 person tents, $120 to $950, in stock counts updated hourly

## Policies
- [Shipping](/pages/shipping): Rates, timelines, and carrier options by region
- [Returns](/pages/returns): 30 day window, prepaid label, exceptions for clearance items

The guidance worth following: point to canonical product or category pages instead of cluttered listing HTML, keep the file to roughly 20 to 50 curated links rather than an exhaustive catalog dump, and write descriptions with concrete facts, prices, size ranges, stock status, not marketing language. An agent deciding whether to recommend or transact against a product needs the facts in the description, not an invitation to click through and find out.

The bigger fix llms.txt cannot substitute for

Adobe's holiday research found that roughly 34 percent of retail product pages are effectively inaccessible to AI systems today, and that is not primarily an llms.txt problem. It is a rendering, structured data, and catalog architecture problem: pages that require JavaScript execution an agent cannot reliably run, pricing and availability data that lives only inside a client side cart state, and product information scattered across systems that were never built to answer a machine reader directly. llms.txt is a reasonable, low effort addition on top of that foundation. It is not a fix for a catalog an agent cannot actually parse in the first place, and treating it as one means shipping a curated index that points at pages an agent still cannot read.

When this applies to your stack

If your team is debating whether to spend a sprint on an llms.txt file, the honest framing is that it is cheap to build, genuinely useful for the IDE and MCP agents that already fetch it, and not yet a lever for general AI search visibility given how rarely the major crawlers request it. The higher leverage work is making sure your actual product pages, the ones any agent would need to parse to transact, carry structured, machine readable pricing and availability data in the first place. That catalog and storefront architecture work is where we spend most of our time helping headless commerce teams get agent ready, with llms.txt as a reasonable last step on top of it, not the first one.

FAQ

Does publishing an llms.txt file help with ChatGPT or Perplexity visibility? Not demonstrably yet. An analysis of more than 515 million LLM bot events found requests to llms.txt from the crawlers behind those products were negligible, and none of the major AI providers have publicly committed to using the file.

What is the actual adoption rate of llms.txt? An SE Ranking study of 300,000 domains found 10.13 percent adoption as of its analysis.

Which AI tools actually use llms.txt today? IDE and MCP based coding and agent tools, including Cursor, Claude Code, and GitHub Copilot, fetch it routinely when working against a connected codebase or site.

Should an ecommerce site build an llms.txt file instead of fixing structured data? No. llms.txt is a curated index layered on top of a catalog; it does not fix pages that an agent cannot render or parse in the first place. Structured, machine readable product data is the higher leverage investment, with llms.txt as a complement to it.