OpenAI, Google, and Microsoft Just Backed One robots.txt Token for AI Training Opt Out
On October 31, 2026, Anthropic's Kevin Kelley proposed a single robots.txt policy token, usage AI training, that lets a site opt its content out of AI training with one rule instead of a separate rule per crawler. Google, OpenAI, and Microsoft said the same day they would honor it. The token is a preference, not a crawl order, and the companies are not implementing it on the same timeline. Here is what actually changes for a content heavy site, and what does not.
OpenAI, Google, and Microsoft Just Backed One robots.txt Token for AI Training Opt Out
| Company | What they committed to | Timeline |
|---|---|---|
| Anthropic | Proposed the token, work on support has started | No firm date, described as taking time across services |
| OpenAI | Said it will honor the signal | Coming months |
| Said it will support the signal | 2027, crawler documentation says early 2027 | |
| Microsoft | Will honor the preference, replacing its NoArchive and NoCache tags | February 1, 2027 |
| Meta | Communications head voiced support for a common signal publicly | Did not commit to honoring it |
On October 31, 2026, Anthropic's Kevin Kelley posted an individual submission to the IETF's AI preferences mailing list proposing a shared robots.txt policy token, written as usage colon AI training, that lets a site state in one place that its content should not be used to train generative AI models. Instead of maintaining a separate disallow rule for every company's crawler, a site adds one group with this token and lists the paths it does not want used for training. The draft's own example blocks an entire site except its press kit. Google, OpenAI, and Microsoft announced the same day that they would honor the signal. Meta's communications head, Andy Stone, posted public support for a common signal without committing Meta to actually honoring it, and the IETF itself has not endorsed the draft, which currently has no formal standing in the standards process.
The detail most coverage skipped past is that this is explicitly a preference, not an instruction a crawler is mechanically required to obey, and private agreements between a site and an AI company can override it. Microsoft's Bing principal product manager Krishna Madhavan put the intent plainly: "Publishers should not have to choose between being discoverable in Bing and deciding how generative models use their work." That framing matters because it tells you what problem this actually solves, letting a site separate "please index and show my content in search or AI answers" from "please do not train a model on my content," which current robots.txt syntax conflates for most sites.
What the token actually controls, and what it does not
For a content heavy enterprise site, three things determine whether this standard is worth acting on now or waiting to see how it plays out.
First, the timelines are not aligned, and that gap is the most actionable fact in the entire announcement. Microsoft commits to a specific date, February 1, 2027, when it will start honoring the preference and retiring the NoArchive and NoCache tags it has used since 2023. Google's own crawler documentation says early 2027, and explicitly states that a site using the new token without also addressing Google Extended will be treated as blocking Google Extended specifically, meaning the new token does not automatically map cleanly onto Google's existing crawler controls. OpenAI's commitment is the vaguest, "coming months," with existing controls like GPTBot continuing to apply in the meantime. A site that adds the new token today and removes its existing per crawler rules, assuming the new standard already covers everything, will have gaps for months depending on which company's crawler reaches it.
Second, the token says nothing about licensing or payment, and Microsoft is explicit that this is intentional: the preferences "don't establish licensing or payment arrangements." That distinction matters because it leaves the actual commercial question, whether AI companies should pay for training data, exactly where it was before this proposal, with publishers like the Financial Times, BBC, and Bertelsmann, all thanked in the draft's acknowledgments, continuing to pursue separate payment arrangements through mechanisms like Cloudflare's pay or block crawler policy rather than through this signal.
Third, and most relevant to any brand running a content marketing or SEO program, blocking training under this standard is explicitly decoupled from search visibility and from showing up in AI generated answers. Microsoft states directly that blocking AI grounding does not change a page's position in Bing search results, and a separate, page level grounding control lets a site allow Microsoft's own products, like Copilot, while blocking grounding services Microsoft sells to other companies. This is the same distinction that already matters for how a brand's product pages surface inside AI Overviews and similar answer engines: training opt out, answer engine grounding, and traditional search indexing are now three separate levers, not one setting, and a content strategy that treats them as the same decision will misconfigure at least one of them.
The decision most content teams actually need to make
The real question for a brand running a content program is not whether to adopt the new token, since honoring a widely backed, low cost signal carries little downside, but how to decide what to actually block. Training exclusion and answer engine discoverability pull in opposite directions for most commercial content. A brand that wants its product information surfaced accurately inside ChatGPT, Gemini, or Copilot answers, the entire premise behind the shift toward machine readable discoverability files like llms.txt for ecommerce content, has a direct interest in NOT blocking training on at least its core product and policy pages, since training data and retrieval grounding are not always cleanly separable in how these systems actually work in practice. The clearer use case for the new token is narrower: editorial or research content a brand has invested in, proprietary data, or long form thought leadership meant to build domain authority through being read by humans and cited by other sites, rather than being absorbed wholesale into a competitor's model.
A practical starting point is to apply the new token selectively rather than site wide. Block it on original research, proprietary benchmark data, and long form analysis the brand wants attributed and linked to rather than paraphrased by a model with no citation. Leave core product, pricing, and policy pages open, since those pages benefit from being accurately represented inside AI assisted shopping and research flows, where the point is to be found and transacted with, not withheld.
When this applies to your stack
If your organization runs a content marketing or technical publishing program, this is worth a short review now rather than an immediate wholesale robots.txt rewrite, because the standard has no formal status yet and the four companies backing it are not implementing it on the same schedule. Teams should inventory which pages are genuinely proprietary research worth protecting from training versus product and policy content that benefits from broad AI visibility, and should not assume that adding this one token replaces the crawler specific controls, like GPTBot and Google Extended rules, that still determine most practical outcomes today.
FAQ
What exactly is the new robots.txt AI training opt out token? A policy token, written as usage colon AI training, proposed by Anthropic's Kevin Kelley on October 31, 2026, that lets a site state in one robots.txt group that listed paths should not be used to train generative AI models, replacing the need for separate rules per AI company's crawler.
Which companies have actually committed to honoring it, and when? Google, OpenAI, and Microsoft said they will honor it. Microsoft gave a specific date, February 1, 2027. Google's documentation targets early 2027. OpenAI says coming months with no specific date. Meta's communications lead voiced support without committing to honor it.
Does blocking AI training also block a page from appearing in search or AI generated answers? No. Microsoft explicitly states that blocking AI grounding does not change a page's position in Bing search results, and the token does not affect traditional indexing. Training opt out, answer engine grounding, and search indexing are three separate controls.
Should a brand block AI training on all of its content? Not necessarily. Core product, pricing, and policy pages generally benefit from being visible to AI shopping and research tools. The stronger case for using the new token is on proprietary research, benchmark data, or long form content a brand wants cited and linked to rather than absorbed without attribution.
Sources:
More from the lab.
What Anthropic's Nasdaq Filing Tells Enterprise AI Buyers About Claude's Future
Anthropic filed a public Form S-1 with the SEC on October 26, 2026, naming the Nasdaq and the ticker ANT, disclosing about 16.3 billion dollars in first half 2026 revenue against a net loss of roughly 80.7 billion dollars, a figure the filing attributes mostly to non cash fair value changes on convertible bonds rather than operating losses. A 2 trillion dollar valuation and a listing before Thanksgiving have both been reported but are not confirmed in the filing itself. None of this is investment guidance. It is a read of what the filing actually says for a team that has Claude built into its product roadmap and wants to understand its vendor's position before the IPO news cycle moves on.
Meta and Microsoft Just Cut Internal Claude Spending: What the AI Cost Math Actually Shows
Reports this week say Meta has cut its internal Claude user base roughly in half and Microsoft has trimmed its Anthropic spending budget by about a third, steering employees toward GitHub Copilot instead. Neither company is claiming Claude got worse. They are claiming it got too expensive to run at internal scale, and that distinction matters a great deal if you are budgeting your own AI tool stack for next year.
Mistral Large 4, Le Chonk: Europe's 1 Trillion Parameter Bet Against Chinese Open Weights
Mistral just previewed Large 4, nicknamed Le Chonk internally, a 1 trillion parameter, natively multimodal model with only 49 billion active parameters at inference time. Mistral's own claim is specific: best open weights model out of the US or Europe on aggregated benchmarks, with particular strength in cyber defense, manufacturing, and finance workloads. The open weights release is still pending, which matters as much as the benchmark claim itself.