One robots.txt rule to opt out of AI training: what it does at Google, Bing, OpenAI and Anthropic

The question

What is the shared robots.txt AI training opt-out, and should a business use it?

The short answer

It is a robots.txt group, User-Agent: __usage_ai-training with Disallow: /, that Google, Microsoft and OpenAI said on 30 October 2026 they will honor, and that Anthropic has started implementing. It comes from an individual IETF draft with no formal standing. Microsoft starts on 1 February 2027; Google's crawler docs say early 2027; OpenAI says the coming months; Anthropic gave no date. On Bing it covers training only, and grounding is a separate page-level setting. On Google it counts as disallowing Google-Extended, which also covers grounding in Gemini Apps, unless you list Google-Extended yourself. If you want AI answers to cite you, check what it does on each engine before you add it.

TL;DR: On 30 October, Google, Microsoft and OpenAI committed to one robots.txt opt-out from AI training, and Anthropic says it has started implementing it. None of them reads it yet, and the same two lines do different things at each company. Our read: decide training and grounding separately, per engine, before anyone adds it.

The likely default: someone reads the headline, adds the opt-out to robots.txt, and assumes the site still shows up in AI answers.

On Bing it would. On Google, by default, it wouldn't in Gemini Apps.

What changed

One opt-out group, four companies

The spec is draft-kelley-aipref-training-preference-signal-00, submitted on 30 October by Kevin Kelley of Anthropic, its sole author. The IETF datatracker lists it as an individual Internet-Draft that is "not endorsed by the IETF" and has "no formal standing" in the standards process. It expires on 3 May 2027 unless revised.

It adds no new robots.txt syntax. A site writes an ordinary group whose User-Agent line names a "policy token":

User-Agent: __usage_ai-training
Disallow: /

That reads as a preference against AI training for the whole site. Paths work as they do today, so you can exempt a folder with an Allow: line.

Three details in the draft matter more than the headline:

• User-Agent: * does not count. Wildcard groups must not be read as matching the token. A file without a __usage_ai-training line expresses no training preference at all.
• It is not a crawl rule. The draft says the token "is not a directive for crawling behavior." To keep a crawler out, you still need a group for that crawler's own name.
• Company-specific tokens stay separate. A rule for one company's own token doesn't count as a preference under the shared signal, and the draft lets companies treat their own token as more specific than the shared one.

"AI training" here means using content to change the learned parameters of a generative AI model. Training models used only for tasks like classification or ranking is not included.

Who starts when

• Microsoft: honors the new preferences as it crawls, starting 1 February 2027.
• Google: the blog post says "in 2027." Google's crawler documentation says early 2027.
• OpenAI: aims "to implement the signal in the coming months," and says its existing crawler controls stay in place. Its post describes the signal as covering "foundational generative AI training" and doesn't say whether that wording is meant to narrow the draft's definition.
• Anthropic: Kelley wrote on the IETF ai-control mailing list that Anthropic "has already started implementing support," that it complements Anthropic's existing crawler controls, and that it will "take us some time." No date.

On Google, training and Gemini grounding share one switch

Google says that if a site opts out with the shared token and does not name Google-Extended, Google will treat it as if the site had disallowed Google-Extended.

Google-Extended is not only a training control. Google's crawler documentation says it covers training future Gemini models and grounding in Gemini Apps and in Grounding with Google Search on Vertex AI. Grounding means feeding pages to the model at answer time. Google's post adds that sites that "specifically allow Google-Extended are eligible to appear in responses and receive clicks via grounding in Gemini Apps."

So the shared line, on its own, also takes your pages out of Gemini Apps answers. If you allow Google-Extended explicitly to stay in them, Google's docs describe no separate setting that keeps your pages out of Gemini training. On Google, today, training and Gemini grounding are one decision. That conclusion is our reading of the two documents; Google didn't put it that way.

Google's documentation also says Google-Extended does not affect a site's inclusion in Google Search and is not a ranking signal.

Bing splits training from grounding

Microsoft now offers two independent preferences:

• AI Training: the shared robots.txt group, for a site or a path. Microsoft also reads it per page, and the page setting overrides robots.txt.
• AI Grounding: page level only. There is no site-wide robots.txt version. Three options: Allow (the default), DisallowExtended, which keeps grounding in Microsoft-operated products such as Copilot and Microsoft 365 but excludes grounding services Microsoft sells to third parties, and Disallow.

Both default to Allow. Microsoft's own example is training set to Disallow and grounding set to Allow, which keeps a page eligible to inform AI answers without being used for training. Microsoft says these settings don't change eligibility for regular Bing results, and warns that DisallowExtended or Disallow for grounding will reduce your visibility in AI experiences. It also says neither choice guarantees inclusion, traffic or payment.

The page-level syntax, as a meta tag or an HTTP header:

<meta name="Bingbot" content="ai-training:disallow">
<meta name="Bingbot" content="ai-grounding:disallow">

X-Robots-Tag: Bingbot: ai-grounding:disallowextended

NoArchive and NoCache stop counting

Since 2023, Microsoft has read NoArchive and NoCache as AI preferences. That ends. The dates:

• 15 December 2026: a temporary grounding preference opens in Bing Webmaster Tools, only for sites using NoArchive or NoCache on that date.
• 1 February 2027: the new preferences start taking effect as pages are crawled.
• 1 March 2027: after this, NoArchive and NoCache are no longer read for training or grounding. Sites that want to keep restrictions must publish the new preferences, and for grounding also record the temporary preference in Bing Webmaster Tools, before this date. Otherwise those pages default to Allow for grounding until Microsoft recrawls them.
• 30 September 2027: any temporary preference still in place expires.

Microsoft notes that publishing a setting doesn't activate it. It has to crawl the page first, and IndexNow can encourage a recrawl but doesn't guarantee one.

What it means for a business

This section is our read, not the sources'.

There are now three separate questions, and robots.txt used to blur them: can a crawler fetch the page, can the page train a model, and can the page be used to answer a question with a link back to you. The shared token answers the second. Crawl rules answer the first. Grounding is answered differently by each company, and on Google it is tied to training.

The cost of the shared line depends on what you want from AI answers. A publisher protecting paid content may be happy to leave Gemini Apps. A company that wants to be cited when buyers ask an AI assistant about its category is giving up an answer surface on Google, by default, in exchange for a training preference that no one honors until 2027.

None of this is enforced. OpenAI calls its support a "voluntary effort," and the draft describes a preference, not a control. It works for the companies that choose to read it.

What to do this week

If you only do three: 1, 2 and 5.

1. Write down your position on each use, per engine: search results, AI answers that cite you, and model training. If your position is "allow search, allow citations, block training," Google has no setting for it today.
2. If you want AI citations, don't add the line on reflex. No change is needed to stay eligible. Bing's grounding default is Allow, and a robots.txt with no Google-Extended or shared-token group puts no limit on Google-Extended.
3. If you opt out of training, choose your Google trade-off explicitly. Either accept leaving Gemini Apps grounding, or add your own Google-Extended group and accept Gemini training. The example below shows the second option.
4. Don't treat it as a block. If you want a crawler to stop fetching pages, keep or add a group for that crawler's own name. The new token doesn't control crawling, and User-Agent: * doesn't express a training preference.
5. If you use NoArchive or NoCache, put 15 December 2026 and 1 March 2027 in the calendar. Decide your Bing training and grounding settings, and record the temporary grounding preference in Bing Webmaster Tools if you need it.
6. Check your pages, not just robots.txt. Bing's grounding setting lives on each page. If templates or a CDN add headers, confirm what actually ships on a live page.
7. Revisit when OpenAI and Google publish implementation details. Both promised updates. Until then, the only fixed start date is Microsoft's.

A robots.txt that opts out of training for every participating company except Google, and stays in Gemini Apps grounding:

User-Agent: __usage_ai-training
Disallow: /

User-Agent: Google-Extended
Allow: /

Delete the second group if you would rather be out of Gemini Apps than in Gemini training. Neither group changes which crawlers can fetch your pages.