The robots.txt AI Training Opt-Out (__usage_ai-training): Syntax, Dates, and Who Honors It

What is __usage_ai-training?

__usage_ai-training is a policy token for robots.txt that lets a website state one preference about AI training to every AI company that reads it. It sits in the User-Agent line of an ordinary robots.txt group, but it names a use, not a crawler. Disallow means "please do not use these pages to train AI." Allow means the opposite.

The token comes from an IETF Internet-Draft titled "Machine-Readable Training Preference Signal," posted on Oct 30, 2026. Its author is Kevin Kelley of Anthropic. The same day, Google, Microsoft and OpenAI each published their plans to honor it, and the author wrote that Anthropic "has already started implementing support for this draft." The draft builds on the IETF's AI Preferences (AIPREF) working group, whose job is to standardize "the expression of preferences about how content is collected and processed" for AI.

Two cautions belong next to that table. The definition of AI training comes from a vocabulary draft that says its contents "DO NOT REFLECT CONSENSUS of the Working Group." And the token is a preference. The AIPREF charter lists "technical enforcement of preferences" as out of scope. Companies honor it because they said they would.

How does __usage_ai-training relate to Google-Extended, GPTBot and ClaudeBot?

__usage_ai-training sits above the company-specific training controls, not in place of them. Google-Extended, GPTBot, ClaudeBot, Applebot-Extended and meta-externalagent are each one company's control. The shared token speaks to every company that has agreed to read it. The draft lets each company document that its own token takes precedence over the shared one. Google has: if the file names Google-Extended, Google plans to follow that instead. OpenAI and Anthropic say their existing controls continue alongside the token, but have not said which wins if the two disagree.

Put plainly: the token is a shortcut, and the old controls still work. If you already block GPTBot, ClaudeBot and Google-Extended, those rules keep working. The token adds a way to reach every company that adopts it, including ones that publish a new training crawler later. The same split runs through Meta's crawlers, covered in the guide to meta-externalagent.

How do you opt out of AI training in robots.txt?

Add a group for the policy token to the robots.txt file at the root of your site, with Disallow: / to cover every page. These two lines are the whole opt-out:

User-Agent: __usage_ai-training
Disallow: /

The draft's own example scopes the preference by path. This one asks for no AI training anywhere except a press kit, which the site is happy to have used:

User-Agent: __usage_ai-training
Disallow: /
Allow: /press-kit/

• A User-Agent: * group never counts. The draft says wildcard groups "MUST NOT be interpreted as matching the policy token." A site that blocks every crawler under * has still stated no training preference.
• No token group means no preference. If the file has no __usage_ai-training line, "no preference has been expressed."
• It does not block crawling. "This policy token is not a directive for crawling behavior." Crawlers still follow their own groups.
• Case does not matter, and the usual RFC 9309 path rules apply: the longest matching rule wins.
• Older robots.txt parsers simply ignore it, as they ignore any user agent they do not know.

One trap catches sites that already manage AI crawlers by name. If your file names Google-Extended at all, Google plans to follow your Google-Extended rules and ignore the shared token, so a group that allows Google-Extended keeps Google training on. This site is an example: its robots.txt lists Google-Extended, GPTBot and ClaudeBot in the same group as *, with Allow: /. Adding the token alone would not change Google's reading. To opt out everywhere, add the token group and give each named training crawler its own Disallow: / group. For companies other than Google, nobody has said yet how a named crawler group and the token combine, so keeping them consistent is the safe choice.

Should you block AI training bots?

Block AI training if you do not want your pages used to build AI models and you accept losing Gemini Apps grounding. Keep it allowed if being used and cited by AI matters more to you. Opting out of training does not remove you from Google Search or Bing results, by both companies' own documentation. The cost sits in Google's AI apps, because Google ties two things to one switch.

Google's crawler documentation says Google-Extended covers training future Gemini models and "grounding" in Gemini Apps, which means feeding search results to the model while it answers. Google's announcement adds: "Websites that specifically allow Google-Extended are eligible to appear in responses and receive clicks via grounding in Gemini Apps." So a site that opts out through the shared token, without naming Google-Extended, gives up that grounding too. For Google, there is no way to refuse training and keep Gemini grounding.

Two more things to weigh. First, the opt-out is not retroactive. The draft says the preferences that apply to a page "are those expressed in the robots.txt file in effect when the asset was retrieved." Copies collected before you add the line carry no preference, and no company acts on the token until it starts honoring it. Second, the choice is separate from AI search. OpenAI's OAI-SearchBot and Anthropic's Claude-SearchBot are search crawlers, and the token does not touch them. Being cited by ChatGPT depends on those, as the guide to getting cited by ChatGPT explains.

Disclosure: jnabear.com allows AI training on purpose, because being cited is the goal here, and I sell work aimed at AI citations. Weigh my view with that in mind. Many publishers reasonably decide the other way, as the survey below shows.

What is changing at Bing: ai-training and ai-grounding?

Microsoft goes further than the shared token. Its Oct 30 Bing post says it will honor __usage_ai-training in robots.txt, and adds page-level controls for both training and grounding. Grounding means using a page to answer in Microsoft products such as Copilot. "These settings govern use of content, not crawl access," Microsoft says, and they do not change eligibility for Bing search results.

<meta name="Bingbot" content="ai-training:disallow">
<meta name="Bingbot" content="ai-grounding:disallow">
X-Robots-Tag: Bingbot: ai-training:disallow

• A page-level ai-training setting overrides robots.txt for that page.
• Grounding has no robots.txt setting. Microsoft says: "There is no site-wide robots.txt grounding preference."
• ai-grounding:disallowextended keeps grounding to Microsoft's own products and excludes the grounding services Microsoft provides to third parties, such as Web IQ and grounding through Foundry.
• The old NoArchive and NoCache tags stop counting as training or grounding preferences after Mar 1, 2027.
• Once Bing detects any of the new preferences on a site, it ignores NoArchive and NoCache there, even before Mar 1, 2027. If you rely on NoArchive or NoCache to keep pages out of grounding, add ai-grounding tags before you add the token.

How is it different from Cloudflare's Content-Signal?

Cloudflare's Content Signals Policy, launched in September 2025, uses a new line inside a robots.txt group, such as "Content-Signal: ai-train=no, search=yes". The shared token takes the opposite approach. It "introduces no new directive names and no new grammar," and it never applies through the * group. Neither of the two Cloudflare pages I read mentions __usage_ai-training (checked Oct 31, 2026).

They can sit in the same file. A site that wants the clearest possible signal can carry both, as long as each line says the same thing.

Which AI companies honor the AI training opt-out, and from when?

Four companies have said they will support it, with different timing, and none of them honors the token yet. Microsoft gives the firmest date. Anthropic's statement came from the draft's author on the IETF mailing list. Every row below comes from the company's own page, read on Oct 31, 2026.

Anthropic's crawler help page, dated April 7, 2026, does not mention the token yet, so its statement so far is the author's list post. I will update this table as each company publishes details. OpenAI says it will share another update "with details about our implementation."

How many websites use __usage_ai-training yet?

Almost none, a day after launch. Of 76 big sites whose robots.txt I read on Oct 31, 2026, exactly one has a __usage_ai-training group: x.com, which disallows everything. Company-specific training controls are far more common: 33 of the same 76 files name GPTBot and 30 name Google-Extended.

How the sample was made: the 50 sites on Wikipedia's list of most-visited websites (as of September 2026) plus the 41 news, reference, retail and website-builder sites from an earlier survey. Five sites are on both lists, so 86 in all. Both lists were fixed before any file was read. Ten files could not be read, because the server did not answer or returned 403, 404, 405 or 417, so they are left out. Each file was parsed with the AI Crawler Checker's own RFC 9309 parser, which now reads the token the way the draft says, by named groups only. Five sites name Google-Extended but allow it, so the shared token would not change Google's reading for them. This is a convenience sample of large sites, not a measure of the web. I plan to re-run it monthly.

How do you add the AI training opt-out and check that it works?

Add the group, make your named crawler groups agree with it, and test the live file. The whole job takes a few minutes on most sites.

• Open yoursite.com/robots.txt and find out how it is edited: a file at the site root, a CMS setting, or a plugin.
• Add "User-Agent: __usage_ai-training" and "Disallow: /" as a group of its own, separated from the others by a blank line.
• Look for groups that name GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, meta-externalagent or CCBot. If any of them allows training, change it to Disallow: /. Google has said a named Google-Extended group wins over the token, and the others have not said.
• If you use NoArchive or NoCache to keep pages out of Bing's AI answers, add the ai-grounding tag to those pages first, since Bing ignores the old tags once it sees the new preferences.
• Leave search crawlers such as Googlebot, Bingbot and OAI-SearchBot alone, unless you also want out of search.
• Paste your domain into the AI Crawler Checker, which I built. It reads your live robots.txt and shows whether the token is set, and the line that decided each AI crawler.
• Check again after Feb 1, 2027, when Microsoft says it will start honoring the token.