robots.txt for AI Crawlers: Which Bots to Allow, and What Blocking Costs

What does robots.txt control for AI crawlers?

Every well-behaved crawler checks one plain-text file before it fetches pages, and that file decides which URLs it may request. The file sits at the root of your site, and the major AI companies' crawlers follow it. It controls crawling, not what an engine already knows about you, and it only works on crawlers that choose to follow it.

The important change in the last two years is that AI companies split their crawling by purpose. One crawler fetches pages so an AI search product can cite them. Another collects content that may be used to train models. A third fetches a page on demand when a user asks the assistant to read it. Each has its own name, so robots.txt can treat each purpose differently.

Blocking a training crawler and blocking a search crawler are different decisions with different costs. Make them separately.

Which AI crawlers should you know?

OpenAI, Anthropic, Perplexity, Google, and Apple each document their crawlers, and the table below describes each one in the company's own terms where possible. Every entry was checked against that documentation in September 2026.

Two of those are not crawlers at all. Google says Google-Extended "doesn't have a separate HTTP request user agent string"; crawling is done by Google's existing crawlers, and the token only controls use. Apple says the same of Applebot-Extended: it "does not crawl webpages" and "is only used to determine how to use the data crawled by the Applebot user agent."

Should you block AI training crawlers?

Whether to block training crawlers depends on what your content is for. For most business sites, the goal is to be found, understood, and recommended, and the pages that explain your services exist to be read. Blocking training use of those pages costs you little and gains you little. For publishers whose content is the product, the calculation is different: training use can substitute for a visit, and declining it is a reasonable business decision.

What matters is not mixing the two up. A site that blocks everything from an AI company in one rule, to keep its content out of training, also leaves that company's search product. OpenAI is explicit about the cost: sites opted out of OAI-SearchBot "will not be shown in ChatGPT search answers." Google and Apple are explicit in the other direction: their training tokens do not affect search.

This site allows both search and training crawlers, on purpose. Its content exists to be cited, and the robots.txt file says so in a comment. Disclosure: getting cited is what I sell, so weigh that as you decide.

What does a good robots.txt for AI look like?

A good file names the crawlers you care about, keeps the rules you already have, and is short enough to check at a glance. The version below allows every search and answer crawler, allows the training crawlers too, and keeps two private paths closed to all of them.

User-agent: *
User-agent: Googlebot
User-agent: Bingbot
User-agent: OAI-SearchBot
User-agent: ChatGPT-User
User-agent: Claude-SearchBot
User-agent: Claude-User
User-agent: PerplexityBot
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
Allow: /
Disallow: /admin
Disallow: /cart

Sitemap: https://example.com/sitemap.xml

To opt out of training while staying in AI search, move GPTBot, ClaudeBot, Google-Extended, and Applebot-Extended into a second group with "Disallow: /", and leave the search crawlers where they are.

Why are all the bots in one group?

Because a crawler follows only the most specific group that names it. A group that says "User-agent: GPTBot" with just "Allow: /" does not inherit the Disallow lines in the * group, so GPTBot would crawl the paths you meant to close. This site had exactly that bug until September 2026. Listing named bots in the same group as *, or repeating every Disallow line in each named group, avoids it.

What mistakes do sites make most?

• Blocking search while meaning to block training: one "Disallow: /" for every AI company removes you from their search answers too.
• Named groups that silently drop your Disallow lines, as described above.
• A leftover staging rule, "Disallow: /" under "User-agent: *", shipped with a new site.
• Blocking at the network edge without knowing it. Since July 1, 2025, Cloudflare asks every new domain at sign-up whether to allow AI crawlers, and a block there happens before robots.txt is ever read.
• Expecting robots.txt to remove pages from an index. It controls crawling. A blocked URL can still be indexed if other pages link to it.

How do you check what your robots.txt allows?

Start by reading the file at yoursite.com/robots.txt and listing every group. For each crawler that matters to you, find the group that names it, or the * group if none does, and read only that group's rules. Then check the layers above the file: your CDN or firewall settings for AI crawler blocking, and your server logs for requests from the crawlers you expect. Google Search Console's robots.txt report shows how Google itself reads the file.

Finally, look at what the AI products actually do. Ask ChatGPT, Perplexity, and Claude questions your customers ask and see whether your pages are cited. A crawler you allow is only the first requirement for an AI citation; the page still has to be the clearest answer.