Skip to content

AI Crawler Checker

Enter a website to see which AI crawlers its robots.txt allows: 17 crawlers and tokens from OpenAI, Anthropic, Perplexity, Google, Apple, Meta and Common Crawl, with the exact rule behind each answer. Free, no signup, and there is a JSON API.

How do you check which AI crawlers a site blocks?

You read the site's robots.txt the way each crawler does: find the group that names the crawler, fall back to the * group if none does, and apply the longest matching rule. This checker does that for every crawler below, following RFC 9309, and shows the rule that decided each one.

Deciding what to allow? Read the guide: robots.txt for AI crawlers.

(this site). The address you enter is sent to jnabear.com, which fetches that site's public robots.txt. Nothing else is stored.

Which crawlers does it check?

It checks 17 crawlers and tokens, grouped by what each operator says it is for. Each row links to the operator's own page, read on Oct 30, 2026.

Search: OAI-SearchBot, Claude-SearchBot, PerplexityBot, Googlebot, Applebot, meta-webindexer
Crawls so an AI search or answer product can show and link your site.
Training: GPTBot, ClaudeBot, meta-externalagent
Crawls for, or decides on, use in training AI models.
User-triggered: ChatGPT-User, Claude-User, Perplexity-User, meta-externalfetcher
Fetches a page because a person asked. Several operators say robots.txt may not apply.
Control token: Google-Extended, Applebot-Extended, __usage_ai-training
Sends no requests of its own. It controls how another crawler's data may be used.
Archive: CCBot
Crawls for a public web archive.

Seeing Meta's crawlers in your logs? What meta-externalagent is, and which of Meta's five crawlers to block, with a survey of 37 big sites' robots.txt files.

Want out of AI training but still in AI search? How the robots.txt AI training opt-out works, from GPTBot and ClaudeBot to Google-Extended and the shared __usage_ai-training token.

What it can't see

  • Firewalls and bot-blocking services. Cloudflare and others can block a crawler that robots.txt allows.
  • Whether a crawler obeys. Some user-triggered fetchers say robots.txt may not apply to them.
  • Page-level rules such as a noindex meta tag or an X-Robots-Tag header. Those control indexing, not crawling.

Is there an API?

Yes, and it is free. Send a GET request with a domain or page URL. The answer is JSON, any site may call it from the browser, and each site's robots.txt is cached for an hour. Please keep to 20 requests a minute; more returns HTTP 429. Each crawler's result is allowed or blocked, or not_set for the __usage_ai-training policy token when no group names it.

curl "https://jnabear.com/api/ai-crawlers?url=example.com/blog/post"
{
  "url": "https://example.com/blog/post",
  "robots_url": "https://example.com/robots.txt",
  "path": "/blog/post",
  "status": "ok",
  "http_status": 200,
  "summary": { "allowed": 12, "blocked": 4 },
  "crawlers": [
    {
      "token": "GPTBot",
      "operator": "OpenAI",
      "kind": "training",
      "result": "blocked",
      "rule": "Disallow: /",
      "applies": "Its own group",
      "rules": ["Disallow: /"],
      "purpose": "…",
      "note": null,
      "source": "https://developers.openai.com/api/docs/bots"
    }
  ],
  "sitemaps": [],
  "warnings": [],
  "fetched_at": "2026-10-30T12:00:00+00:00"
}

That example shows the shape of an answer, not a real site. status is ok, unavailable (a 4xx: crawlers may fetch anything) or unreachable (a 5xx or no answer: crawlers must assume everything is blocked). Bad input returns HTTP 422 with an error message.

Enter your domain above. The checker fetches your robots.txt and reports, for each of 17 AI crawlers and tokens, whether it may fetch the page you entered and which rule decided it. It follows RFC 9309, the robots.txt standard.

Not always. robots.txt only states rules. A firewall or a bot-blocking setting, such as Cloudflare's AI bot controls, can block a crawler that robots.txt allows. This checker reads robots.txt only, so check those settings separately.

Under RFC 9309, if robots.txt returns a 4xx status such as 404, crawlers may fetch anything. If it returns a 5xx error or cannot be reached, crawlers must assume the whole site is disallowed until they can fetch it.

Apple says that if robots.txt does not mention Applebot but does mention Googlebot, Applebot follows the Googlebot instructions. The checker applies that rule and labels it in the result.

Yes. Send a GET request to https://jnabear.com/api/ai-crawlers?url= followed by a domain or page URL. It returns JSON, allows requests from any site, and is limited to 20 requests a minute per visitor. Results are cached for an hour.

Get a free 15-minute audit of your top 3 ranking gaps

The free audit is a quick teardown of where you are leaking rankings and the first cluster I would build. No pitch deck, no obligation.

What happens next

  1. Send your site and what you want to rank for
  2. I reply within 24 hours
  3. A 15-minute teardown of your three biggest ranking gaps
Get your free audit Read the FindMyPlace.co case study