Quick answer
What is ai.txt? It is a plain text file at a website's root that says whether the site's images, text and other media may be used to train AI models. Spawning, a startup, launched it in May 2023. No standards body has adopted it, and none of the AI crawler docs I checked on Nov 13, 2026 mention it. That day, 0 of 616 top domains that answered served a real ai.txt, while 119 served an llms.txt.
Key takeaways
- Spawning describes ai.txt as a file that "sets machine readable permissions for commercial text and data mining." It launched on May 30, 2023.
- A Spawning ai.txt looks like robots.txt: "User-Agent: *", then Disallow and Allow lines that name file types, such as "Disallow: *.jpg".
- The only Internet-Draft for ai.txt, filed June 12, 2026, "has no formal standing in the IETF standards process" and defines a different format. IANA has not registered the name.
- None of the crawler pages from OpenAI, Anthropic, Google, Perplexity, Meta, Apple or Common Crawl mention ai.txt, and neither does Microsoft's Bing post on training preferences (checked Nov 13, 2026). All of them point to robots.txt.
- Of the Tranco top 1,000 domains, 616 answered a request for /ai.txt on Nov 13, 2026. None served a real ai.txt. 119 of 616 served a real llms.txt.
- 91 of the 616 answered /ai.txt with an HTML page and a 200 status, mostly soft 404s plus a few bot challenges, that a careless check would count as a file.
On this page (13)
What is ai.txt?
What is ai.txt? It is a plain text file at a website's root that states whether the site's images, text, audio, video and code may be used to train AI models, a format the startup Spawning launched in May 2023.
That short answer comes from Spawning's own words. Its ai.txt page says: "An ai.txt file sets machine readable permissions for commercial text and data mining." Spawning's launch post, dated May 30, 2023, adds the key design choice: "Unlike robots.txt, which is commonly read when a website is crawled, ai.txt is read when a site's media is downloaded." The idea was simple. A dataset can hold links to your images for years, and a file checked at download time can still say no after the links were collected.
Nobody owns the name, though, and three different things now share it. The table sorts them out. For most readers, ai.txt means Spawning's file. That is the meaning this article uses unless it says otherwise.
| Entity | What it is | Where it lives |
|---|---|---|
| Spawning ai.txt (2023) | A list of file types that AI data miners may or may not use for training | /ai.txt at the site root (Spawning also allows /.well-known/) |
| Spawning | The company that made the format, an ai.txt generator and a WordPress plugin | spawning.ai |
| draft-car-ai-txt-wellknown (2026) | An individual Internet-Draft by Kayla Cardillo for a different, larger ai.txt with training, scraping and licensing fields | /.well-known/ai.txt |
| ai.txt DSL paper (2025) | A research proposal on arXiv by Yuekang Li and ten co-authors for a language that sets element-level rules for AI | A paper, not a deployed file |
| robots.txt | The crawl permission file, an IETF standard since RFC 9309 (September 2022) | /robots.txt |
| llms.txt | A Markdown index of a site's key pages for AI tools, proposed at llmstxt.org | /llms.txt |
Spawning tied the format to EU law from the start. Its page says ai.txt "adheres to the EU's Digital Single Market TDM Article 4 exception by providing a machine readable opt-out targeted at commercial AI model training." TDM means text and data mining. Article 4 of EU Directive 2019/790 lets rights holders reserve their works from it. Whether a court would treat an ai.txt file as a valid reservation is a legal question this article does not answer.
What does an ai.txt file look like?
A Spawning ai.txt file uses robots.txt-style lines: one User-Agent line, then Disallow and Allow lines that name file types, such as "Disallow: *.jpg" to refuse JPEG images for AI training.
No site in my sample served one, so the example comes from outside it. It is the Louvre's file at louvre.fr/ai.txt, which I fetched on Nov 13, 2026: 4,296 bytes of plain text. The two comment lines at the top are the ones Spawning's generator writes. The file refuses ten image types and allows text, audio, video and code. Lines are trimmed here. The full file lists 54 file types, 10 under Disallow and 44 under Allow, then "Allow: /" and a block of ASCII art in comment lines.
# Spawning AI # Prevent datasets from using the following file types User-Agent: * Disallow: *.bmp Disallow: *.gif Disallow: *.jpeg Disallow: *.jpg Disallow: *.png Disallow: *.webp Allow: *.txt Allow: *.pdf Allow: *.mp3 Allow: *.mp4 Allow: *.py Allow: /
- The generator is a set of toggles. Spawning's page says: "By default all content is opted out. Selecting allow for any content type will let data miners know that they may use content on your website of that media type."
- Spawning's page says to upload the file "to the root directory (/ or /.well-known/) of your website, alongside your robots.txt file."
- Spawning's test is simple: visit yourdomain.com/ai.txt, and "If you see the contents of the ai.txt file as plain text, it's in the right place."
- Spawning makes no promise that it works. Its terms say it "does not guarantee enforcement" and that "not all parties may choose to respect the guidelines." Its FAQ also says "many AI miners respect the rules set in ai.txt files," but it names none, and none of the AI crawler docs I checked say so.
The 2026 Internet-Draft uses a different format with the same name. Its minimal file has lines such as "Spec-Version: 1.0", "Site-Name: My Blog" and "Training: deny", plus optional Agent blocks with rate limits. It says the file "MUST be served at" /.well-known/ai.txt, not at the root. A parser built for one format will not read the other. So the answer to what is ai.txt depends on which spec a reader has in mind.
Is ai.txt an official standard?
No standards body has adopted ai.txt, and the only Internet-Draft for it, filed on June 12, 2026 by an independent author, has no formal standing at the IETF and expires on Dec 14, 2026.
The draft is draft-car-ai-txt-wellknown-00, titled "AI.TXT: A Declaration File for AI Usage Preferences, Licensing, and Policy." The IETF Datatracker lists it as an individual draft and says it "is not endorsed by the IETF and has no formal standing in the IETF standards process." It asks IANA to register ai.txt and ai.json as well-known URIs. On Nov 13, 2026, the IANA Well-Known URIs registry did not list either name. The registry was last updated Oct 26, 2026. The draft calls Spawning's file "An earlier file at "/ai.txt"" and describes its own format as "a strict superset" of it.
Part of the confusion over what is ai.txt comes from reuse of the name. Two other efforts used it without becoming standards. In September 2024, Guardian News and Media submitted a draft paper on "an AI.txt Protocol" to the IAB's AI-CONTROL workshop. In May 2025, eleven researchers posted "ai.txt: A Domain-Specific Language for Guiding AI Interactions with the Internet" to arXiv. Compare robots.txt: RFC 9309 is an IETF Standards Track document that "represents the consensus of the IETF community." The standards work on AI preferences now runs through the IETF AIPREF working group. Its attachment draft, version 05 from Aug 19, 2026, "defines attachment methods using the Robots Exclusion Protocol and HTTP header fields." A separate ai.txt file is not one of them.
Do AI crawlers read ai.txt?
None of the AI crawler documentation I checked on Nov 13, 2026, from OpenAI, Anthropic, Google, Perplexity, Meta, Microsoft, Apple or Common Crawl, mentions ai.txt, and each one points site owners to robots.txt instead.
I fetched each company's crawler page and searched the text for "ai.txt". The table gives the result, and the company's own words on how to control its AI crawlers. A page that does not mention ai.txt is not proof that a company ignores the file. It does mean no company has told site owners it reads one.
| Company and page (checked Nov 13, 2026) | Mentions ai.txt? | What it says to use |
|---|---|---|
| OpenAI: Overview of OpenAI crawlers | No | "OpenAI uses OAI-SearchBot and GPTBot robots.txt tags to enable webmasters to manage how their sites and content work with AI." |
| Anthropic: crawler help article | No | Its bots "respect "do not crawl" signals by honoring industry standard directives in robots.txt." |
| Google: Google's common crawlers (updated Oct 30, 2026) | No | For Google-Extended, "the robots.txt user-agent token is used in a control capacity." |
| Perplexity: Perplexity crawlers | No | "we recommend allowing PerplexityBot in your site's robots.txt file" |
| Meta: Meta web crawlers | No | "In order to block these crawlers, add a disallow for the relevant crawler to robots.txt." |
| Microsoft: Bing Blog, training and grounding preferences (Oct 30, 2026) | No | "AI Training preferences in robots.txt apply to the relevant site or path" |
| Apple: About Applebot (published Sep 4, 2026) | No | Opt out of training "by disallowing Applebot-Extended in the robots.txt file." |
| Common Crawl: CCBot | No | "To prevent Common Crawl from crawling your website, include the following in your robots.txt" |
Microsoft's Bing crawler help page loads its text with JavaScript, and my plain fetch got only its title. The Microsoft row uses the Bing Blog post instead. Google comes closest to the topic. Its guide to generative AI features, last updated Jul 10, 2026, says "You don't need to create new machine readable files, AI text files, markup, or Markdown to appear in Google Search (including its generative AI capabilities), as Google Search itself doesn't use them." The guide names llms.txt as its example, not ai.txt. The wording still covers files like it.
Spawning's route to AI companies was never the crawlers. Its 2023 launch post said "Spawning's API will communicate the permissions set by ai.txt files to our growing network of AI researchers and partners, including Hugging Face and Stability AI." On Nov 13, 2026, spawning.ai, spawning.ai/api and haveibeentrained.com all showed an "Under Maintenance" page. Spawning's Python package for model trainers, datadiligence, was last released on Oct 9, 2024. It lists three opt-out methods it supports. The Spawning API is the first, and ai.txt is not named.
How is ai.txt different from robots.txt?
robots.txt tells crawlers which URLs they may fetch, under an IETF standard, RFC 9309, while ai.txt only states whether downloaded media may be used for AI training, and no crawler is bound to read it.
RFC 9309 describes robots.txt as a way "for service owners to control how content served by their services may be accessed, if at all, by automatic clients known as crawlers." Access is the key word. A Disallow line in robots.txt asks a crawler not to request a URL at all. A Disallow line in ai.txt asks a data miner not to use a file it may already hold. Spawning made that split on purpose: its launch post says that with ai.txt, site owners "can continue to use robots.txt to manage permissions for popular search engines."
The two files also match differently. robots.txt rules match URL paths, so "Disallow: /images/" covers one folder. Spawning's ai.txt rules match file types, so "Disallow: *.png" covers every PNG on the site. And robots.txt names crawlers one by one, such as GPTBot or CCBot. A Spawning ai.txt usually has a single "User-Agent: *" group for all miners.
What is the difference between ai.txt and llms.txt?
ai.txt states permissions for AI training, while llms.txt is a Markdown index that points AI tools to a site's most useful pages, so one file restricts use and the other invites reading.
The llms.txt proposal at llmstxt.org asks for "An H1 with the name of the project or site," and says "This is the only required section." After it come a short summary and lists of links. Nothing in llms.txt grants or refuses anything. People who compare ai txt vs llms txt are often deciding which one to publish. The two do different jobs, so a site can publish both, either or neither. The llms.txt vs robots.txt guide covers how the index sits next to crawl rules. The companion llms-full.txt file is a separate convention again.
| File | Job | Format | Standing (Nov 13, 2026) | Read by |
|---|---|---|---|---|
| robots.txt | Crawl permission per URL path | User-agent groups with Allow and Disallow paths | IETF standard, RFC 9309 | Every major crawler named above |
| ai.txt (Spawning) | AI training permission per file type | User-Agent: * with Disallow and Allow file patterns | Company format; an unrelated individual draft has no standing | Not mentioned by any crawler page above |
| llms.txt | An index of key pages for AI tools | Markdown: H1, summary, link lists | Proposal at llmstxt.org | Not named by the crawler pages above; Google Search says it does not use such files |
| __usage_ai-training token | AI training preference | A group inside robots.txt | Individual Internet-Draft, Oct 30, 2026 | None yet: Microsoft from Feb 1, 2027; Google aims for 2027; OpenAI and Anthropic have announced support |
Is ai.txt the same as the __usage_ai-training token in robots.txt?
No: __usage_ai-training is a policy token that goes inside robots.txt, from an Oct 30, 2026 Internet-Draft, and Microsoft, Google, OpenAI and Anthropic have each announced plans to support it, which no AI company has said about ai.txt.
Both state a training preference, but in different places. The token is a robots.txt group with "User-Agent: __usage_ai-training" and "Disallow: /". Its draft says "This policy token is not a directive for crawling behavior." So it does not block crawling either. Microsoft's Bing post says "Starting February 1, 2027, Microsoft will honor the new preferences as it crawls." Google's post says "We aim to support this new control along with other AI providers in 2027." OpenAI says it will follow the signal "in the coming months," and the draft's author, Kevin Kelley of Anthropic, wrote that Anthropic "has already started implementing support for this draft." None of the four honors the token yet. The guide to the __usage_ai-training token covers its syntax, the other companies' positions, and how it interacts with GPTBot and Google-Extended.
Want your pages kept out of AI training? Today, the documented opt-outs are the named training crawlers in robots.txt, such as GPTBot, ClaudeBot and Google-Extended. The __usage_ai-training token is promised, not live: Microsoft starts on Feb 1, 2027, and Google aims for 2027. An ai.txt file can sit beside them, as long as it says the same thing.
How many top websites publish an ai.txt file?
None of the Tranco top 1,000 domains served a real ai.txt file when I checked on Nov 13, 2026: 616 answered the request for /ai.txt, and 0 of them returned a plain text file there.
The sample is the same one I used for the Organization schema sameAs study two days earlier: the top 1,000 domains of Tranco list Y895G, with no swaps. Tranco ranks domains, not websites, and 384 of the 1,000 got no request or gave no answer. 323 had no reachable robots.txt, 225 of them because of DNS failures, many on CDN and ad-serving names with no site at all. 8 had a robots.txt that answered with a server error or a 429. 45 disallow crawlers in robots.txt, among them facebook.com, x.com, reddit.com and netflix.com. And 8 timed out, dropped the connection or redirected more than 5 times. That leaves 616 that answered. For the practical side of what is ai.txt, the result is blunt: on the biggest sites, nobody serves one.
| Step (Tranco top 1,000, checked Nov 13, 2026) | ai.txt | llms.txt |
|---|---|---|
| Domains in the sample | 1,000 | 1,000 |
| Skipped: robots.txt unreachable (225 of these were DNS failures) | 323 | 323 |
| Skipped: robots.txt answered 5xx or 429 | 8 | 8 |
| Skipped: robots.txt disallows my crawler or all crawlers | 45 | 46 |
| No answer, or more than 5 redirects | 8 | 7 |
| Answered with an HTTP status | 616 | 616 |
| Answered 404 | 408 | 306 |
| Answered 403 | 82 | 79 |
| Answered 200 | 103 | 212 |
| 200, but an HTML page (mostly soft 404s) | 91 | 83 |
| 200, but another page, a JSON error, an IP address, or a compressed body | 12 | 10 |
| Real file: 200, plain text or Markdown, not HTML, not empty | 0 of 616 | 119 of 616 (19.3%) |
The overlap is empty, since no domain served an ai.txt. 618 domains answered at least one of the two requests, and 614 answered both. One more domain, bilibili.com, answered /llms.txt with a compressed text/plain body that my check could not read. It is not counted in the 119. Ten adult domains in the sample are flagged in the CSV and kept in every count. None of them served either file.
ai.txt and llms.txt check, top 1,000 domains, Nov 13, 2026 (CSV)
One row per Tranco domain: result, status, final scheme and host and content type for each request, bytes for each file served, the robots.txt result, and whether llms.txt opens with an H1. Final hosts that point to a country or regional edition are withheld and marked. Free to reuse with a link to this page.
How many top websites publish llms.txt instead?
119 of the 616 top domains that answered served a real llms.txt on Nov 13, 2026, about one in five, and 113 of those 119 files opened with the Markdown H1 that the llms.txt proposal requires.
Anyone asking what is ai.txt after hearing about llms.txt should know which file the big sites chose. The publishers are mostly software, security and commerce companies. Among them are cloudflare.com, github.com, stripe.com, shopify.com, paypal.com, dropbox.com, adobe.com, nvidia.com, slack.com, atlassian.com, target.com, etsy.com, weather.com and foxnews.com. 115 of the 119 files came as text/plain and 4 as text/markdown. The median file was 12,879 bytes. The CSV withholds one file's size, so the median of the other 118 there is 12,760.5 bytes. The smallest, tailscale.com's, was 415 bytes. Two, sourceforge.net's and twilio.com's, were larger than the 2 MB my crawler reads.
Six files skipped the H1. cisco.com's file starts with its own URL. salesforce.com's and twilio.com's start with a list of links. agora.io's and trendyol.com's start with an H2. playstation.com's starts with a plain sentence naming the tool that made it. The proposal allows an optional byte-order mark before the H1, and six files had one. I counted those as starting with the H1.
Why does a request for ai.txt sometimes return a web page?
Many sites answer any unknown URL with an ordinary web page and a 200 status, called a soft 404, and some answer bots with a challenge page, so a careless check would credit 91 of the 616 top domains with an ai.txt file that does not exist.
Among them were spotify.com, tiktok.com, aliexpress.com, duckduckgo.com and claude.ai. Each sent an HTML page with status 200 for /ai.txt, often after a redirect to a home page or a "page not found" page. Not all 91 were soft 404s. At least six were bot checks or sign-in pages: linkedin.com and lemonde.fr sent a browser check, walmart.com a page titled "Robot or human?", ups.com and certcenter.com an access-denied page, and office.com a sign-in page. 81 domains did the same for both files. Spawning's own domain does it too: on Nov 13, 2026, spawning.ai answered /ai.txt with its HTML maintenance page and a 200, and the older site.spawning.ai answered with a 404.
Redirects did most of this. 78 of the 103 domains that answered /ai.txt with a 200 ended at a URL other than /ai.txt, and 68 of the 91 HTML pages got there that way. Ten more redirected to a page my HTML test missed. Eight landed on a page served in Markdown, such as workers.dev landing on a Cloudflare product page. That was mostly content negotiation: my request listed text/markdown in its Accept header, and a recheck with the AI Crawler Checker's header got HTML from workers.dev and agora.io, though vercel.app still sent Markdown. One landed on a 1-pixel GIF, and 163.com sent a compressed HTML home page. One domain is an IP address echo service, so it answered every path with my own address. Another sent a JSON error with a 200. So check four things before you trust a file: the final status, the final URL, the Content-Type, and the first line of the body.
Does jnabear.com have an ai.txt file?
No: on Nov 13, 2026, jnabear.com answered /ai.txt with a 404 and /llms.txt with a 15,795-byte plain text file that opens with the H1 "# JNAbear".
That is one data point, not a benchmark, and this site is not in the Tranco top 1,000. The choice is on purpose. This site allows training crawlers and answer crawlers in robots.txt, because being read and cited is the point here. It also publishes an llms.txt so AI tools can find the main pages. An ai.txt would add a third file that no crawler documents reading. To see what your own robots.txt allows each AI crawler, try the free AI Crawler Checker. It reads the live file and shows the line that decides each bot. It does not read ai.txt.
Should you add an ai.txt file to your site?
Add one only as an extra signal: an ai.txt file costs a few minutes and does no harm, but no major AI company I checked says it reads one, so the training opt-outs that are documented today all live in robots.txt.
- If you want out of AI training, start in robots.txt: a Disallow group for each training crawler you care about, such as GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, meta-externalagent and CCBot, plus the __usage_ai-training group.
- Add a Spawning ai.txt if you also want to tell dataset tools that check it, and make it agree with robots.txt. A file that allows images while robots.txt blocks every training bot sends mixed signals.
- Put it at yourdomain.com/ai.txt and serve it as text/plain. Then open it in a browser and confirm you see the lines, not a web page.
- Make sure a missing file returns a 404. If your site answers unknown URLs with a 200 page, every checker will think you have an ai.txt, and an llms.txt, that you do not.
- Do not expect search effects either way. Spawning says the file "does not impact traditional search engine crawlers or your website's SEO."
- Watch the IETF work. If the 2026 draft or an AIPREF document adopts a /.well-known/ai.txt format, the file you write today may need a new location and new fields.
So here is the practical answer to what is ai.txt for a site owner in late 2026. It is a polite note to AI data miners that no major crawler has promised to read. If you publish one, keep it consistent with robots.txt. Keep robots.txt as the file that does the work.
How was the ai.txt check run?
I requested https://domain/ai.txt and https://domain/llms.txt once each for the Tranco top 1,000 domains on Nov 13, 2026, obeying robots.txt, following up to 5 redirects, with a 20 second timeout.
This check answers a narrower question than what is ai.txt: who actually serves one. The sample is the top 1,000 domains of Tranco list Y895G, generated on Nov 10, 2026. They are the same 1,000 domains as the sameAs study, in the same order. I wrote the list to a file before the first request and made no swaps. Every request used the user agent JNAbear-AICrawlerCheck/1.0, the one the AI Crawler Checker sends. It never imitates Googlebot or a browser. The Accept header differed from the checker's. The checker sends "text/plain,*/*;q=0.5", and so did my robots.txt requests. For /ai.txt I sent "text/plain,text/markdown,*/*;q=0.5", and for /llms.txt "text/markdown,text/plain,*/*;q=0.5". Eight domains were checked at a time, and each domain got its two requests one after the other.
Before each request, I read robots.txt for every host in the redirect chain. If the rules for my crawler, or for all crawlers, disallowed the path, I skipped it. A robots.txt that answered 5xx or 429 counted as a full disallow. So did one that could not be reached because of DNS, timeout, TLS or connection errors, as RFC 9309 requires for unreachable files. Some TLS failures came from my crawler's client, not the site: Python could not connect to huawei.com or flashtalking.com, and a recheck with curl fetched both robots.txt files with a 200. Any other 4xx counted as no rules. gosuslugi.ru was the one domain whose rules allowed /ai.txt and disallowed /llms.txt.
A file counted as real only if it passed five tests. The final status was 200. The Content-Type was text/plain or text/markdown, or missing with a plain text body. The body was not empty. The first 4 KB held no "<html" or "<!doctype". And the final URL still ended in /ai.txt or /llms.txt. Short bodies that said "not found" would have been dropped, and none turned up. For an ai.txt, I also looked for User-Agent lines and file-type patterns. For an llms.txt, I checked whether the first non-blank text was "# ". I read up to 2 MB of each body. The check did not request /.well-known/ai.txt, the draft's location. A site that serves only that path is not counted. Results can differ by country, by user agent and by day. 82 domains answered /ai.txt with a 403, which often points to a bot rule rather than a missing file.
Sources
- Spawning: Spawning's ai.txt Generator (page and FAQ) (opens in a new tab) (verified Nov 2026)
- Spawning Blog: ai.txt: A new way for websites to set permissions for AI (Cullen Miller, May 30, 2023) (opens in a new tab) (verified Nov 2026)
- PyPI: datadiligence 0.1.7 (Spawning, released Oct 9, 2024) (opens in a new tab) (verified Nov 2026)
- IETF Datatracker: draft-car-ai-txt-wellknown-00, AI.TXT: A Declaration File for AI Usage Preferences, Licensing, and Policy (June 12, 2026) (opens in a new tab) (verified Nov 2026)
- IANA: Well-Known URIs registry (updated Oct 26, 2026) (opens in a new tab) (verified Nov 2026)
- IETF Datatracker: Guardian News and Media draft paper on an AI.txt Protocol (IAB AI-CONTROL workshop, Sep 2024) (opens in a new tab) (verified Nov 2026)
- arXiv: ai.txt: A Domain-Specific Language for Guiding AI Interactions with the Internet (Li et al., May 2, 2025) (opens in a new tab) (verified Nov 2026)
- RFC 9309: Robots Exclusion Protocol (opens in a new tab) (verified Nov 2026)
- llmstxt.org: The /llms.txt file, v2 (opens in a new tab) (verified Nov 2026)
- IETF: Machine-Readable Training Preference Signal (draft-kelley-aipref-training-preference-signal-00, Oct 30, 2026) (opens in a new tab) (verified Nov 2026)
- OpenAI: Making it easier for websites to express AI training preferences (Oct 30, 2026) (opens in a new tab) (verified Oct 2026)
- IETF ai-control list: New draft: draft-kelley-aipref-training-preference-signal-00 (Kevin Kelley, Anthropic, Oct 30, 2026) (opens in a new tab) (verified Nov 2026)
- OpenAI: Overview of OpenAI crawlers (opens in a new tab) (verified Nov 2026)
- Anthropic: Does Anthropic crawl data from the web, and how can site owners block the crawler? (opens in a new tab) (verified Nov 2026)
- Google: Google's common crawlers (updated Oct 30, 2026) (opens in a new tab) (verified Nov 2026)
- Google Search Central: Optimizing your website for generative AI features on Google Search (updated Jul 10, 2026) (opens in a new tab) (verified Nov 2026)
- Google Search Central Blog: Supporting a shared AI training control for websites (Oct 30, 2026) (opens in a new tab) (verified Nov 2026)
- Perplexity: Perplexity crawlers (opens in a new tab) (verified Nov 2026)
- Meta for Developers: Meta web crawlers (opens in a new tab) (verified Nov 2026)
- Bing Blog: Your content, your choice: clearer preferences for training and grounding (Oct 30, 2026) (opens in a new tab) (verified Nov 2026)
- Apple: About Applebot (published Sep 4, 2026) (opens in a new tab) (verified Nov 2026)
- Common Crawl: CCBot (opens in a new tab) (verified Nov 2026)
- Tranco: list Y895G (generated Nov 10, 2026) (opens in a new tab) (verified Nov 2026)
- IETF: Associating AI Usage Preferences with Content in HTTP (draft-ietf-aipref-attach-05, Aug 19, 2026) (opens in a new tab) (verified Nov 2026)
Topics & entities in this article
Frequently asked questions
ai.txt tells AI data miners whether a site's images, text, audio, video and code may be used to train AI models. Spawning launched the format in May 2023. A Spawning ai.txt lists file types under Disallow and Allow, such as "Disallow: *.jpg" to refuse JPEG images.
Neither says so. OpenAI's and Google's crawler pages do not mention ai.txt (checked Nov 13, 2026). OpenAI points to GPTBot and OAI-SearchBot in robots.txt, and Google uses the Google-Extended token in robots.txt to control AI training.
No. ai.txt states whether content may be used for AI training. llms.txt is a Markdown index of a site's key pages for AI tools, and it grants or refuses nothing. On Nov 13, 2026, 119 of 616 top domains that answered served llms.txt, and none served ai.txt.
Spawning says to put ai.txt in the root directory, "/ or /.well-known/", next to robots.txt, so it loads at yourdomain.com/ai.txt. A separate 2026 Internet-Draft with a different format says /.well-known/ai.txt. Serve it as text/plain.
No. Spawning says the file "does not impact traditional search engine crawlers or your website's SEO." Search crawlers follow robots.txt, and Google says Google Search does not use AI text files.
No. The only Internet-Draft, draft-car-ai-txt-wellknown-00 from June 12, 2026, "has no formal standing in the IETF standards process" and expires Dec 14, 2026. IANA had not registered ai.txt as a well-known URI as of Nov 13, 2026.
Very few large ones. On Nov 13, 2026, none of the Tranco top 1,000 domains served a real ai.txt. 616 answered the request, 408 with a 404, and 91 with an HTML page and a 200 status, mostly soft 404s.
Related service
Web Design & SEO
Web design and SEO in one project: custom WordPress websites for businesses that need a new or rebuilt site.
Disclosure: this is a service I sell.