What Is llms.txt? A Practical Guide to the AI Crawler File

What is llms.txt?

llms.txt is a plain Markdown file that sits at the root of a website. It tells AI systems what the site is, who runs it, and which pages matter most. Jeremy Howard, co-founder of Answer.AI, proposed the format in September 2024. The full spec lives at llmstxt.org. Instead of making an AI crawler figure out your site from menus and links, you hand it a guided tour.

The format is simple by design. One H1 names the site. A blockquote sums it up. Short sections list your key pages with one-line notes. Any text editor can produce the file, and any language model can read it.

llms.txt is a Markdown file at a site's root that gives AI crawlers a curated summary of the site and annotated links to its most important pages.

Why does the file exist?

llms.txt exists to solve two problems. First, language models read pages as raw text, and real HTML is messy. Menus, scripts, cookie banners, and footers crowd out the actual content. Models also have limited working memory, called a context window. Every word spent on clutter is a word not spent understanding your business. A clean summary fixes that.

Second, it lets you control the story. When an AI assistant describes your company, it builds that answer from whatever it found. This file is the one place where you say, in your own words, what your business is and who it serves. In AI search, the site that defines itself clearly beats the site that leaves it to guesswork.

llms.txt vs robots.txt vs sitemap.xml

The three files answer three different questions. They work together, not against each other.

• robots.txt answers "may you crawl this?" It grants or denies access to crawlers, including AI agents like GPTBot.
• sitemap.xml answers "what exists?" It lists every URL with last-modified dates so crawlers miss nothing.
• llms.txt answers "what matters, and what does it mean?" It curates the short list and adds context.

Permission, inventory, meaning. A well-run site publishes all three. For the deeper llms.txt vs robots.txt comparison, including where agents.md and MCP fit, I keep a dedicated guide.

What goes inside an llms.txt file?

The spec asks for an H1 with the site name, a blockquote with a one-paragraph summary, optional detail lines, and then sections of Markdown links with short descriptions. Here is a compact example for a fictional plumbing company:

# Acme Plumbing

> Acme Plumbing is a family-run plumbing company serving Portland since 1998. We handle residential repairs, remodels, and 24/7 emergencies.

Licensed and bonded in Oregon. Contact: hello@acmeplumbing.com.

## Services

- [Emergency Plumbing](https://acme.com/emergency): 24/7 response, flat call-out fee
- [Kitchen Remodels](https://acme.com/remodels): design-to-install renovations

## About

- [Our Story](https://acme.com/about): three generations of Portland plumbers

A related convention, llms-full.txt, goes further. It packs the full text of the site into one file for models that want depth instead of a summary. Most small sites do not need it. The summary file carries most of the value. To see how far the format stretches in practice, I compared real llms.txt examples from Anthropic, Stripe, Zapier, and Supabase in a separate breakdown.

Does llms.txt actually work?

For Google Search, llms.txt does nothing: Google Search does not use the file, which is the honest answer most guides skip. Google's own guidance for its AI features says "You don't need to create new machine readable files, AI text files, or markup to appear in these features." Its newer guide to generative AI features, updated in July 2026, says of maintaining llms.txt files: "Doing so will neither harm nor help your site's visibility or rankings in Google Search, as Google Search ignores them." Anyone promising that the file will boost your rankings is selling something.

Outside Google Search, the best data so far says the file is rarely fetched at all. Ahrefs studied May 2026 traffic on 137,210 domains in its web analytics product, found about 38,000 with a valid llms.txt, and reported that "97% of those files received zero traffic in May 2026." Some AI agents and training crawlers do fetch the file, but the bots that retrieve pages to answer questions barely register: "AI retrieval bots account for just 1.1% of AI bot requests." Ahrefs adds two limits. Its customers "skew more technical and SEO-aware than the web at large," and a fetch is not proof the file was read, so it calls every figure "a ceiling on actual llms.txt consumption."

So why bother? Three reasons. Adoption is growing on the AI side: documentation platforms generate the file automatically, and AI companies, including Anthropic, publish one for their own docs. The cost is close to zero: one file, a few minutes, no code. And it is a cheap bet with real upside. If AI assistants lean on the file more over time, early adopters win the framing. If they never do, you lost fifteen minutes.

llms.txt is not a ranking factor. It is a low-cost, low-risk way to control how the AI systems that do read it summarize your site.

What happened when I added llms.txt to this site

I practice what I write about, so this site publishes its own file at jnabear.com/llms.txt. Writing it took about fifteen minutes: the business summary, the six services with starting prices, every article, and the free tools. I also left a note in robots.txt pointing crawlers to it.

Two honest observations. First, publishing the file is the easy part; getting it read is not. In the Ahrefs study above, 97 percent of the roughly 38,000 valid llms.txt files drew no requests at all in May 2026, and Ahrefs, whose sample skews technical, calls every figure in the study a ceiling on how much the file is actually consumed. Second, the exercise itself had value I did not expect. Forcing your whole business into one clear paragraph is a positioning drill. If you cannot write that paragraph, no AI summary of your site will be sharp either.

One practical habit: I update the file whenever prices or services change. A stale file feeds AI systems old facts in your own voice, which is worse than silence.

How to create an llms.txt file in five minutes

• Write a one-paragraph summary: what the site is, who runs it, what it offers.
• List the 10 to 30 pages that best represent you, with a one-line description each.
• Group them into sections a stranger would understand: Services, Guides, About.
• Produce the Markdown by hand, or use a free llms.txt generator to enforce the format.
• Upload the file to your web root so it resolves at yoursite.com/llms.txt.
• Update it whenever pricing, services, or key facts change.

Test it the way a crawler reads it: open yoursite.com/llms.txt in a browser and read it cold. If a stranger could describe your business accurately from that file alone, it works. For platform-specific steps, I have a separate guide on how to add llms.txt to WordPress, Shopify, and static sites.

Should you add one to your site?

Most sites should add one, because it costs minutes and cannot hurt. It is the cheapest move in the wider game of answer engine optimization. But keep it in proportion. The file is one small layer, and it will not rescue a page that is not ranking for deeper reasons. AI search visibility is earned by the bigger stack: entity-rich content, schema markup, quotable definitions, and genuine topical authority. The file introduces your site; the content still has to deserve the citation.

I publish one for this site, generate them for clients, and built a free llms.txt generator so you can make your own. That is about as honest as an endorsement gets: small bet, small cost, upside if AI search keeps growing the way it has.