What Is llms.txt? A Practical Guide to the AI Crawler File

What is llms.txt?

llms.txt is a plain Markdown file that sits at the root of a website. It tells AI systems what the site is, who runs it, and which pages matter most. Jeremy Howard, co-founder of Answer.AI, proposed the format in September 2024. The full spec lives at llmstxt.org. Instead of making an AI crawler figure out your site from menus and links, you hand it a guided tour.

The format is simple by design. One H1 names the site. A blockquote sums it up. Short sections list your key pages with one-line notes. Any text editor can produce the file, and any language model can read it.

llms.txt is a Markdown file at a site's root that gives AI crawlers a curated summary of the site and annotated links to its most important pages.

Why does the file exist?

It solves two problems. First, language models read pages as raw text, and real HTML is messy. Menus, scripts, cookie banners, and footers crowd out the actual content. Models also have limited working memory, called a context window. Every word spent on clutter is a word not spent understanding your business. A clean summary fixes that.

Second, it lets you control the story. When an AI assistant describes your company, it builds that answer from whatever it found. This file is the one place where you say, in your own words, what your business is and who it serves. In AI search, the site that defines itself clearly beats the site that leaves it to guesswork.

llms.txt vs robots.txt vs sitemap.xml

The three files answer three different questions. They work together, not against each other.

• robots.txt answers "may you crawl this?" It grants or denies access to crawlers, including AI agents like GPTBot.
• sitemap.xml answers "what exists?" It lists every URL with last-modified dates so crawlers miss nothing.
• llms.txt answers "what matters, and what does it mean?" It curates the short list and adds context.

Permission, inventory, meaning. A well-run site publishes all three. For the deeper llms.txt vs robots.txt comparison, including where agents.md and MCP fit, I keep a dedicated guide.

What goes inside an llms.txt file?

The spec asks for an H1 with the site name, a blockquote with a one-paragraph summary, optional detail lines, and then sections of Markdown links with short descriptions. Here is a compact example for a fictional plumbing company:

# Acme Plumbing

> Acme Plumbing is a family-run plumbing company serving Portland since 1998. We handle residential repairs, remodels, and 24/7 emergencies.

Licensed and bonded in Oregon. Contact: hello@acmeplumbing.com.

## Services

- [Emergency Plumbing](https://acme.com/emergency): 24/7 response, flat call-out fee
- [Kitchen Remodels](https://acme.com/remodels): design-to-install renovations

## About

- [Our Story](https://acme.com/about): three generations of Portland plumbers

A related convention, llms-full.txt, goes further. It packs the full text of the site into one file for models that want depth instead of a summary. Most small sites do not need it. The summary file carries most of the value. To see how far the format stretches in practice, I compared real llms.txt examples from Anthropic, Stripe, Zapier, and Supabase in a separate breakdown.

Does llms.txt actually work?

Here is the honest answer most guides skip: no major search engine has confirmed using the file. Google has not committed to it, and Google's John Mueller has publicly compared it to the old keywords meta tag. Anyone promising that the file will boost your rankings is selling something.

So why bother? Three reasons. Adoption is growing on the AI side: documentation platforms generate the file automatically, and AI companies, including Anthropic, publish one for their own docs. The cost is close to zero: one file, a few minutes, no code. And it is a cheap bet with real upside. If AI assistants lean on the file more over time, early adopters win the framing. If they never do, you lost fifteen minutes.

llms.txt is not a ranking factor. It is a low-cost, low-risk way to control how the AI systems that do read it summarize your site.

What happened when I added llms.txt to this site

I practice what I write about, so this site publishes its own file at jnabear.com/llms.txt. Writing it took about fifteen minutes: the business summary, the six services with starting prices, every article, and the free tools. I also left a note in robots.txt pointing crawlers to it.

Two honest observations from doing it. First, I cannot yet measure whether AI systems read the file. There is no report for that, and anyone who claims exact numbers is guessing. Second, the exercise itself had value I did not expect. Forcing your whole business into one clear paragraph is a positioning drill. If you cannot write that paragraph, no AI summary of your site will be sharp either.

One practical habit: I update the file whenever prices or services change. A stale file feeds AI systems old facts in your own voice, which is worse than silence.

How to create an llms.txt file in five minutes

• Write a one-paragraph summary: what the site is, who runs it, what it offers.
• List the 10 to 30 pages that best represent you, with a one-line description each.
• Group them into sections a stranger would understand: Services, Guides, About.
• Produce the Markdown by hand, or use a free llms.txt generator to enforce the format.
• Upload the file to your web root so it resolves at yoursite.com/llms.txt.
• Update it whenever pricing, services, or key facts change.

Test it the way a crawler reads it: open yoursite.com/llms.txt in a browser and read it cold. If a stranger could describe your business accurately from that file alone, it works. For platform-specific steps, I have a separate guide on how to add llms.txt to WordPress, Shopify, and static sites.

Should you add one to your site?

For most sites, yes. It costs minutes, it cannot hurt, and it is the cheapest move in the wider game of answer engine optimization. But keep it in proportion. The file is one small layer, and it will not rescue a page that is not ranking for deeper reasons. AI search visibility is earned by the bigger stack: entity-rich content, schema markup, quotable definitions, and genuine topical authority. The file introduces your site; the content still has to deserve the citation.

I publish one for this site, generate them for clients, and built a free llms.txt generator so you can make your own. That is about as honest as an endorsement gets: small bet, small cost, upside if AI search keeps growing the way it has.