What Is llms.txt? A Practical Guide to the AI Crawler File

What is llms.txt?

llms.txt is a plain Markdown file that sits at the root of a website and tells AI systems what the site is, who runs it, and which pages matter most. Jeremy Howard, co-founder of Answer.AI, proposed the format in September 2024, and the specification lives at llmstxt.org. Think of it as a curated front door for machines: instead of forcing an AI crawler to reverse-engineer your navigation, you hand it the tour.

The format is deliberately simple. A single H1 names the site, a blockquote summarizes it, and short sections list annotated links. Any text editor can produce one, and any language model can read it without special parsing.

llms.txt is a Markdown file at a site's root that gives AI crawlers a curated summary of the site and annotated links to its most important pages.

Why does the file exist?

Two problems motivated it. First, language models read pages as text, and real HTML is noisy: navigation menus, scripts, cookie banners, and footers surround the actual content. Models also work within limited context windows, so every token spent parsing boilerplate is a token not spent understanding your business. A clean Markdown summary spends the model's attention on substance.

Second, it gives you control of the framing. When an AI assistant summarizes your company, it assembles that answer from whatever it found. The file is the one place where you state, in your own words, what your business is, who it serves, and which pages represent it best. In AI search, the site that defines itself clearly beats the site that leaves the definition to inference.

llms.txt vs robots.txt vs sitemap.xml

The three files answer three different questions, and they work together rather than competing.

• robots.txt answers "may you crawl this?" It grants or denies access to crawlers, including AI agents like GPTBot.
• sitemap.xml answers "what exists?" It inventories every URL with last-modified dates so crawlers miss nothing.
• llms.txt answers "what matters, and what does it mean?" It curates the short list and adds the context.

Permission, inventory, meaning. A well-run site publishes all three, and none of them replaces the others.

What goes inside an llms.txt file?

The spec asks for an H1 with the site name, a blockquote with a one-paragraph summary, optional detail paragraphs, and then sections of Markdown links with one-line descriptions. Here is a compact example for a fictional plumbing company:

# Acme Plumbing

> Acme Plumbing is a family-run plumbing company serving Portland since 1998. We handle residential repairs, remodels, and 24/7 emergencies.

Licensed and bonded in Oregon. Contact: hello@acmeplumbing.com.

## Services

- [Emergency Plumbing](https://acme.com/emergency): 24/7 response, flat call-out fee
- [Kitchen Remodels](https://acme.com/remodels): design-to-install renovations

## About

- [Our Story](https://acme.com/about): three generations of Portland plumbers

A related convention, llms-full.txt, goes further and packs the full text of the site into one file for models that want depth instead of a summary. Most small sites do not need it; the summary file carries most of the value.

Does llms.txt actually work?

Here is the honest answer most guides skip: no major search engine has confirmed using the file. Google has not committed to it, and Google's John Mueller has publicly compared it to the old keywords meta tag. Anyone promising that the file will boost your rankings is selling something.

So why bother? Three reasons. Adoption is growing on the AI side of the ecosystem: documentation platforms generate the file automatically, and AI companies, including Anthropic, publish one for their own docs. The cost is close to zero: one file, a few minutes, no code. And the bet is asymmetric: if AI assistants lean on it more over time, early adopters win the framing; if they never do, you lost fifteen minutes.

llms.txt is not a ranking factor. It is a low-cost, low-risk way to control how the AI systems that do read it summarize your site.

How to create an llms.txt file in five minutes

• Write a one-paragraph summary: what the site is, who runs it, what it offers.
• List the 10 to 30 pages that best represent you, with a one-line description each.
• Group them into sections a stranger would understand: Services, Guides, About.
• Produce the Markdown by hand, or use a free llms.txt generator to enforce the format.
• Upload the file to your web root so it resolves at yoursite.com/llms.txt.
• Update it whenever pricing, services, or key facts change.

Test it the same way a crawler reads it: open yoursite.com/llms.txt in a browser and read it cold. If a stranger could describe your business accurately from that file alone, it works. One warning: a stale file is worse than none, because you are feeding AI systems outdated facts in your own voice.

Should you add one to your site?

For most sites, yes. It costs minutes, it cannot hurt, and it is the cheapest move in the wider game of answer engine optimization. But keep it in proportion: the file is one small layer. AI search visibility is earned by the bigger stack, which means entity-rich content, schema markup, quotable definitions, and genuine topical authority. The file introduces your site; the content still has to deserve the citation.

I publish one for this site, generate them for clients, and built a free llms.txt generator so you can make your own. That is about as honest as an endorsement gets: small bet, small cost, upside if AI search keeps growing the way it has.