Skip to content
AI Search

llms-full.txt vs llms.txt: What Each File Holds, and How Big Real Ones Are

Joseph Nicholas Abear 12 min read
llms-full.txt vs llms.txt compared: an index of links with short notes, against the full docs text in one file, with median sizes from 30 sites.
From the llms.txt proposal, Mintlify, GitBook and Fern docs, and a JNAbear fetch of 30 docs sites on Nov 10, 2026.

Quick answer

llms.txt is a short Markdown index of a site's key pages, each with a one-line note. llms-full.txt is a separate file that puts the full text of a documentation site in one download. Only llms.txt is in the llms.txt proposal. Size is the other big gap in llms-full.txt vs llms.txt: in my Nov 10, 2026 check of 30 docs sites, the median llms.txt was 31.6 KB and the median llms-full.txt was 3.7 MB.

Key takeaways

  • The llms.txt proposal at llmstxt.org (v2, August 2026) never names llms-full.txt. It says the index "stays small enough to fit in context" (read Nov 10, 2026).
  • llms-full.txt is a docs-platform convention. Mintlify and GitBook generate it. Fern does not, and says full-site files "saw little use".
  • Of 30 docs sites checked on Nov 10, 2026, 23 served a separate llms-full.txt, 4 redirected it to llms.txt, and 3 returned a 404.
  • Median sizes: llms.txt 31.6 KB (about 7,900 tokens) and llms-full.txt 3.7 MB (about 936,000 tokens), estimated as bytes divided by 4.
  • The largest was Cloudflare's llms-full.txt at 70.3 MB. Ten of the 23 files are over 1 million tokens by the same estimate.
  • jnabear.com publishes a small llms.txt (13.9 KB on Nov 10, 2026) and no llms-full.txt. A small business site rarely needs the full file.
On this page (9)

What is the difference between llms-full.txt and llms.txt?

llms.txt is a short Markdown index: a site name, a summary, and links to key pages with one-line notes. llms-full.txt is a separate file that holds the full text of every docs page in one download.

The names come from two different places. The llms.txt format was proposed by Jeremy Howard of Answer.AI in September 2024, and the proposal lives at llmstxt.org. llms-full.txt has no such home: it is a file name that documentation platforms generate, and that other sites copy. That origin is the first thing to know about llms-full.txt vs llms.txt. Two more terms matter here. A token is the unit a language model counts text in, and a context window is how many tokens a model can take in at once. File size is the main thing that separates the two files.

Point llms.txt llms-full.txt
What it holds H1 name, short summary, sections of links with notes The full Markdown text of the docs, page after page
Defined by The llms.txt proposal at llmstxt.org No spec; docs platforms such as Mintlify and GitBook
Median size, 30 docs sites (Nov 10, 2026) 31.6 KB, about 7,900 tokens 3.7 MB, about 936,000 tokens (23 sites)
Median Markdown links About 192 Not the point of the file
Served by the 30 sampled sites 30 of 30 (the sample rule) 23 of 30 as a separate file

The two files answer different needs. The index tells an AI agent where to look, so the agent fetches only the pages it needs. The full file hands over everything at once. That is the whole of llms-full.txt vs llms.txt in one line: a map, or the territory in a single file.

Does the llms.txt proposal define llms-full.txt?

No. The llms.txt proposal at llmstxt.org, version 2 from August 2026, does not mention llms-full.txt anywhere. I read the full text and its changes page on Nov 10, 2026.

The proposal points the other way. It says "Agents are expected to view or search llms.txt to find the information they need, then follow the relevant links." It adds: "The file itself stays small enough to fit in context. The detail lives behind the links, and is fetched only when needed." So in the spec's own model, depth comes from the linked pages, not from one large file.

Version 1 came close to the idea under different names. Its text, as it stood before the v2 rewrite, described how the FastHTML project expanded its index into "two markdown files with the contents of the linked URLs." The first was llms-ctx.txt, "which does not include the optional URLs." The second was llms-ctx-full.txt, "which does include them." A command line tool, llms_txt2ctx, built them. The v2 changes page drops that: "The context-expansion tooling is no longer part of the proposal." On Nov 10, 2026, FastHTML's docs served an llms.txt and no llms-full.txt.

So where did the name come from? Mintlify, a docs platform, says on its blog that "llms-full.txt is a standard Mintlify pioneered with Anthropic." That is a vendor's account of its own work, and I found no primary source from Anthropic on it. What is certain is that the name is a convention, not part of the llms.txt spec.

What goes inside an llms-full.txt file?

An llms-full.txt file holds the full Markdown text of a documentation site, usually page by page, each with its title and source URL. No spec sets its format, so each platform builds it a little differently.

Mintlify's docs say its file "combines your entire documentation site into a single file as context for AI tools and LLM indexing." Each page appears as "its title, source URL, description, and full Markdown content." GitBook's docs say the file "includes your full documentation content in one place." They also note that "Hidden pages are included in llms-full.txt." That second point is worth knowing: a page you hid from your table of contents can still ship in the full file.

The 23 real files I fetched did not share one shape. Seventeen opened with a Markdown H1, the one part llms.txt requires. Three opened with YAML front matter instead (Cloudflare, Next.js and Vite). Two opened with a line wrapped in SYSTEM tags (Svelte and Hono). GitBook's opened with a note that it is "part 1 of 8." Docker's listed page URLs on plain lines instead of as Markdown links. Here is the common shape, as an illustration only:

# Example Docs

> Example is a payments API. This file holds the full text of the docs.

# Getting started
Source: https://docs.example.com/getting-started

Install the SDK, then create an API key in the dashboard...

# Webhooks
Source: https://docs.example.com/webhooks

Webhooks send an event to your server when a payment changes state...

How big are real llms-full.txt and llms.txt files?

Big. On Nov 10, 2026, across 30 docs sites, the median llms.txt was 31.6 KB and the median llms-full.txt was 3.7 MB, about 118 times larger. Site by site, the median full file was 95 times its own index. The largest full file, Cloudflare's, was 70.3 MB.

Measure (30 docs sites, Nov 10, 2026) llms.txt llms-full.txt
Sites serving the file 30 of 30 (the sample rule) 23 of 30 (4 redirect to llms.txt, 3 return 404)
Median size 31.6 KB 3.7 MB
Smallest 1.7 KB (Svelte) 402 KB (Hono)
Largest 564.3 KB (Clerk) 70.3 MB (Cloudflare)
Median lines About 247 89,890
Median tokens, bytes divided by 4 About 7,900 About 936,000
Files over 1 million tokens, same estimate 0 10 of 23
Median size ratio, full to index n/a 95 times
Served as text/markdown 3 of 30 3 of 23
Survey figures for llms-full.txt: 23 of 30 docs sites publish one, the median is 3.7 MB against 31.6 KB for llms.txt, and the largest is 70.3 MB.
Results of a JNAbear fetch of 30 docs sites on Nov 10, 2026.

For llms-full.txt vs llms.txt, size is the clearest dividing line, and the spread is wide. The full file ran from 4 times the index (GitBook, whose file is paginated) to over 5,000 times. That top case is LangChain: a 3.1 KB index in front of a 17.0 MB full file. Size follows the docs, not the format. Every site in the table below is a developer docs site, which is where both files are most common.

Site llms.txt llms-full.txt Full vs index
Cloudflare 16.8 KB 70.3 MB 4,189x
Anthropic 98.8 KB 51.3 MB 519x
LangChain 3.1 KB 17.0 MB 5,428x
Vercel 5.4 KB 12.4 MB 2,301x
OpenAI 6.2 KB 9.7 MB 1,557x
Google Gemini 67.6 KB 9.3 MB 138x
Neon 29.5 KB 7.6 MB 257x
Prisma 8.4 KB 7.3 MB 871x
Supabase 3.3 KB 6.8 MB 2,048x
Next.js 51.4 KB 4.5 MB 87x
Perplexity 43.2 KB 3.9 MB 90x
Drizzle ORM 39.3 KB 3.7 MB 95x
Pinecone 40.8 KB 3.6 MB 87x
Zapier 54.7 KB 2.5 MB 46x
Resend 61.4 KB 2.5 MB 41x
Bun 33.8 KB 2.1 MB 63x
Mintlify 25.5 KB 1.8 MB 71x
Svelte 1.7 KB 1.3 MB 760x
Turso 34.6 KB 951 KB 28x
Docker 6.6 KB 467 KB 71x
GitBook 132.0 KB 466 KB (part 1 of 8; 11.3 MB in all) 4x
Vite 3.5 KB 434 KB 123x
Hono 5.9 KB 402 KB 68x
Stripe 99.2 KB 404 n/a
Netlify 25.1 KB 404 n/a
FastHTML 4.8 KB 404 n/a
Fern 4.8 KB Redirects to llms.txt n/a
ElevenLabs 252.7 KB Redirects to llms.txt n/a
Clerk 564.3 KB Redirects to llms.txt n/a
Expo 57.7 KB Redirects to llms.txt n/a

llms-full.txt survey data (CSV)

All 30 sites plus the four that failed the sample rule and jnabear.com: status, final URL, content type, bytes, lines, words, token estimate, link count and ratio. Fetched Nov 10, 2026.

Download

How the survey was run: I wrote a fixed list of 30 docs sites and five reserves before fetching anything. Each site got one request for /llms.txt and one for /llms-full.txt at its docs path, following redirects. The requests used the same plain user agent the AI Crawler Checker sends, never a browser or Googlebot. Four sites on the list did not serve a real llms.txt (Astro, Cursor, Shopify, and the reserve Hugging Face), so reserves took their places in the order written. Sizes are bytes as delivered, in decimal units (1 KB is 1,000 bytes). Tokens are estimated as bytes divided by 4, a rough rule: real tokenizers vary, and code-heavy docs often run higher. Links count Markdown links of the form [text](url). The CSV names every redirect and failure.

Is there a file size limit for llms.txt or llms-full.txt?

No standard sets a byte limit for either file. The llms.txt proposal only says the index "stays small enough to fit in context," and no spec covers llms-full.txt at all. Platforms set their own limits.

Mintlify is the clearest case. Its docs say auto-generated llms.txt files "cannot exceed 100,000 characters," and that larger indexes are split into linked files under /_llms/. Then: "This limit does not apply to llms-full.txt." Three index files in my sample were over 100,000 characters: Clerk (564,302), ElevenLabs (252,686) and GitBook (131,873). GitBook splits its full file instead: /llms-full.txt is part 1 of 8, and the eight parts came to 11.3 MB on Nov 10, 2026.

In practice, the limit that matters in llms-full.txt vs llms.txt is the reader. Anthropic's docs give Claude's context window as "up to 1M tokens, depending on the model" (read Nov 10, 2026). By the bytes divided by 4 estimate, 10 of the 23 llms-full.txt files are over that, and Cloudflare's is about 17.6 million tokens. Files that size do not fit in one request, so a tool has to search them or split them into chunks. Most of the llms.txt files, by contrast, fit with room to spare: the median is about 7,900 tokens.

Which tools generate llms-full.txt?

Mintlify and GitBook generate both llms.txt and llms-full.txt automatically for docs they host. Fern generates only llms.txt and says so in its docs, as read on Nov 10, 2026.

Platform llms.txt llms-full.txt What its docs say (Nov 10, 2026)
Mintlify Generated Generated Index capped at 100,000 characters; "This limit does not apply to llms-full.txt."
GitBook Generated Generated "This file includes your full documentation content in one place."
Fern Generated Not generated "Fern doesn't generate llms-full.txt."

Fern gives its reason plainly: "Full-site concatenation exceeded most model context windows, added heavy serving overhead, and saw little use compared to llms.txt combined with per-page Markdown access." The survey showed the same choice on live sites. Four sites, Fern, ElevenLabs, Clerk and Expo, answered a request for /llms-full.txt by redirecting to their llms.txt. So the platforms themselves split on llms-full.txt vs llms.txt. Mintlify also lets you replace its generated files with your own.

Do AI tools read llms-full.txt more than llms.txt?

No independent study I found answers that yet. The only published numbers I found come from Mintlify, a vendor that generates the file, and they date from June 2025, so treat them as vendor data.

Mintlify's blog post, written with Profound, a platform that tracks AI crawler visits, looked at 25 companies over seven days. It reported a median of 79 visits to llms-full.txt against 14 to llms.txt, and said ChatGPT made most of the llms-full.txt visits. A visit is a fetch, not proof that a model used the content. On usage, the evidence for llms-full.txt vs llms.txt is thin either way. Fern's docs report the opposite experience: full-site files "saw little use." No primary source I found says any named model reads llms-full.txt by default. The index itself is rarely fetched at all: Ahrefs studied 137,210 domains and reported that 97% of the roughly 38,000 valid llms.txt files among them "received zero traffic in May 2026." Ahrefs calls its figures a ceiling.

One sign of how loosely the two files are linked: only 9 of the 30 llms.txt files in my sample link to their own llms-full.txt. A tenth, Mintlify's, names the file only in the title of a help page. The other 14 sites that publish a full file do not point to it from the index.

Does a small business site need llms-full.txt?

Usually not. A small business site has a handful of pages, and a good llms.txt already links every one of them. The full file exists for documentation sets of hundreds of pages.

This site is the example. On Nov 10, 2026, before this post was added to it, jnabear.com served a 13.9 KB llms.txt: 87 lines, 61 links, about 3,500 tokens by the same estimate. It serves no llms-full.txt; that URL returns a 404. Every page the file links to is short enough to fetch on its own. A full file would only repeat those pages in one place, and it would be one more thing to keep current. To see how other files are laid out, the llms.txt examples breakdown compares five real ones. The llms.txt vs robots.txt guide covers which AI file does which job.

A full file starts to make sense when the content is the product: API docs, a large help center, a knowledge base that people ask AI tools about. If that is you, the docs platform you use may already generate one. If not, the next section has the short version.

For everyone else, the index is the job. You can build or check yours with the free llms.txt generator, and the guide on how to add llms.txt covers placement on WordPress, Shopify and static hosts. If you are building a new site, the file can be part of the build: that is included in our web design work, delivered with Stoute Web Solutions. Disclosure: that is a service I sell, so weigh the source. For a small site, llms-full.txt vs llms.txt is an easy call: publish the index, and skip the full file. The free generator and fifteen minutes are all most people need.

How do you make an llms-full.txt file?

On Mintlify or GitBook you do nothing, since both platforms generate the file. Elsewhere, join the Markdown of each docs page into one file, give each page its title and source URL, and serve it at /llms-full.txt.

  • Start the file with an H1 naming the site, the one part llms.txt requires. Mintlify requires it in a custom llms-full.txt too.
  • Give each page its own heading and its source URL, so a tool can cite the page and not just the file.
  • Leave out pages you would not publish. GitBook's docs note that hidden pages are included in its generated file.
  • Serve it as plain text. In my sample, 20 of 23 full files came as text/plain and 3 as text/markdown.
  • Link to it from your llms.txt, so agents can find it. Only 9 of 30 index files in my sample did.
  • Regenerate it whenever the docs change. A stale full file repeats old facts at scale.

Watch the size as the docs grow, because the answer to llms-full.txt vs llms.txt can change with them. If your file heads toward the millions of tokens, split it by section, as GitBook does. Or skip it, as Fern does, and let the index and per-page Markdown carry the load.

Topics & entities in this article

llms-full.txt llms.txt llmstxt.org Jeremy Howard Answer.AI Markdown llms-ctx-full.txt Context window Token Mintlify GitBook Fern Anthropic Cloudflare AI agents

Frequently asked questions

No. The llms.txt proposal at llmstxt.org (v2, August 2026) does not mention llms-full.txt. It says the index "stays small enough to fit in context" and that detail lives behind the links. llms-full.txt is a convention that docs platforms such as Mintlify and GitBook generate.

Neither is better; they do different jobs. In llms-full.txt vs llms.txt, the index is the file the proposal defines. The full file suits large docs sets: in a Nov 10, 2026 check of 30 docs sites with an llms.txt, 23 also served llms-full.txt.

llms-ctx-full.txt was a file the FastHTML project built from its llms.txt, described in v1 of the proposal, with the contents of every linked URL including optional ones. llms-full.txt is the later platform convention. The v2 proposal dropped the context-expansion tooling.

In a Nov 10, 2026 check of 30 docs sites, the 23 llms-full.txt files ran from 402 KB (Hono) to 70.3 MB (Cloudflare), with a median of 3.7 MB. The median llms.txt was 31.6 KB, so the median full file was about 118 times larger. Site by site, the median full file was 95 times its own index.

If you publish both, yes, so agents can find the full file without guessing. Only 9 of 30 llms.txt files checked on Nov 10, 2026 linked to their llms-full.txt, though 23 of those sites served one.

Often not. Anthropic documents Claude's context window as "up to 1M tokens, depending on the model," and 10 of 23 llms-full.txt files in a Nov 10, 2026 check were over 1 million tokens by a bytes divided by 4 estimate. Tools search or chunk files that size.

No. On Nov 10, 2026, before this post was added to it, jnabear.com served a 13.9 KB llms.txt with 61 links, and /llms-full.txt returned a 404. Every linked page is short enough to fetch on its own, so a full file would add upkeep without adding content.

Related service

Web Design & SEO

Web design and SEO in one project: custom WordPress websites for businesses that need a new or rebuilt site.

Disclosure: this is a service I sell.

Keep reading in The Content Lab

AI Search 8 min read

What Is llms.txt? A Practical Guide to the AI Crawler File

llms.txt is a plain-text file that tells AI crawlers what your site is about. Learn what goes inside, see a real example, and decide if you need one.

Jul 14, 2026
AI Search 7 min read

llms.txt Examples: Real Files From Real Sites (and What to Copy)

Real llms.txt examples from Anthropic, Stripe, Zapier, Supabase, and Cloudflare. See how the files differ, which patterns to copy, and how small yours can be.

Jul 17, 2026
AI Search 12 min read

What Is ai.txt? Who Publishes It, and Whether Any AI Crawler Reads It

Spawning's 2023 file explained: what is ai.txt, what the 2026 Internet-Draft proposes, what AI crawler docs say, and a check of 1,000 top sites.

Nov 14, 2026