Glossary
llms-full.txt: What It Is and How to Use It
Definition
llms-full.txt is a single plain text file holding the full content of a site’s key pages, not just links to them. llms.txt is the shorter companion, a curated index pointing at where things live. Both are proposed conventions, and no major AI lab has confirmed that it reads either one.
At a glance
- CategoriesTechnical
- Related fieldsSEO, Technical
- DifficultyIntermediate
At a glance
- llms-full.txt sits at
yoursite.com/llms-full.txtand is written as plain Markdown. - The file carries the text of your pages rather than links, which makes it large.
- A documentation set or a glossary is the realistic candidate. A blog archive is not.
- Adoption is unconfirmed, so treat the file as a cheap experiment rather than a channel.
One file, your whole reference section
The idea is to remove the round trips. Rather than fetching thirty documentation pages one at a time, a consumer reads the set in a single pass.
The llms.txt proposal covers both files. The short one is an index of links with one-line descriptions. The long one is the content itself, concatenated as Markdown with headings preserved.
There is no schema and no size rule, which is the weak point. A full-text dump of a large site runs to megabytes, and nothing in the convention says what anyone should do with that.
Keep it to a section that reads as a unit: documentation, a glossary, an API reference.
Where a full-text file earns its keep
Everyone wants this to be a distribution channel. It is closer to a convenience.
The case for it is narrow and real. If you publish a reference set an assistant would want whole, one clean Markdown file removes every chance of it hitting a broken render, a cookie wall or a page skeleton waiting on server-side rendering that never arrives.
The case against is that nothing is obliged to read it, and no lab has said it does. A file nobody has committed to fetching is not a plan.
So size the project accordingly. It is a generated artifact, not a content program.
Generate it from what you already publish
- Pick one section that reads as a set. If you cannot name it in three words, skip the whole thing.
- Export those pages to Markdown from your CMS, headings intact, navigation and footers stripped.
- Concatenate into one file with a heading per page and the source URL underneath each, so attribution survives.
- Serve it at
/llms-full.txtas plain text. Confirm it does not come back wrapped in your site template. - Regenerate it in your build. A hand-maintained copy goes stale within a month, and a stale file is worse than none.
Leave this alone until crawlability is sorted. A crawler that cannot read your pages is not rescued by a file repeating them.
llms.txt vs llms-full.txt
| llms.txt | llms-full.txt | |
|---|---|---|
| What it holds | Links with one-line descriptions | The full text of those pages |
| Typical size | A page or two | Whatever your reference section weighs |
| Effort | About an hour by hand | Only sensible if generated |
| How fast it goes stale | Slowly | Every time you edit a page |
| Worth it when | You want to signal priority | You publish a reference set |
Frequently asked questions
-
Do AI crawlers actually read llms-full.txt?
No lab has confirmed that it does. Some agents fetch the file, which server logs will show, but a fetch is not evidence that anything downstream changed. Publish it because generating it is nearly free, not because anyone has promised it counts.
-
Is llms-full.txt the same as a sitemap?
No. A sitemap lists URLs so a crawler can discover them, and carries no content. llms-full.txt carries the content and lists nothing for discovery. A sitemap is a long-standing protocol every crawler uses. This is a proposal with unconfirmed uptake.
-
How large should the file be?
Small enough that reading it beats crawling the pages, which in practice means one section rather than a whole site. If you are trimming a multi-megabyte export to make it usable, this was the wrong tool for that content.
-
Should I write llms.txt or llms-full.txt first?
llms.txt, and possibly only that. It takes an hour, it is easy to keep current, and it covers the whole site. The full-text version only makes sense once you have a reference section worth handing over whole and a build step that can produce it.