Back to Glossary

Glossary

llms.txt: What It Is and How to Use It

Updated September 2026TechnicalIntermediate

Definition

llms.txt is a plain text file at the root of a website that tells AI systems what the site is about and which pages matter most. It is a proposed convention, not a standard any model is obliged to follow. Unlike robots.txt, which controls what a crawler may access, llms.txt suggests what a crawler should read first.

At a glance

  • CategoriesTechnical
  • Related fieldsSEO, Technical
  • DifficultyIntermediate

At a glance

  • llms.txt lives at yoursite.com/llms.txt and is written in plain Markdown.
  • It is a curation signal, not an access control. robots.txt is the file that grants or blocks access.
  • Adoption by the major AI labs is partial and unconfirmed. No lab has committed to reading it.
  • A companion file, llms-full.txt, holds the full text of key pages rather than just links.
  • It takes about an hour to write and costs nothing to keep.

What actually goes in the file

The format is deliberately simple. An H1 with your site name, a blockquote summarizing what you do, then Markdown link lists grouped by section, each link with a short description of why it matters.

# Parsnipp

> AI search visibility platform for brands.

## Product
- [AI Visibility](https://parsnipp.com/ai-visibility/): how brands appear in ChatGPT, Gemini and Perplexity
- [Agentic Commerce](https://parsnipp.com/agentic-commerce/): readiness for AI shopping agents

## Reference
- [Glossary](https://parsnipp.com/glossary/): 113 AI search terms, maintained

That is the whole thing. No schema, no validation, no submission process. You write it, you upload it, you are done.

The honest reality

No major AI lab has publicly committed to reading llms.txt. Some crawlers appear to fetch it. None have said it changes anything about how they rank or cite you.

So treat it as a cheap bet, not a lever. It takes an hour, it cannot hurt you, and if the convention gets picked up you already have one. Anyone selling llms.txt as an AI visibility strategy is selling you an hour of work as a quarter of work.

It also gets confused with agents.md, which is a different file in a different place for a different reader: coding agents working inside a source repository.

The thing that genuinely moves crawlability is more boring: server-rendered HTML, an accurate sitemap, no content trapped behind JavaScript, and robots.txt that does not accidentally block GPTBot. Fix those first. Then write llms.txt on a Friday afternoon.

llms.txt vs robots.txt

  robots.txt llms.txt
What it does Grants or denies crawler access Suggests what to read and in what order
Status Long-established, universally honoured Proposed convention, adoption unclear
If you ignore it Crawlers may hit pages you wanted private Nothing happens
Who reads it Every crawler Some, unconfirmed
Effort Minutes About an hour

How to write one this week

  1. List your 15 to 25 most important pages. Product, pricing, reference, best content. Not everything.
  2. Write one line per link explaining what it covers. Plain description, not a pitch. The description is doing the work.
  3. Add one blockquote at the top saying what your company does in a sentence. Use the same sentence you use everywhere else.
  4. Save as Markdown, upload to your root, confirm it loads at /llms.txt as plain text and not as an HTML page.
  5. Add llms-full.txt if you have a reference section worth ingesting whole. A glossary or a docs set is the obvious candidate.

Keep it short. Get the boring crawlability work right. Revisit when a lab actually says it reads the file.

Frequently asked questions

  • Do AI crawlers actually read llms.txt?

    Some fetch it. None of the major labs have confirmed they use it for anything. Server logs show requests for the file from various agents, but a fetch is not evidence of influence. Write one because it is cheap, not because it is proven.

  • Is llms.txt the same as robots.txt?

    No. robots.txt controls access and every crawler respects it. llms.txt suggests priority and is advisory at best. They solve different problems and you want both.

  • Will llms.txt improve my AI visibility?

    There is no evidence it will on its own. Structured, readable, well-linked content is what gets cited. llms.txt is a pointer to that content, and a pointer to nothing points at nothing.

  • What is llms-full.txt?

    A companion file holding the actual text of your key pages rather than links to them. Useful if you have a reference section, like a glossary or documentation, that an assistant would benefit from reading in one pass.

  • Should a small site bother?

    Yes, if you have an hour. No, if that hour would otherwise go to fixing server-side rendering or getting your product pages out from behind JavaScript. Order matters more than coverage here.

Keep reading

Related terms