Glossary
robots.txt: What It Is and How to Use It
Definition
robots.txt is a plain text file at a site’s root that tells each named crawler which paths it may request. A noindex tag is the other control, and they are not interchangeable: robots.txt governs fetching, noindex governs whether a fetched page is stored. Blocking a URL here can leave it in search results with no description.
At a glance
- CategoriesTechnical
- Related fieldsSEO, Technical
- DifficultyBeginner
At a glance
- robots.txt sits at
yoursite.com/robots.txtand nowhere else. No crawler looks for it in a subfolder. - Rules are grouped by User-agent, and a crawler obeys only the most specific group that names it.
- Disallow prevents a fetch. It does not remove a URL from results and it does not make a page private.
- AI crawlers are named in this file the same way search crawlers always have been.
Four directives and one matching rule
The file is line-based and short. User-agent opens a group. Disallow and Allow list path prefixes. Sitemap points at your sitemap and applies to the whole site.
User-agent: *
Disallow: /checkout/
User-agent: GPTBot
Allow: /
Sitemap: https://example.com/sitemap.xml
Two behaviors catch people out. A crawler reads one group only, the most specific one naming it, so anything under User-agent: * is ignored by a bot with its own group. And when an Allow and a Disallow both match a URL, the longer rule wins.
RFC 9309 standardized all of this, including how a crawler should behave when the file cannot be fetched. Google’s robots.txt documentation covers the same ground with worked examples.
Disallow is not the same as hide
The most expensive misunderstanding about this file is treating Disallow as a privacy control.
Disallow stops the fetch. It does not stop the URL existing. If another site links to a blocked page, a search engine can list that URL with no description, because it was never allowed to read the page and find your noindex tag. Publicly listed and unexplained is the worst of both.
If a page must not be public, put it behind authentication. If it may be fetched but must not be stored, allow the crawl and use noindex, since the crawler has to read the page to obey one. Indexability is a separate decision from access.
The quieter cost is retrieval. Blocking a directory to save crawl capacity removes it from everything downstream, including the pool an assistant draws on when it writes an answer. One line, and a whole product category stops being quotable.
Audit the file, then verify with a fetch
- Load your robots.txt in a browser and read every line. Anything longer than a screen is mostly historical.
- List each User-agent group and who owns it. An agent nobody recognizes is a candidate for deletion, not for keeping just in case.
- Check the wildcard group last. A bot with its own group never reads it, which is how sites accidentally allow exactly what they meant to block.
- Name AI crawlers deliberately. Anthropic documents how to block ClaudeBot, and other operators publish the same guidance. Do not copy someone else’s list unread.
- Verify with a real request. Fetch a blocked URL and an allowed URL as the relevant user agent, and confirm you got what you expected.
Then leave it alone. This file needs changing far less often than people assume, and most of the damage comes from edits made in passing during some other project.
robots.txt vs noindex
| robots.txt Disallow | noindex tag | |
|---|---|---|
| What it stops | The fetch | The storing |
| Crawler must read the page | No | Yes |
| URL can still appear in results | Yes, with no description | No |
| Makes the page private | No | No |
| Right tool for | Saving crawl capacity, refusing a bot | Keeping a readable page out of the index |
Frequently asked questions
-
Does robots.txt actually stop AI crawlers?
The major ones honour it, and their operators document the user agent to use. Compliance is voluntary in the sense that the protocol has no enforcement, so a badly behaved scraper can ignore the file. For anything that must never be fetched, use access control rather than a request.
-
Is robots.txt the same as noindex?
No, and confusing the two causes real problems. robots.txt decides whether a page is fetched. noindex decides whether a fetched page is kept. A page blocked in robots.txt can still be listed as a bare URL, because the crawler never got to read the tag telling it not to.
-
Should I block AI crawlers here?
Decide it by name rather than in one sweep. Blocking bulk collection is a defensible choice for publishers who license content. Blocking the fetchers that read pages while writing an answer removes you from your own category. An AI crawler audit takes an hour and is worth more than any default list.
-
Where does llms.txt fit alongside this?
Different jobs, and you want both. robots.txt grants or refuses access and every crawler respects it. llms.txt suggests which pages matter most and is a proposal nobody has committed to reading. One is infrastructure, the other is a cheap experiment.
-
Can I tell whether crawlers are obeying the rules?
Partly. Server logs show which user agents requested which paths, so a bot fetching a disallowed directory is visible. What logs cannot tell you is what happened to the content after it left. Crawlability checks confirm access, and nothing on your side confirms use.