Definition
An AI crawler is an automated client that fetches web pages for an AI company, either to train a model or to source a live answer. Unlike Googlebot, which feeds a ranked search index, an AI crawler feeds a training corpus or a single generated response. The two kinds often need different rules in robots.txt.
At a glance
- CategoriesTechnical
- Related fieldsSEO, Technical
- DifficultyBeginner
At a glance
- An AI crawler identifies itself with a user agent string, the same way Googlebot does.
- Some AI crawlers collect content on a schedule. Others fetch a page the moment a person asks a question.
- Access is granted or refused per user agent in robots.txt, not site-wide.
- A server log is your only first-party record of which AI crawlers actually reached you, and crawlability problems show up there first.
Two jobs, two kinds of bot
The word crawler covers two behaviors that call for different decisions.
The first is bulk collection. A bot works through your site on its own schedule, stores what it finds, and that content may end up in the training data behind a future model. GPTBot and Common Crawl’s CCBot work this way. Nothing you publish on Monday shows up in an answer on Tuesday because of them.
The second is retrieval while an answer is being written. Somebody asks Perplexity a question, the system decides your page is worth reading, and a fetcher pulls it in real time. Perplexity’s crawler documentation splits the two across different user agents, one for indexing and one for requests a user started.
Both arrive over ordinary HTTP and announce themselves in the user agent header. Your server treats them like any other visitor unless you have told it not to.
Blocking a crawler is a trade, not a safety measure
The default advice going round is to block the AI bots. Look at what you are buying.
Blocking a collection crawler keeps your content out of a future model. Publishers with licensing revenue at stake make that choice deliberately. Anthropic publishes how to block ClaudeBot, Google documents its own crawler tokens, and the mechanism does what it says.
Blocking an answer-time fetcher is a different trade. You are telling an assistant it may not read your page while writing an answer about your own category. It writes that answer anyway, from whoever it could read. This is why Google-Extended and the retrieval agents are separate controls rather than one switch.
The failure we run into most often is accidental. A bot-mitigation product or a stale robots.txt line blocks a fetcher nobody meant to block, and a brand spends a quarter rewriting content no assistant can reach.
Take an inventory of who you currently allow
An hour of work, no tooling required.
- Open your robots.txt and read it line by line. Note every user agent carrying a Disallow rule and who owns that agent.
- Check the wildcard first. A blanket
Disallow: /underUser-agent: *blocks every bot you have not named separately. - Pull a week of server logs and list the AI user agents that arrived. Log file analysis is the only way to spot a bot you believed you allowed and never actually see.
- Decide collection and retrieval separately, and write down the reason so nobody reverses it by accident.
- Re-check after any CDN or security change. Bot-mitigation settings sit above robots.txt and get changed without telling marketing.
Collection crawlers vs answer-time fetchers
| Collection crawler | Answer-time fetcher | |
|---|---|---|
| What starts the fetch | A scheduled crawl | A person asking a question now |
| What the content feeds | A future model | The answer being written this minute |
| Cost of blocking it | Your pages stay out of the next model | Your page cannot be cited live |
| Speed of effect | Months, tied to training cycles | Immediate |
| Worth blocking? | Sometimes, a genuine decision | Rarely, it costs citations directly |
Frequently asked questions
-
Should I block AI crawlers?
Not by default. Decide the two kinds separately: blocking collection crawlers protects your content from future training, while blocking answer-time fetchers removes you from live answers in your own category. Publishers with content licensing revenue often block collection and allow retrieval. Most brands trying to be found should allow both.
-
How do I tell a real AI crawler from a fake one?
User agent strings are self-reported, so anything can claim to be GPTBot. The major operators publish IP ranges or reverse DNS patterns you can check a request against. Verify before you make an access decision, particularly if you are about to block something.
-
Is an AI crawler the same as Googlebot?
No, though they behave the same way on the wire. Googlebot fetches pages for the search index that produces ranked results. An AI crawler fetches pages for a model or for a generated answer. Google runs both kinds under different tokens, which is why one robots.txt rule cannot cover the question.
-
Will allowing AI crawlers send me traffic?
Often very little, and it is worth being honest about that. An assistant reading your page may name you without linking, or link in a way your analytics files as direct. The return shows up as mentions inside answers, not as sessions. Judge the decision on where your brand appears, not on clicks.