Glossary
Google-Extended: What It Is and How to Use It
Definition
Google-Extended is a robots.txt token, not a crawler, that controls whether Google may use your content to improve its Gemini models. Unlike Googlebot, which actually fetches pages, Google-Extended never appears in your server logs. Blocking it does not remove you from Google Search or change how Googlebot crawls you.
At a glance
- CategoriesTechnical
- Related fieldsSEO, Technical
- DifficultyIntermediate
At a glance
- Google-Extended is written into robots.txt like a user agent, but nothing crawls under that name.
- It governs whether your content helps improve Gemini apps and Google’s grounded generative APIs.
- Google’s documentation states that blocking it does not affect inclusion in Google Search.
- The decision is reversible by editing one line.
A control token with no crawler behind it
You add it to robots.txt the way you would add any user agent:
User-agent: Google-Extended
Disallow: /
Nothing then arrives at your server calling itself Google-Extended. Googlebot keeps doing the fetching, exactly as before. The token is a declaration about use, applied at the point Google decides what a fetched page may feed.
Google’s crawler overview lists it next to the real crawlers and sets out what it governs.
Two things follow. You will never see the effect in your logs, because there is nothing to see. And you cannot verify compliance from your side: you are taking Google at its word, which for a public, documented token is a fair bet.
What you give up, and what you keep
The appeal is a single line that keeps your writing out of a large model. Be precise about where that line stops.
What you keep: your pages stay in Google Search, Googlebot keeps crawling, indexability is untouched, and the snippet controls you already use behave the same.
What you give up: your content stops feeding the Gemini products the token covers. If your buyers ask Gemini about your category, you have removed yourself from the material it draws on.
Who this suits is a short list. Publishers and rights holders whose content is the product have a real reason to block. Most brands do not. You would be paying to be less present in a place your customers already ask questions.
Set it once, deliberately
- Decide who owns the call. This is a licensing and brand question before it is a technical one. Marketing and legal both sign it off.
- Read what is in robots.txt today. Plenty of sites blocked Google-Extended during an early clean-up and never revisited it. Crawlability audits turn this up regularly.
- Write the rule explicitly if you want it. A named block, not a wildcard, so the next person can see what was intended.
- Leave Googlebot alone. Blocking Googlebot to control AI use removes you from Search too, which is almost never what anyone meant.
- Add a comment giving the reason. One line in the file settles an argument two years from now.
Googlebot vs Google-Extended
| Googlebot | Google-Extended | |
|---|---|---|
| Is it a crawler? | Yes, it fetches your pages | No, a control token only |
| Shows in server logs | Yes | Never |
| Cost of blocking it | Removal from Google Search | Use of your content in Gemini apps |
| Effect on rankings | Total | None |
| Worth blocking? | Almost never | Only if you license your content |
Frequently asked questions
-
Does blocking Google-Extended remove me from AI Overviews?
No. AI Overviews sit inside Google Search, and Google’s documentation is explicit that Google-Extended does not affect Search inclusion. If you want to limit how your content appears in Search results, the levers are the ordinary snippet directives, not this token.
-
Is Google-Extended the same as blocking GPTBot?
Similar intent, different mechanism. GPTBot is a real AI crawler, so blocking it stops the fetch happening. Google-Extended is a statement about permitted use: Googlebot still fetches your page, and only what the content may feed changes.
-
Should I block Google-Extended?
Probably not, and it turns entirely on your business model. If your content is the product and licensing it is a revenue line, block it on purpose. If you sell something else and want to be found, blocking makes you less present in Gemini and buys nothing back.
-
Can I reverse the decision later?
Yes. Delete the rule and the permission returns. What you cannot recover is the period it was in force, so treat it as reversible rather than free.