AI crawler directory / Google-Extended
Google-Extended robots.txt reference.
Google's robots.txt control token for Gemini model training and grounding uses of content already crawled by Google.
What Google-Extended is documented to do
This summary is tied to a first-party operator page reviewed on Aug 13, 2026. It should be rechecked when the registry version changes.
- Operator
- Documented purpose
- Training and grounding control
- Separate HTTP fetcher
- No — documented as a robots.txt control token
- Official documentation
- https://developers.google.com/crawling/docs/crawlers-fetchers/google-common-crawlers#google-extended
- Important operator note
- Control token only; Google says it is not a separate HTTP fetcher and does not affect Google Search inclusion or ranking.
Common full-site robots.txt block pattern
This example states a crawl preference for the named token. It does not authenticate requests, enforce blocking, or automatically control another token from the same operator.
User-agent: Google-Extended
Disallow: /Check the official operator source before publishing a policy. User-requested fetchers and non-fetcher control tokens can have different behavior from automatic crawlers.
Latest observed pilot policies
These states are calculated from each domain's latest available robots.txt record using the versioned CrawlLedger parser. They are observations of published text, not proof of crawler behavior.
| Domain | Parsed state | Matched group | Last observed |
|---|---|---|---|
| anthropic.com | allowed | * | |
| apnews.com | partial | * | |
| apple.com | partial | * | |
| cloudflare.com | allowed | Google-Extended | |
| commoncrawl.org | partial | * | |
| google.com | partial | *, Yandex | |
| medium.com | partial | * | |
| openai.com | partial | * | |
| oreilly.com | disallowed | GPTBot, anthropic-ai, Google-Extended, cohere-ai, CCBot, AI2Bot, Amazonbot, Amazonbot-Video, Bytespider, meta-externalagent, Diffbot, omgili, TimpiBot, SeznamBot, Exabot, YandexBot, Sogou, 360Spider, YisouSpider, Baiduspider-Render/2.0, MJ12bot, DataForSeoBot, AliyunSecBot, ArchiveTeam, ArchiveTeam ArchiveBot, AwarioBot, ZoominfoBot, ali-implementer, Blueno, BIGO-baiguoyuan, WorksOgCrawler, OnPageBot, DoCoMo, SAMSUNG-SGH-E250, TA-Googlebot | |
| perplexity.ai | partial | * | |
| quora.com | partial | Google-Extended | |
| reddit.com | disallowed | * | |
| rslcollective.org | allowed | * | |
| rslstandard.org | allowed | * | |
| stackoverflow.com | unverified | No matching group observed | |
| theguardian.com | partial | * | |
| usatoday.com | disallowed | Google-Extended | |
| voxmedia.com | partial | * | |
| yahoo.com | disallowed | ADmantX, AlphaBot, anthropic-ai, AwarioRssBot, AwarioSmartBot, BLEXBot, Buzzbot, Bytespider, CCBot, ChatGPT-User, claritybot, Claude-Web, ClaudeBot, cohere-ai, Diffbot, FacebookBot, FriendlyCrawler, Google-Extended, GPTBot, huggingface, ImagesiftBot, img2dataset, magpie-crawler, Meltwater, Neevabot, news-please, NewsNow, Nutch, omgili, omgilibot, panscient.com, Perplexity-ai, PerplexityBot, PetalBot, PiplBot, scoop.it, Scrapy, Seekr, SentiBot, SeznamBot, TurnitinBot, YouBot, ZumBot | |
| ziffdavis.com | partial | * |
Interpret the state carefully
“Allowed” means a selected group was observed without a blocking rule for the tested policy shape. “Partial” means at least one non-empty disallow path was observed. “No matching group” is silence, not an affirmative grant. None of these states authenticate the requester or establish legal permission.