Automatic crawlers
GPTBot, OAI-SearchBot, ClaudeBot, Claude-SearchBot, PerplexityBot, and CCBot identify automatic crawl or indexing functions in their operators' documentation.
First-party sources · observed policy states
Know which AI crawler a rule actually controls.
Training, search, user-requested fetching, grounding, and open-web archiving use different robots.txt tokens. This directory combines first-party operator documentation with CrawlLedger's latest observed pilot records.
11documented tokens
6operators
20observed domains
0Content Signal adopters
Current robots.txt policy landscape
Counts come from the latest available robots.txt observation for each domain in the bounded pilot. “No matching group” means the file did not select that token or a wildcard group; it is not proof of deliberate permission.
| Crawler token | Documented purpose | Disallowed | Partial | Allowed | No match / unverified |
|---|---|---|---|---|---|
| GPTBot OpenAI | Model training | 7 | 8 | 4 | 1 |
| OAI-SearchBot OpenAI | AI search | 3 | 12 | 4 | 1 |
| ChatGPT-User OpenAI | User-requested fetch | 4 | 11 | 4 | 1 |
| ClaudeBot Anthropic | Model training | 7 | 8 | 4 | 1 |
| Claude-SearchBot Anthropic | AI search | 5 | 10 | 4 | 1 |
| Claude-User Anthropic | User-requested fetch | 5 | 10 | 4 | 1 |
| Google-Extended | Training and grounding control | 4 | 11 | 4 | 1 |
| Applebot-Extended Apple | Model training | 6 | 9 | 4 | 1 |
| PerplexityBot Perplexity | AI search | 6 | 9 | 4 | 1 |
| Perplexity-User Perplexity | User-requested fetch | 2 | 13 | 4 | 1 |
| CCBot Common Crawl | Open web archive | 6 | 9 | 4 | 1 |
Registry version 2026-08-13 · 20 current robots records represented.
Why the distinctions matter
A single operator may publish separate controls for model training, search discovery, and a human-requested page visit. Blocking one token does not automatically describe the others.
User-requested fetchers
ChatGPT-User, Claude-User, and Perplexity-User act after a person asks for content. Some operators explicitly say ordinary robots rules may not apply.
Control tokens
Google-Extended and Applebot-Extended express downstream-use choices but are not separate HTTP fetcher identities.
Evidence boundaries
Every reference page links to the operator's documentation and states when it was reviewed. Observed policy states describe parser output, not authenticated traffic, operator compliance, site ownership, or legal permission.