Live research · bounded evidence set
A live matrix of observed AI crawler rules.
This research page turns the latest retained robots.txt record for each observed domain into a reproducible token-by-domain matrix. It updates from append-only evidence instead of a manually rewritten headline.
20domains represented
12documented tokens
20current robots records
1domains with observed Content Signals
Bounded snapshot, not a web-wide survey.
The denominator is the 20 domains with evidence in CrawlLedger's curated set. The selection is not random or population-representative, so these counts must not be generalized to the web.
Latest state counts by crawler token
Every cell shows a count over the records represented for that token. “Allowed” is a parser result from selected rules, not authenticated access, operator compliance, consent, or legal permission.
| Crawler | Documented purpose | Disallowed | Partial | Allowed | No match / unverified |
|---|---|---|---|---|---|
| GPTBot | Model training | 7 / 20 | 8 / 20 | 4 / 20 | 1 / 20 |
| OAI-SearchBot | AI search | 3 / 20 | 12 / 20 | 4 / 20 | 1 / 20 |
| OAI-AdsBot | Ad landing-page validation | 1 / 20 | 14 / 20 | 4 / 20 | 1 / 20 |
| ChatGPT-User | User-requested fetch | 4 / 20 | 11 / 20 | 4 / 20 | 1 / 20 |
| ClaudeBot | Model training | 7 / 20 | 8 / 20 | 4 / 20 | 1 / 20 |
| Claude-SearchBot | AI search | 5 / 20 | 10 / 20 | 4 / 20 | 1 / 20 |
| Claude-User | User-requested fetch | 5 / 20 | 10 / 20 | 4 / 20 | 1 / 20 |
| Google-Extended | Training and grounding control | 3 / 20 | 12 / 20 | 4 / 20 | 1 / 20 |
| Applebot-Extended | Model training | 6 / 20 | 9 / 20 | 4 / 20 | 1 / 20 |
| PerplexityBot | AI search | 6 / 20 | 9 / 20 | 4 / 20 | 1 / 20 |
| Perplexity-User | User-requested fetch | 2 / 20 | 13 / 20 | 4 / 20 | 1 / 20 |
| CCBot | Open web archive | 6 / 20 | 9 / 20 | 4 / 20 | 1 / 20 |
Domain-by-crawler policy matrix
Use the horizontal table to compare how one current robots.txt record resolves across independently documented tokens. Open a domain or crawler page for timestamps, matched groups, parser versions, and evidence boundaries.
| Observed domain | GPTBot | OAI-SearchBot | OAI-AdsBot | ChatGPT-User | ClaudeBot | Claude-SearchBot | Claude-User | Google-Extended | Applebot-Extended | PerplexityBot | Perplexity-User | CCBot |
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| anthropic.com Oct 11, 2026 | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed |
| apnews.com Oct 11, 2026 | disallowed | partial | partial | partial | disallowed | disallowed | disallowed | partial | disallowed | disallowed | partial | disallowed |
| apple.com Oct 11, 2026 | partial | partial | partial | partial | partial | partial | partial | partial | partial | partial | partial | partial |
| cloudflare.com Oct 11, 2026 | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed |
| commoncrawl.org Oct 11, 2026 | partial | partial | partial | partial | partial | partial | partial | partial | partial | partial | partial | partial |
| google.com Oct 11, 2026 | partial | partial | partial | partial | partial | partial | partial | partial | partial | partial | partial | partial |
| medium.com Oct 11, 2026 | disallowed | partial | partial | partial | disallowed | partial | partial | partial | disallowed | partial | partial | partial |
| openai.com Oct 11, 2026 | partial | partial | partial | partial | partial | partial | partial | partial | partial | partial | partial | partial |
| oreilly.com Oct 11, 2026 | disallowed | partial | partial | partial | partial | partial | partial | partial | partial | partial | partial | disallowed |
| perplexity.ai Oct 11, 2026 | partial | partial | partial | partial | partial | partial | partial | partial | partial | partial | partial | partial |
| quora.com Oct 11, 2026 | disallowed | disallowed | partial | disallowed | disallowed | disallowed | disallowed | partial | disallowed | disallowed | partial | partial |
| reddit.com Oct 11, 2026 | disallowed | disallowed | disallowed | disallowed | disallowed | disallowed | disallowed | disallowed | disallowed | disallowed | disallowed | disallowed |
| rslcollective.org Oct 11, 2026 | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed |
| rslstandard.org Oct 11, 2026 | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed | allowed |
| stackoverflow.com Oct 10, 2026 | unverified | unverified | unverified | unverified | unverified | unverified | unverified | unverified | unverified | unverified | unverified | unverified |
| theguardian.com Oct 10, 2026 | partial | partial | partial | partial | disallowed | disallowed | disallowed | partial | disallowed | disallowed | partial | disallowed |
| usatoday.com Oct 10, 2026 | disallowed | disallowed | partial | disallowed | disallowed | disallowed | disallowed | disallowed | disallowed | disallowed | disallowed | disallowed |
| voxmedia.com Oct 10, 2026 | partial | partial | partial | partial | partial | partial | partial | partial | partial | partial | partial | partial |
| yahoo.com Oct 11, 2026 | disallowed | partial | partial | disallowed | disallowed | partial | partial | disallowed | partial | disallowed | partial | disallowed |
| ziffdavis.com Oct 11, 2026 | partial | partial | partial | partial | partial | partial | partial | partial | partial | partial | partial | partial |
Latest represented observation: · crawler registry 2026-09-20.
How to reproduce and cite the snapshot
The public JSON returns the same current observations used here. Domain pages add bounded history, artifact hashes, parser versions, and per-signal chain links; exact fetched bytes remain private.
2 / Download current states
3 / Verify the boundaries
Review fetching, parsing, provenance, and uncertainty rules.