llms.txt
Unverified after HTTP response
- Observed
- HTTP
- 301
- Artifact
No retained artifact- Parser
- Not applied
Domain record · observed metadata
Archive / yahoo.com
yahoo.com policy history.
Timestamped observations of public machine-readable policy files. The states below describe fetched records and parser results; they do not establish legal permission.
20observations
2signal types
0parsed changes
Aug 13, 2026last observed
Current observations
Current means the latest ledger event for each observed signal. Conditional 304 responses reuse the prior artifact hash while extending the event chain.
robots.txt
Observed and parsed
- Observed
- HTTP
- 200
- Artifact
021adb45d8aa…- Parser
- robots-rfc9309-rsl-0.1.0
AI crawler rules in robots.txt
CrawlLedger evaluates a versioned registry of first-party-documented crawler tokens against the parsed robots groups.
Tracked crawler observations
These are parser results for the published robots rules, not authentication or legal conclusions.
| Crawler token | Observed state | Matched group |
|---|---|---|
| GPTBot OpenAI · training | disallowed | ADmantX, AlphaBot, anthropic-ai, AwarioRssBot, AwarioSmartBot, BLEXBot, Buzzbot, Bytespider, CCBot, ChatGPT-User, claritybot, Claude-Web, ClaudeBot, cohere-ai, Diffbot, FacebookBot, FriendlyCrawler, Google-Extended, GPTBot, huggingface, ImagesiftBot, img2dataset, magpie-crawler, Meltwater, Neevabot, news-please, NewsNow, Nutch, omgili, omgilibot, panscient.com, Perplexity-ai, PerplexityBot, PetalBot, PiplBot, scoop.it, Scrapy, Seekr, SentiBot, SeznamBot, TurnitinBot, YouBot, ZumBot |
| OAI-SearchBot OpenAI · search | partial | Claude-SearchBot, OAI-SearchBot |
| ChatGPT-User OpenAI · user_action | disallowed | ADmantX, AlphaBot, anthropic-ai, AwarioRssBot, AwarioSmartBot, BLEXBot, Buzzbot, Bytespider, CCBot, ChatGPT-User, claritybot, Claude-Web, ClaudeBot, cohere-ai, Diffbot, FacebookBot, FriendlyCrawler, Google-Extended, GPTBot, huggingface, ImagesiftBot, img2dataset, magpie-crawler, Meltwater, Neevabot, news-please, NewsNow, Nutch, omgili, omgilibot, panscient.com, Perplexity-ai, PerplexityBot, PetalBot, PiplBot, scoop.it, Scrapy, Seekr, SentiBot, SeznamBot, TurnitinBot, YouBot, ZumBot |
| ClaudeBot Anthropic · training | disallowed | ADmantX, AlphaBot, anthropic-ai, AwarioRssBot, AwarioSmartBot, BLEXBot, Buzzbot, Bytespider, CCBot, ChatGPT-User, claritybot, Claude-Web, ClaudeBot, cohere-ai, Diffbot, FacebookBot, FriendlyCrawler, Google-Extended, GPTBot, huggingface, ImagesiftBot, img2dataset, magpie-crawler, Meltwater, Neevabot, news-please, NewsNow, Nutch, omgili, omgilibot, panscient.com, Perplexity-ai, PerplexityBot, PetalBot, PiplBot, scoop.it, Scrapy, Seekr, SentiBot, SeznamBot, TurnitinBot, YouBot, ZumBot |
| Claude-SearchBot Anthropic · search | partial | Claude-SearchBot, OAI-SearchBot |
| Claude-User Anthropic · user_action | partial | * |
| Google-Extended Google · mixed | disallowed | ADmantX, AlphaBot, anthropic-ai, AwarioRssBot, AwarioSmartBot, BLEXBot, Buzzbot, Bytespider, CCBot, ChatGPT-User, claritybot, Claude-Web, ClaudeBot, cohere-ai, Diffbot, FacebookBot, FriendlyCrawler, Google-Extended, GPTBot, huggingface, ImagesiftBot, img2dataset, magpie-crawler, Meltwater, Neevabot, news-please, NewsNow, Nutch, omgili, omgilibot, panscient.com, Perplexity-ai, PerplexityBot, PetalBot, PiplBot, scoop.it, Scrapy, Seekr, SentiBot, SeznamBot, TurnitinBot, YouBot, ZumBot |
| Applebot-Extended Apple · training | partial | * |
| PerplexityBot Perplexity · search | disallowed | ADmantX, AlphaBot, anthropic-ai, AwarioRssBot, AwarioSmartBot, BLEXBot, Buzzbot, Bytespider, CCBot, ChatGPT-User, claritybot, Claude-Web, ClaudeBot, cohere-ai, Diffbot, FacebookBot, FriendlyCrawler, Google-Extended, GPTBot, huggingface, ImagesiftBot, img2dataset, magpie-crawler, Meltwater, Neevabot, news-please, NewsNow, Nutch, omgili, omgilibot, panscient.com, Perplexity-ai, PerplexityBot, PetalBot, PiplBot, scoop.it, Scrapy, Seekr, SentiBot, SeznamBot, TurnitinBot, YouBot, ZumBot |
| Perplexity-User Perplexity · user_action | partial | * |
| CCBot Common Crawl · archive | disallowed | ADmantX, AlphaBot, anthropic-ai, AwarioRssBot, AwarioSmartBot, BLEXBot, Buzzbot, Bytespider, CCBot, ChatGPT-User, claritybot, Claude-Web, ClaudeBot, cohere-ai, Diffbot, FacebookBot, FriendlyCrawler, Google-Extended, GPTBot, huggingface, ImagesiftBot, img2dataset, magpie-crawler, Meltwater, Neevabot, news-please, NewsNow, Nutch, omgili, omgilibot, panscient.com, Perplexity-ai, PerplexityBot, PetalBot, PiplBot, scoop.it, Scrapy, Seekr, SentiBot, SeznamBot, TurnitinBot, YouBot, ZumBot |
Recent observation timeline
The public timeline exposes hashes and metadata. Exact fetched bytes stay private to avoid redistributing source files.
- llms.txt · Unverified after HTTP response
· HTTP 301 ·
No retained artifact - robots.txt · Observed and parsed
· HTTP 200 ·
021adb45d8aa… - llms.txt · Unverified after HTTP response
· HTTP 301 ·
No retained artifact - robots.txt · Observed and parsed
· HTTP 200 ·
021adb45d8aa… - llms.txt · Unverified after HTTP response
· HTTP 301 ·
No retained artifact - robots.txt · Observed and parsed
· HTTP 200 ·
021adb45d8aa… - llms.txt · Unverified after HTTP response
· HTTP 301 ·
No retained artifact - robots.txt · Observed and parsed
· HTTP 200 ·
021adb45d8aa… - llms.txt · Unverified after HTTP response
· HTTP 301 ·
No retained artifact - robots.txt · Observed and parsed
· HTTP 200 ·
021adb45d8aa… - llms.txt · Unverified after HTTP response
· HTTP 301 ·
No retained artifact - robots.txt · Observed and parsed
· HTTP 200 ·
021adb45d8aa… - llms.txt · Unverified after HTTP response
· HTTP 301 ·
No retained artifact - robots.txt · Observed and parsed
· HTTP 200 ·
021adb45d8aa… - llms.txt · Unverified after HTTP response
· HTTP 301 ·
No retained artifact - robots.txt · Observed and parsed
· HTTP 200 ·
021adb45d8aa…
How to interpret this record
Robots rules address cooperative crawl access. Content Signals express downstream-use preferences. llms.txt is publisher guidance, while RSL can express machine-readable terms. CrawlLedger only reports what it observed and parsed.