CrawlLedger
Domain record · observed metadata

Archive / google.com

google.com policy history.

Timestamped observations of public machine-readable policy files. The states below describe fetched records and parser results; they do not establish legal permission.

Current JSONHistory JSON
20observations
2signal types
0parsed changes
Aug 13, 2026last observed

Current observations

Current means the latest ledger event for each observed signal. Conditional 304 responses reuse the prior artifact hash while extending the event chain.

llms.txt

Not observed at the expected URL

Observed
HTTP
404
Artifact
47d71ed470df…
Parser
Not applied

Observed source URL ↗

robots.txt

Observed and parsed

Observed
HTTP
200
Artifact
1aba570a1b32…
Parser
robots-rfc9309-rsl-0.1.0

Observed source URL ↗

AI crawler rules in robots.txt

CrawlLedger evaluates a versioned registry of first-party-documented crawler tokens against the parsed robots groups.

Tracked crawler observations

These are parser results for the published robots rules, not authentication or legal conclusions.

Crawler tokenObserved stateMatched group
GPTBot
OpenAI · training		partial*, Yandex
OAI-SearchBot
OpenAI · search		partial*, Yandex
ChatGPT-User
OpenAI · user_action		partial*, Yandex
ClaudeBot
Anthropic · training		partial*, Yandex
Claude-SearchBot
Anthropic · search		partial*, Yandex
Claude-User
Anthropic · user_action		partial*, Yandex
Google-Extended
Google · mixed		partial*, Yandex
Applebot-Extended
Apple · training		partial*, Yandex
PerplexityBot
Perplexity · search		partial*, Yandex
Perplexity-User
Perplexity · user_action		partial*, Yandex
CCBot
Common Crawl · archive		partial*, Yandex

Recent observation timeline

The public timeline exposes hashes and metadata. Exact fetched bytes stay private to avoid redistributing source files.

  1. llms.txt · Not observed at the expected URL

    · HTTP 404 · 47d71ed470df…

  2. robots.txt · Observed and parsed

    · HTTP 200 · 1aba570a1b32…

  3. llms.txt · Not observed at the expected URL

    · HTTP 404 · 47d71ed470df…

  4. robots.txt · Observed and parsed

    · HTTP 200 · 1aba570a1b32…

  5. llms.txt · Not observed at the expected URL

    · HTTP 404 · 47d71ed470df…

  6. robots.txt · Observed and parsed

    · HTTP 200 · 1aba570a1b32…

  7. llms.txt · Not observed at the expected URL

    · HTTP 404 · 47d71ed470df…

  8. robots.txt · Observed and parsed

    · HTTP 200 · 1aba570a1b32…

  9. llms.txt · Not observed at the expected URL

    · HTTP 404 · 47d71ed470df…

  10. robots.txt · Observed and parsed

    · HTTP 200 · 1aba570a1b32…

  11. llms.txt · Not observed at the expected URL

    · HTTP 404 · 47d71ed470df…

  12. robots.txt · Observed and parsed

    · HTTP 200 · 1aba570a1b32…

  13. llms.txt · Not observed at the expected URL

    · HTTP 404 · 47d71ed470df…

  14. robots.txt · Observed and parsed

    · HTTP 200 · 1aba570a1b32…

  15. llms.txt · Not observed at the expected URL

    · HTTP 404 · 47d71ed470df…

  16. robots.txt · Observed and parsed

    · HTTP 200 · 1aba570a1b32…

How to interpret this record

Robots rules address cooperative crawl access. Content Signals express downstream-use preferences. llms.txt is publisher guidance, while RSL can express machine-readable terms. CrawlLedger only reports what it observed and parsed.

Read the methodCompare the signals