CrawlLedger
Domain record · observed metadata

Archive / commoncrawl.org

commoncrawl.org policy history.

Timestamped observations of public machine-readable policy files. The states below describe fetched records and parser results; they do not establish legal permission.

Current JSONHistory JSON
20observations
2signal types
0parsed changes
Aug 13, 2026last observed

Current observations

Current means the latest ledger event for each observed signal. Conditional 304 responses reuse the prior artifact hash while extending the event chain.

llms.txt

Not observed at the expected URL

Observed
HTTP
404
Artifact
c1b8467efd09…
Parser
Not applied

Observed source URL ↗

robots.txt

Observed and parsed

Observed
HTTP
200
Artifact
87e17d4f9c85…
Parser
robots-rfc9309-rsl-0.1.0

Observed source URL ↗

AI crawler rules in robots.txt

CrawlLedger evaluates a versioned registry of first-party-documented crawler tokens against the parsed robots groups.

Tracked crawler observations

These are parser results for the published robots rules, not authentication or legal conclusions.

Crawler tokenObserved stateMatched group
GPTBot
OpenAI · training		partial*
OAI-SearchBot
OpenAI · search		partial*
ChatGPT-User
OpenAI · user_action		partial*
ClaudeBot
Anthropic · training		partial*
Claude-SearchBot
Anthropic · search		partial*
Claude-User
Anthropic · user_action		partial*
Google-Extended
Google · mixed		partial*
Applebot-Extended
Apple · training		partial*
PerplexityBot
Perplexity · search		partial*
Perplexity-User
Perplexity · user_action		partial*
CCBot
Common Crawl · archive		partial*

Recent observation timeline

The public timeline exposes hashes and metadata. Exact fetched bytes stay private to avoid redistributing source files.

  1. llms.txt · Not observed at the expected URL

    · HTTP 404 · c1b8467efd09… · new bytes observed

  2. robots.txt · Observed and parsed

    · HTTP 200 · 87e17d4f9c85…

  3. llms.txt · Not observed at the expected URL

    · HTTP 404 · ffa87ff7e775… · new bytes observed

  4. robots.txt · Observed and parsed

    · HTTP 200 · 87e17d4f9c85…

  5. llms.txt · Not observed at the expected URL

    · HTTP 404 · 9fd51c0ba05b… · new bytes observed

  6. robots.txt · Observed and parsed

    · HTTP 200 · 87e17d4f9c85…

  7. llms.txt · Not observed at the expected URL

    · HTTP 404 · b05bb082fa37…

  8. robots.txt · Observed and parsed

    · HTTP 200 · 87e17d4f9c85…

  9. llms.txt · Not observed at the expected URL

    · HTTP 404 · b05bb082fa37…

  10. robots.txt · Observed and parsed

    · HTTP 200 · 87e17d4f9c85…

  11. llms.txt · Not observed at the expected URL

    · HTTP 404 · b05bb082fa37…

  12. robots.txt · Observed and parsed

    · HTTP 200 · 87e17d4f9c85…

  13. llms.txt · Not observed at the expected URL

    · HTTP 404 · b05bb082fa37…

  14. robots.txt · Observed and parsed

    · HTTP 200 · 87e17d4f9c85…

  15. llms.txt · Not observed at the expected URL

    · HTTP 404 · b05bb082fa37…

  16. robots.txt · Observed and parsed

    · HTTP 200 · 87e17d4f9c85…

How to interpret this record

Robots rules address cooperative crawl access. Content Signals express downstream-use preferences. llms.txt is publisher guidance, while RSL can express machine-readable terms. CrawlLedger only reports what it observed and parsed.

Read the methodCompare the signals