CrawlLedger
Domain record · observed metadata

Archive / cloudflare.com

cloudflare.com policy history.

Timestamped observations of public machine-readable policy files. The states below describe fetched records and parser results; they do not establish legal permission.

Current JSONHistory JSON
20observations
2signal types
0parsed changes
Aug 13, 2026last observed

Current observations

Current means the latest ledger event for each observed signal. Conditional 304 responses reuse the prior artifact hash while extending the event chain.

llms.txt

Unverified after HTTP response

Observed
HTTP
301
Artifact
No retained artifact
Parser
Not applied

Observed source URL ↗

robots.txt

Observed and parsed

Observed
HTTP
200
Artifact
c6e360cc6e93…
Parser
robots-rfc9309-rsl-0.1.0

Observed source URL ↗

AI crawler rules in robots.txt

CrawlLedger evaluates a versioned registry of first-party-documented crawler tokens against the parsed robots groups.

Tracked crawler observations

These are parser results for the published robots rules, not authentication or legal conclusions.

Crawler tokenObserved stateMatched group
GPTBot
OpenAI · training		allowedGPTBot
OAI-SearchBot
OpenAI · search		allowed*
ChatGPT-User
OpenAI · user_action		allowedChatGPT-User
ClaudeBot
Anthropic · training		allowed*
Claude-SearchBot
Anthropic · search		allowed*
Claude-User
Anthropic · user_action		allowed*
Google-Extended
Google · mixed		allowedGoogle-Extended
Applebot-Extended
Apple · training		allowed*
PerplexityBot
Perplexity · search		allowedPerplexityBot
Perplexity-User
Perplexity · user_action		allowed*
CCBot
Common Crawl · archive		allowedCCBot

Recent observation timeline

The public timeline exposes hashes and metadata. Exact fetched bytes stay private to avoid redistributing source files.

  1. llms.txt · Unverified after HTTP response

    · HTTP 301 · No retained artifact

  2. robots.txt · Observed and parsed

    · HTTP 200 · c6e360cc6e93…

  3. llms.txt · Unverified after HTTP response

    · HTTP 301 · No retained artifact

  4. robots.txt · Observed and parsed

    · HTTP 200 · c6e360cc6e93…

  5. llms.txt · Unverified after HTTP response

    · HTTP 301 · No retained artifact

  6. robots.txt · Observed and parsed

    · HTTP 200 · c6e360cc6e93…

  7. llms.txt · Unverified after HTTP response

    · HTTP 301 · No retained artifact

  8. robots.txt · Observed and parsed

    · HTTP 200 · c6e360cc6e93…

  9. llms.txt · Unverified after HTTP response

    · HTTP 301 · No retained artifact

  10. robots.txt · Observed and parsed

    · HTTP 200 · c6e360cc6e93…

  11. llms.txt · Unverified after HTTP response

    · HTTP 301 · No retained artifact

  12. robots.txt · Observed and parsed

    · HTTP 200 · c6e360cc6e93…

  13. llms.txt · Unverified after HTTP response

    · HTTP 301 · No retained artifact

  14. robots.txt · Observed and parsed

    · HTTP 200 · c6e360cc6e93…

  15. llms.txt · Unverified after HTTP response

    · HTTP 301 · No retained artifact

  16. robots.txt · Observed and parsed

    · HTTP 200 · c6e360cc6e93…

How to interpret this record

Robots rules address cooperative crawl access. Content Signals express downstream-use preferences. llms.txt is publisher guidance, while RSL can express machine-readable terms. CrawlLedger only reports what it observed and parsed.

Read the methodCompare the signals