CrawlLedger
Domain record · observed metadata

Archive / usatoday.com

usatoday.com policy history.

Timestamped observations of public machine-readable policy files. The states below describe fetched records and parser results; they do not establish legal permission.

Current JSONHistory JSON
18observations
2signal types
2parsed changes
Aug 12, 2026last observed

Current observations

Current means the latest ledger event for each observed signal. Conditional 304 responses reuse the prior artifact hash while extending the event chain.

llms.txt

Unverified after HTTP response

Observed
HTTP
301
Artifact
No retained artifact
Parser
Not applied

Observed source URL ↗

robots.txt

Observed and parsed

Observed
HTTP
200
Artifact
9bf8ba4edf17…
Parser
robots-rfc9309-rsl-0.1.0

Observed source URL ↗

AI crawler rules in robots.txt

CrawlLedger evaluates a versioned registry of first-party-documented crawler tokens against the parsed robots groups.

Tracked crawler observations

These are parser results for the published robots rules, not authentication or legal conclusions.

Crawler tokenObserved stateMatched group
GPTBot
OpenAI · training		disallowedGPTBot
OAI-SearchBot
OpenAI · search		disallowedOAI-SearchBot
ChatGPT-User
OpenAI · user_action		disallowedChatGPT-User
ClaudeBot
Anthropic · training		disallowedClaudeBot
Claude-SearchBot
Anthropic · search		disallowedClaude-SearchBot
Claude-User
Anthropic · user_action		disallowedClaude-User
Google-Extended
Google · mixed		disallowedGoogle-Extended
Applebot-Extended
Apple · training		disallowedApplebot-Extended
PerplexityBot
Perplexity · search		disallowedPerplexityBot
Perplexity-User
Perplexity · user_action		disallowedPerplexity-User
CCBot
Common Crawl · archive		disallowedCCBot

Recent observation timeline

The public timeline exposes hashes and metadata. Exact fetched bytes stay private to avoid redistributing source files.

  1. llms.txt · Unverified after HTTP response

    · HTTP 301 · No retained artifact

  2. robots.txt · Observed and parsed

    · HTTP 200 · 9bf8ba4edf17…

  3. llms.txt · Unverified after HTTP response

    · HTTP 301 · No retained artifact

  4. robots.txt · Observed and parsed

    · HTTP 200 · 9bf8ba4edf17… · new bytes observed

  5. llms.txt · Unverified after HTTP response

    · HTTP 301 · No retained artifact

  6. robots.txt · Observed and parsed

    · HTTP 200 · b0d291be5914…

  7. llms.txt · Unverified after HTTP response

    · HTTP 301 · No retained artifact

  8. robots.txt · Observed and parsed

    · HTTP 200 · b0d291be5914…

  9. llms.txt · Unverified after HTTP response

    · HTTP 301 · No retained artifact

  10. robots.txt · Observed and parsed

    · HTTP 200 · b0d291be5914…

  11. llms.txt · Unverified after HTTP response

    · HTTP 301 · No retained artifact

  12. robots.txt · Observed and parsed

    · HTTP 200 · b0d291be5914… · new bytes observed

  13. llms.txt · Unverified after HTTP response

    · HTTP 301 · No retained artifact

  14. robots.txt · Observed and parsed

    · HTTP 200 · 4fb71131ab42…

  15. llms.txt · Unverified after HTTP response

    · HTTP 301 · No retained artifact

  16. robots.txt · Observed and parsed

    · HTTP 200 · 4fb71131ab42…

How to interpret this record

Robots rules address cooperative crawl access. Content Signals express downstream-use preferences. llms.txt is publisher guidance, while RSL can express machine-readable terms. CrawlLedger only reports what it observed and parsed.

Read the methodCompare the signals