robots.txt
Observed and parsed
- Observed
- HTTP
- 200
- Artifact
41aea4211024…- Parser
- robots-rfc9309-rsl-0.1.0
Domain record · observed metadata
Archive / reddit.com
reddit.com policy history.
Timestamped observations of public machine-readable policy files. The states below describe fetched records and parser results; they do not establish legal permission.
10observations
1signal types
0parsed changes
Aug 13, 2026last observed
Current observations
Current means the latest ledger event for each observed signal. Conditional 304 responses reuse the prior artifact hash while extending the event chain.
AI crawler rules in robots.txt
CrawlLedger evaluates a versioned registry of first-party-documented crawler tokens against the parsed robots groups.
Tracked crawler observations
These are parser results for the published robots rules, not authentication or legal conclusions.
| Crawler token | Observed state | Matched group |
|---|---|---|
| GPTBot OpenAI · training | disallowed | * |
| OAI-SearchBot OpenAI · search | disallowed | * |
| ChatGPT-User OpenAI · user_action | disallowed | * |
| ClaudeBot Anthropic · training | disallowed | * |
| Claude-SearchBot Anthropic · search | disallowed | * |
| Claude-User Anthropic · user_action | disallowed | * |
| Google-Extended Google · mixed | disallowed | * |
| Applebot-Extended Apple · training | disallowed | * |
| PerplexityBot Perplexity · search | disallowed | * |
| Perplexity-User Perplexity · user_action | disallowed | * |
| CCBot Common Crawl · archive | disallowed | * |
Recent observation timeline
The public timeline exposes hashes and metadata. Exact fetched bytes stay private to avoid redistributing source files.
- robots.txt · Observed and parsed
· HTTP 200 ·
41aea4211024… - robots.txt · Observed and parsed
· HTTP 200 ·
41aea4211024… - robots.txt · Observed and parsed
· HTTP 200 ·
41aea4211024… - robots.txt · Observed and parsed
· HTTP 200 ·
41aea4211024… - robots.txt · Observed and parsed
· HTTP 200 ·
41aea4211024… - robots.txt · Observed and parsed
· HTTP 200 ·
41aea4211024… - robots.txt · Observed and parsed
· HTTP 200 ·
41aea4211024… - robots.txt · Observed and parsed
· HTTP 200 ·
41aea4211024… - robots.txt · Observed and parsed
· HTTP 200 ·
41aea4211024… - robots.txt · Observed and parsed
· HTTP 200 ·
41aea4211024…· new bytes observed
How to interpret this record
Robots rules address cooperative crawl access. Content Signals express downstream-use preferences. llms.txt is publisher guidance, while RSL can express machine-readable terms. CrawlLedger only reports what it observed and parsed.