llms.txt
Not observed at the expected URL
- Observed
- HTTP
- 404
- Artifact
56286d940a1f…- Parser
- Not applied
Domain record · observed metadata
Archive / rslcollective.org
rslcollective.org policy history.
Timestamped observations of public machine-readable policy files. The states below describe fetched records and parser results; they do not establish legal permission.
40observations
3signal types
0parsed changes
Aug 13, 2026last observed
Current observations
Current means the latest ledger event for each observed signal. Conditional 304 responses reuse the prior artifact hash while extending the event chain.
robots.txt
Observed; bytes unchanged
- Observed
- HTTP
- 304
- Artifact
a311664af788…- Parser
- robots-rfc9309-rsl-0.1.0
RSL
Observed; bytes unchanged
- Observed
- HTTP
- 304
- Artifact
a351dca7a440…- Parser
- rsl-1.0-errata-2026-07-18-0.2.0
AI crawler rules in robots.txt
CrawlLedger evaluates a versioned registry of first-party-documented crawler tokens against the parsed robots groups.
Tracked crawler observations
These are parser results for the published robots rules, not authentication or legal conclusions.
| Crawler token | Observed state | Matched group |
|---|---|---|
| GPTBot OpenAI · training | allowed | * |
| OAI-SearchBot OpenAI · search | allowed | * |
| ChatGPT-User OpenAI · user_action | allowed | * |
| ClaudeBot Anthropic · training | allowed | * |
| Claude-SearchBot Anthropic · search | allowed | * |
| Claude-User Anthropic · user_action | allowed | * |
| Google-Extended Google · mixed | allowed | * |
| Applebot-Extended Apple · training | allowed | * |
| PerplexityBot Perplexity · search | allowed | * |
| Perplexity-User Perplexity · user_action | allowed | * |
| CCBot Common Crawl · archive | allowed | * |
Observed RSL discovery links
Recent observation timeline
The public timeline exposes hashes and metadata. Exact fetched bytes stay private to avoid redistributing source files.
- RSL · Observed; bytes unchanged
· HTTP 304 ·
a351dca7a440… - llms.txt · Not observed at the expected URL
· HTTP 404 ·
56286d940a1f…· new bytes observed - robots.txt · Observed; bytes unchanged
· HTTP 304 ·
a311664af788… - robots.txt · Observed; bytes unchanged
· HTTP 304 ·
a311664af788… - RSL · Observed and parsed
· HTTP 200 ·
a351dca7a440… - llms.txt · Not observed at the expected URL
· HTTP 404 ·
d479191921f0… - robots.txt · Observed; bytes unchanged
· HTTP 304 ·
a311664af788… - robots.txt · Observed; bytes unchanged
· HTTP 304 ·
a311664af788… - RSL · Observed; bytes unchanged
· HTTP 304 ·
a351dca7a440… - llms.txt · Not observed at the expected URL
· HTTP 404 ·
d479191921f0… - robots.txt · Observed; bytes unchanged
· HTTP 304 ·
a311664af788… - robots.txt · Observed; bytes unchanged
· HTTP 304 ·
a311664af788… - RSL · Observed and parsed
· HTTP 200 ·
a351dca7a440… - llms.txt · Not observed at the expected URL
· HTTP 404 ·
d479191921f0… - robots.txt · Observed; bytes unchanged
· HTTP 304 ·
a311664af788… - robots.txt · Observed; bytes unchanged
· HTTP 304 ·
a311664af788…
How to interpret this record
Robots rules address cooperative crawl access. Content Signals express downstream-use preferences. llms.txt is publisher guidance, while RSL can express machine-readable terms. CrawlLedger only reports what it observed and parsed.