robots.txt
Observed and parsed
- Observed
- HTTP
- 200
- Artifact
3073d1e371af…- Parser
- robots-rfc9309-rsl-content-signals-0.2.0+agents-2026-09-20
Domain record · observed metadata
Archive / internetcollective.org
internetcollective.org policy history.
Timestamped observations of public machine-readable policy files. The states below describe fetched records and parser results; they do not establish legal permission.
10observations
1signal types
0parsed changes
Nov 8, 2026last observed
Current observations
Current means the latest ledger event for each observed signal. Conditional 304 responses reuse the prior artifact hash while extending the event chain.
AI crawler rules in robots.txt
CrawlLedger evaluates a versioned registry of first-party-documented crawler tokens against the parsed robots groups.
Tracked crawler observations
These are parser results for the published robots rules, not authentication or legal conclusions.
| Crawler token | Observed state | Matched group |
|---|---|---|
| GPTBot OpenAI · training | disallowed | GPTBot |
| OAI-SearchBot OpenAI · search | allowed | * |
| OAI-AdsBot OpenAI · ads_validation | allowed | * |
| ChatGPT-User OpenAI · user_action | allowed | * |
| ClaudeBot Anthropic · training | disallowed | ClaudeBot |
| Claude-SearchBot Anthropic · search | allowed | * |
| Claude-User Anthropic · user_action | allowed | * |
| Google-Extended Google · mixed | disallowed | Google-Extended |
| Applebot-Extended Apple · training | disallowed | Applebot-Extended |
| PerplexityBot Perplexity · search | allowed | * |
| Perplexity-User Perplexity · user_action | allowed | * |
| CCBot Common Crawl · archive | disallowed | CCBot |
Observed Content-Signal declarations
These emerging robots.txt extensions state publisher preferences about downstream use. CrawlLedger records the declarations without treating them as enforcement or legal permission.
| User-agent group | Path | Search | AI input | AI training | Reuse |
|---|---|---|---|---|---|
| * | all paths | yes | not stated | no | reference |
Recent observation timeline
The public timeline exposes hashes and metadata. Exact fetched bytes stay private to avoid redistributing source files.
- robots.txt · Observed and parsed
· HTTP 200 ·
3073d1e371af… - robots.txt · Observed and parsed
· HTTP 200 ·
3073d1e371af… - robots.txt · Observed and parsed
· HTTP 200 ·
3073d1e371af… - robots.txt · Observed and parsed
· HTTP 200 ·
3073d1e371af… - robots.txt · Observed and parsed
· HTTP 200 ·
3073d1e371af… - robots.txt · Observed and parsed
· HTTP 200 ·
3073d1e371af… - robots.txt · Observed and parsed
· HTTP 200 ·
3073d1e371af… - robots.txt · Observed and parsed
· HTTP 200 ·
3073d1e371af… - robots.txt · Observed and parsed
· HTTP 200 ·
3073d1e371af… - robots.txt · Observed and parsed
· HTTP 200 ·
3073d1e371af…· new bytes observed
How to interpret this record
Robots rules address cooperative crawl access. Content Signals express downstream-use preferences. llms.txt is publisher guidance, while RSL can express machine-readable terms. CrawlLedger only reports what it observed and parsed.