How CrawlLedger turns a fetch into a verifiable record.
The archive is designed to answer a narrow factual question: what machine-readable policy signal did this domain publish when CrawlLedger checked, and what evidence is available to verify that observation?
1. A bounded watched set
The current pilot schedules exactly 20 reviewed domains. It does not accept arbitrary crawl requests, enumerate the web, or turn a user-submitted checker input into a network fetch. That boundary keeps operating cost predictable while the reliability gate is still running.
2. Robots comes first
For each domain, CrawlLedger requests /robots.txt with an honest crawler token and contact URL. The parser follows the Robots Exclusion Protocol's group-selection and longest-match behavior while separately preserving emerging Content-Signal: declarations and RSL License: discovery links. Only an explicit 404 or 410 is treated as no robots policy; server failures and unclear responses fail closed.
3. Secondary signals remain conditional
If the applicable robots rules permit the target, CrawlLedger may request /llms.txt and explicitly linked RSL documents. A cross-host RSL URL triggers an independent robots check on the destination host. The crawler honors bounded crawl delays, conditional requests, HTTPS-only URLs, public-address validation, redirect checks, and a 512 KiB response ceiling.
4. Bytes, parsing, and status are separate facts
The HTTP result, content type, artifact hash, parser version, parsed fields, and error state are stored separately. A response can therefore be “observed but content type differed,” “observed but parse failed,” “not observed at the expected URL,” or “unverified after a network failure” without collapsing those cases into a misleading yes-or-no answer.
5. Evidence is append-only
Unique response bytes receive a SHA-256 content address and remain private in R2. Each public event includes a chain hash derived from its canonical payload and the prior event hash for that domain and signal. D1 triggers reject updates and deletes to evidence rows, and application code never overwrites an artifact key.
6. Records are not legal conclusions
Robots instructions, Content Signals, llms.txt guidance, RSL terms, HTTP status codes, and Cloudflare product signals have different technical and legal meanings. CrawlLedger reports observed states. It does not authenticate publishers or crawlers, decide whether terms bind anyone, or say that a use is legally permitted.
Primary references
- RFC 9309 — Robots Exclusion Protocol
- Content Signals documentation and examples
- Really Simple Licensing 1.0 specification
- llms.txt proposal
- Cloudflare AI Crawl Control documentation
- Google's generative AI search optimization guide
Reproducibility comes from explicit limits, versioned interpretation, and preserved provenance—not from pretending every policy signal means the same thing.