How CrawlLedger turns a fetch into a verifiable record.

The archive is designed to answer a narrow factual question: what machine-readable policy signal did this domain publish when CrawlLedger checked, and what evidence is available to verify that observation?

1. A bounded watched set

The current pilot schedules exactly 20 reviewed domains. It does not accept arbitrary crawl requests, enumerate the web, or turn a user-submitted checker input into a network fetch. The 14-day reliability gate passed; the fixed boundary still keeps operating cost predictable while source-list rights and any expansion remain separately gated.

2. Robots comes first

For each domain, CrawlLedger requests /robots.txt with an honest crawler token and contact URL. The parser follows the Robots Exclusion Protocol's group-selection and longest-match behavior while separately preserving emerging Content-Signal: declarations and RSL License: discovery links. Only an explicit 404 or 410 is treated as no robots policy; server failures and unclear responses fail closed.

3. Secondary signals remain conditional

If the applicable robots rules permit the target, CrawlLedger may request /llms.txt and explicitly linked RSL documents. A cross-host RSL URL triggers an independent robots check on the destination host. The crawler honors bounded crawl delays, conditional requests, HTTPS-only URLs, public-address validation, redirect checks, and a 512 KiB response ceiling.

4. Bytes, parsing, and status are separate facts

The HTTP result, content type, artifact hash, parser version, parsed fields, and error state are stored separately. A response can therefore be “observed but content type differed,” “observed but parse failed,” “not observed at the expected URL,” or “unverified after a network failure” without collapsing those cases into a misleading yes-or-no answer.

5. Evidence is append-only

Unique response bytes receive a SHA-256 content address and remain private in R2. Each public event includes a chain hash derived from its canonical payload and the prior event hash for that domain and signal. D1 triggers reject updates and deletes to evidence rows, and application code never overwrites an artifact key.

6. Records are not legal conclusions

Robots instructions, Content Signals, llms.txt guidance, RSL terms, HTTP status codes, and Cloudflare product signals have different technical and legal meanings. CrawlLedger reports observed states. It does not authenticate publishers or crawlers, decide whether terms bind anyone, or say that a use is legally permitted.

7. Publisher, automation, and review

CrawlLedger is the named publisher. Observation pages and evidence briefs are generated automatically from append-only metadata; interpretive guides are tied to primary sources and dated registry reviews. Source changes pass tests and a reviewable release gate before deployment. Documentation can be corrected prospectively, while historical evidence rows remain untouched.

Primary references

• RFC 9309 — Robots Exclusion Protocol
• Content Signals documentation and examples
• Really Simple Licensing 1.0 specification
• llms.txt proposal
• Cloudflare AI Crawl Control documentation
• Google's generative AI search optimization guide

Reproducibility comes from explicit limits, versioned interpretation, and preserved provenance—not from pretending every policy signal means the same thing.

How this page was produced: technical claims are linked to primary sources, parser behavior is covered by executable tests, and the crawler registry now fails its maintenance gate when a source review becomes more than 90 days old.