Append-only observatory · reliability gate passed

A public change log for the machine-readable web.

CrawlLedger preserves what domains publish in robots.txt, Content Signals, llms.txt, and RSL licensing files—plus when it was observed, how it parsed, and how each record links to the one before it.

Open an evidence brief View live research Check a policy

20watched domains

1,911observations

182unique artifacts

12parsed changes

Evidence that stays inspectable

A normal checker tells you what a file says now. CrawlLedger keeps the observation metadata needed to distinguish a current result, an unchanged response, a parse failure, and a fetch skipped by robots policy.

01 / Observe narrowly

One queued domain at a time, limited to public policy files in a 20-domain archive that passed its 14-day unattended reliability gate.

02 / Preserve provenance

Content hashes, bounded response metadata, parser versions, and per-domain signal chains remain attached to every event.

03 / Report carefully

Pages use factual states such as observed, not observed, unverified, parse failed, or skipped—not legal conclusions.

One robots.txt file, many different AI systems

Training crawlers, search crawlers, user-triggered fetchers, and control tokens are not interchangeable. CrawlLedger keeps their first-party descriptions and observed policy states separate.

Training

Compare rules for GPTBot, ClaudeBot, and Applebot-Extended.

Search and submitted ads

Inspect separate controls for OAI-SearchBot, OAI-AdsBot, Claude-SearchBot, and PerplexityBot.

Content use

Understand the emerging Content-Signal fields for search, AI input, training, and reuse.

Open the crawler reference Build a policy draft

Recent parsed artifact changes

Only post-baseline byte changes that produced a parsed result are shown here. Changing error pages and routine 304 responses are excluded.

oreilly.com