# CrawlLedger

> Append-only observations of public machine-readable web policy signals. Records describe fetched responses and parser output, not legal permission.

## Core pages

- [Archive](https://crawlledger.net/domains): Canonical HTML pages for observed domains
- [AI crawler directory](https://crawlledger.net/ai-crawlers): First-party-sourced crawler purposes and current policy observations
- [Observed changes](https://crawlledger.net/changes): Parsed post-baseline artifact changes
- [Methodology](https://crawlledger.net/methodology): Fetch, parsing, provenance, and status boundaries
- [Crawler policy](https://crawlledger.net/crawler): CrawlLedgerBot identity and operating limits

## Tools and documentation

- [AI robots.txt policy checker](https://crawlledger.net/tools/robots-txt-ai-policy-checker): Parse pasted robots.txt and Content Signals without a network fetch
- [robots.txt, Content Signals, llms.txt, and RSL](https://crawlledger.net/guides/ai-crawler-policy-files): Source-backed comparison guide
- [Content Signals guide](https://crawlledger.net/guides/content-signals-robots-txt): Syntax, meanings, and enforcement limits
- [API documentation](https://crawlledger.net/api): Public JSON endpoints and response boundaries
- [OpenAPI](https://crawlledger.net/openapi.json): Machine-readable API description

## Feeds and discovery

- [XML sitemap](https://crawlledger.net/sitemap.xml): Canonical indexable URLs
- [Atom change feed](https://crawlledger.net/feed.xml): Recent parsed artifact changes
- [Health](https://crawlledger.net/v1/health): Live service and archive status
- [Tracked agents](https://crawlledger.net/v1/agents): Versioned crawler-token registry
- [Policy landscape](https://crawlledger.net/v1/policy-landscape): Current observations grouped by crawler token

## Important limitations

- Raw fetched bytes are retained privately and are not redistributed by default.
- Pay Per Crawl is unverified until Cloudflare Web Bot Auth and authenticated Discovery API access are active.
- Content Signals, robots rules, and llms.txt are recorded as published signals, not proof of legal permission or operator compliance.