Evidence briefs / commoncrawl.org
commoncrawl.org policy evidence brief.
A bounded report of what CrawlLedger observed, when it observed it, how the versioned parser classified it, and which public hashes identify the retained artifacts.
Generated on request from append-only metadata. Refreshing this page does not crawl the domain.
Executive evidence summary
These counts describe the bounded CrawlLedger record for this domain. They do not establish ownership, intent, crawler identity, compliance, contract, or legal permission.
Current observation register
The artifact SHA-256 identifies retained bytes without exposing a private storage location or redistributing the source body.
| Signal | Observed | HTTP | State | Artifact SHA-256 | Parser |
|---|---|---|---|---|---|
| llms.txt | 404 | Not observed at the expected URL | c56033ad28065d877c2d04775f80e45700a8cb6ab7d9d8f77ba2a40e80b56f9d | Not applied | |
| robots.txt | 200 | Observed and parsed | 13ac8bbfcbc869e1eaa4b87e5a805cb83a9b7b6ff0a4471db382793e88263ebb | robots-rfc9309-rsl-content-signals-0.2.0+agents-2026-09-20 |
Current robots.txt interpretation
Each state uses the registry and parser version attached to the observation. “Allowed” is a parser result, not a grant or verified crawler behavior.
Tracked crawler observations
These are parser results for the published robots rules, not authentication or legal conclusions.
| Crawler token | Observed state | Matched group |
|---|---|---|
| GPTBot OpenAI · training | partial | * |
| OAI-SearchBot OpenAI · search | partial | * |
| OAI-AdsBot OpenAI · ads_validation | partial | * |
| ChatGPT-User OpenAI · user_action | partial | * |
| ClaudeBot Anthropic · training | partial | * |
| Claude-SearchBot Anthropic · search | partial | * |
| Claude-User Anthropic · user_action | partial | * |
| Google-Extended Google · mixed | partial | * |
| Applebot-Extended Apple · training | partial | * |
| PerplexityBot Perplexity · search | partial | * |
| Perplexity-User Perplexity · user_action | partial | * |
| CCBot Common Crawl · archive | partial | * |
Recent append-only timeline
Chain identifiers make replacement or branching detectable. A conditional 304 can extend the chain while reusing the prior artifact hash.
| Observed | Signal | State | Changed | Chain hash | Previous chain |
|---|---|---|---|---|---|
| llms.txt | Not observed at the expected URL | No | 17e6fa8c0869a41f54a3ea46506bdff63149ac8b08a12721016f839541d7906a | f9394a24136560370fa7bf7ee5b921057444913e8a8512a9885ff2d4fd531335 | |
| robots.txt | Observed and parsed | No | 062aeb79b39a245139ee2142c35ec68fad0b149b42d889307f932c466845e581 | 31f1bcd951722203e66236fec704eb09dabaee851ea724359fef6e839a4126f9 | |
| llms.txt | Not observed at the expected URL | No | f9394a24136560370fa7bf7ee5b921057444913e8a8512a9885ff2d4fd531335 | b910d5ac19d607cca5dc96caaf526430d097f6e3c5db0e4e6976dc629383b0d4 | |
| robots.txt | Observed and parsed | No | 31f1bcd951722203e66236fec704eb09dabaee851ea724359fef6e839a4126f9 | 4b5bc72c37fff7724be72daef210601eaa1fe67222c70613f044b3cb6975bb37 | |
| llms.txt | Not observed at the expected URL | No | b910d5ac19d607cca5dc96caaf526430d097f6e3c5db0e4e6976dc629383b0d4 | ea119e3c55b3864d77df23376a564e8dcf8f093b218e9c678dd7baa7f9e5324f | |
| robots.txt | Observed and parsed | No | 4b5bc72c37fff7724be72daef210601eaa1fe67222c70613f044b3cb6975bb37 | 4998f209780d21b9b6fd6348545385dfaf4051c565ae6d17a6e172683b3e02ea | |
| llms.txt | Not observed at the expected URL | No | ea119e3c55b3864d77df23376a564e8dcf8f093b218e9c678dd7baa7f9e5324f | 9ecb21b1c66418ecb7abcd5ea76b7c49badfeccfd0095beaa6f68913b05ef415 | |
| robots.txt | Observed and parsed | No | 4998f209780d21b9b6fd6348545385dfaf4051c565ae6d17a6e172683b3e02ea | 1319c8f8d7ca7dc90dcab3054a2b04c134877c8ef7459ca29f65467c8d52e21e | |
| llms.txt | Not observed at the expected URL | No | 9ecb21b1c66418ecb7abcd5ea76b7c49badfeccfd0095beaa6f68913b05ef415 | ff4b259500a70d512be752ec00e4dd4be0faa8a16dac7b32dbe4ec963f7d1f51 | |
| robots.txt | Observed and parsed | No | 1319c8f8d7ca7dc90dcab3054a2b04c134877c8ef7459ca29f65467c8d52e21e | 8967c38fe67d71e28c80f2b1cccdf58a0a63267210d66501011fab86b5f98a91 | |
| llms.txt | Not observed at the expected URL | Yes | ff4b259500a70d512be752ec00e4dd4be0faa8a16dac7b32dbe4ec963f7d1f51 | 83233e6d2f0a9d7fa008c077a01b1e473dba1d9667c461ac12110d2254350c8b | |
| robots.txt | Observed and parsed | No | 8967c38fe67d71e28c80f2b1cccdf58a0a63267210d66501011fab86b5f98a91 | 9056ada7017caf2c71f8a74dc7bd067ecc466c3961957d52203f263160c77162 | |
| llms.txt | Not observed at the expected URL | No | 83233e6d2f0a9d7fa008c077a01b1e473dba1d9667c461ac12110d2254350c8b | fdf462630b9267f66b72beebfc415113d9685a7ed9f435f35c1c6e81c1ffb6ea | |
| robots.txt | Observed and parsed | No | 9056ada7017caf2c71f8a74dc7bd067ecc466c3961957d52203f263160c77162 | 91b84e8f8e14ab995d486acc91fbbb7e999a01e2ff7b498faa7456993c00db22 | |
| llms.txt | Not observed at the expected URL | No | fdf462630b9267f66b72beebfc415113d9685a7ed9f435f35c1c6e81c1ffb6ea | 2a61caa953febceac5b9f0e1d7f2ed9babfe714dd5dd56432645d12c89f03deb | |
| robots.txt | Observed and parsed | No | 91b84e8f8e14ab995d486acc91fbbb7e999a01e2ff7b498faa7456993c00db22 | 9ee5d7e49fdfe420d89770b9e036e621997d9e806bea7fd66341b6cbcc12d59a | |
| llms.txt | Not observed at the expected URL | No | 2a61caa953febceac5b9f0e1d7f2ed9babfe714dd5dd56432645d12c89f03deb | e2d2ef767018bb57735911d4c1a245cf38162041f8bbb85cfe27626c98038af8 | |
| robots.txt | Observed and parsed | No | 9ee5d7e49fdfe420d89770b9e036e621997d9e806bea7fd66341b6cbcc12d59a | e7e865146bc121211b6dba805fa7177928e0e457e7679345cf21aba28d3a99b7 | |
| llms.txt | Not observed at the expected URL | No | e2d2ef767018bb57735911d4c1a245cf38162041f8bbb85cfe27626c98038af8 | 22e26ca8eab07f315c3dceb9901e3fd4ca8e886d07e3f325b6141ae78081a2aa | |
| robots.txt | Observed and parsed | No | e7e865146bc121211b6dba805fa7177928e0e457e7679345cf21aba28d3a99b7 | 411822c4f5c04d730a16e2e792219d0d51d6316c62294d716e0d362f6482c5f8 |
Method and limitations
CrawlLedger observes narrow public policy endpoints, records bounded response metadata, retains source bytes privately by content hash, and appends parser output. It does not authenticate requesting bots, infer unstated intent, or decide legal permission.
Included
Public metadata, exact hashes, parser versions, signal states, and recent chain links from the fixed cohort.
Not included
Private artifact keys, raw source redistribution, ordinary page content, legal conclusions, or customer-specific monitoring.
Verify further
Use the methodology, current JSON, and history JSON to reproduce the public layer.
How this report was produced: CrawlLedger assembled it automatically from append-only observation metadata and a versioned parser registry. Documentation and product boundaries were source-reviewed on Sep 20, 2026. No new network fetch, customer data, or legal interpretation was added for this export.