robots.txt, Content Signals, llms.txt, and RSL solve different problems.
Treating these signals as interchangeable creates bad audits. They address crawl access, downstream use preferences, guidance for language-model systems, and licensing terms at different layers.
| Signal | Primary job | What it does not prove | CrawlLedger state |
|---|---|---|---|
robots.txt | Advisory crawl rules selected by user-agent token and URL path | Crawler identity, enforcement, ownership, or legal permission | Observed and parsed against a versioned registry |
Content-Signal | Emerging robots.txt declarations for search, AI input, model training, and reuse preferences | Enforcement, universal support, or a binding legal outcome | Declarations preserved separately from access rules |
llms.txt | Proposed publisher guidance and curated links for language-model systems | Access permission, search ranking, or adoption by a specific provider | Presence and structure observed when robots policy permits |
| RSL | Discovery and expression of machine-readable usage, licensing, payment, and reporting terms | That terms bind a party or that a crawler obtained authorization | Linked documents fetched conditionally and parsed with versioned rules |
| Pay Per Crawl | Cloudflare-mediated crawler charging and authenticated discovery | Absence from an ordinary HTTP 200 response | Unverified until Web Bot Auth and the Discovery API are active |
robots.txt remains the crawl-control baseline
RFC 9309 standardizes how cooperative crawlers retrieve and interpret robots rules. Specific user-agent groups can override the wildcard group, and the most specific matching path rule wins. A robots result is still advisory: a spoofed or non-compliant client can ignore it, so a policy audit must not claim more than the file demonstrates.
Content Signals describe downstream-use preferences
The emerging Content-Signal: extension can state separate preferences for search indexing, real-time AI input, training, and newer reuse levels. It does not replace Allow or Disallow, and it is not part of RFC 9309. Read the detailed Content Signals guide.
llms.txt is guidance, not a Google ranking shortcut
The llms.txt proposal gives publishers a concise Markdown index of important resources. It may be useful to systems that choose to read it, but it is not a substitute for normal crawlability, semantic HTML, internal links, canonical URLs, or sitemaps. Google's July 2026 guidance explicitly says Google Search ignores llms.txt for ranking and generative AI visibility. CrawlLedger records the signal without inflating its authority.
RSL adds a licensing vocabulary and discovery path
RSL 1.0 defines XML terms and several discovery mechanisms, including License: records in robots.txt. A license URL can live on another host, which is why CrawlLedger checks that destination's robots policy before requesting the document. The parser records permits, prohibits, payment structures, reporting requirements, and warnings as machine-readable observations—not as a legal opinion.
What a reliable monitor should preserve
- The requested and final URL, fetch time, HTTP status, and bounded response headers.
- A content hash and byte count that distinguish new bytes from a conditional
304. - The parser and crawler-registry versions used for interpretation.
- Distinct states for absent, failed, too large, wrong content type, skipped, and unverified.
- A history that cannot silently rewrite earlier observations.
Use the signals together, but keep their boundaries visible
A practical publisher audit starts with robots rules, checks whether important search and agent paths remain crawlable, reviews llms.txt as optional guidance, and treats RSL as a separate terms-discovery layer. Enforcement, authenticated bot identity, contracts, and content rights require additional systems and human judgment.