Content Signals add a use-preference layer to robots.txt.
Allow and Disallow answer whether a cooperative crawler may fetch a path. The emerging Content-Signal: extension tries to answer a different question: how should accessed content be used?
Content Signals are not part of RFC 9309, do not technically block a request, and are not universally honored. They express publisher preferences. CrawlLedger records what was published without deciding whether it creates legal permission or obligation.
The four currently documented fields
| Field | Values | Documented meaning |
|---|---|---|
search | yes / no | Building a search index and returning links or short excerpts; Cloudflare's policy says this does not include AI-generated summaries. |
ai-input | yes / no | Using content as real-time model input, including retrieval, grounding, or generative answers. |
ai-train | yes / no | Training or fine-tuning AI models. |
use | immediate, reference, or full | A newer Cloudflare extension describing increasing levels of retention and reuse: immediate interaction, reference/index/excerpt, or summarization/reproduction. |
Basic search-and-AI-answer example
The following example declares yes for ordinary search and real-time AI answers, while declaring ai-train=no for training:
User-agent: *
Content-Signal: search=yes, ai-input=yes, ai-train=no, use=reference
Allow: /The access and use layers are deliberately separate. Allow: / addresses fetching. The Content Signal fields describe requested downstream treatment.
Preferences can be scoped by crawler and path
Content Signals can appear within a named user-agent group. Current published examples also show an optional path before the fields:
User-agent: OAI-SearchBot
Content-Signal: search=yes, ai-input=no, ai-train=no
Allow: /
User-agent: *
Content-Signal: /research/ search=yes, ai-input=yes, ai-train=no
Allow: /research/Because this extension is evolving, publishers should validate current syntax against the primary specification and the intended crawler's documentation before relying on it.
Content Signals do not replace crawler-specific controls
Operators still document separate robots tokens for different products and purposes. For example, OpenAI distinguishes GPTBot from OAI-SearchBot, and Anthropic distinguishes ClaudeBot from Claude-SearchBot. A broad Content Signal does not authenticate those requesters or prove they will honor it.
How CrawlLedger preserves the signal
- Each declaration remains attached to its observed user-agent group and optional path.
- Missing values stay missing rather than being converted to yes or no.
- Malformed values create parser warnings instead of guessed meanings.
- The parser records
useseparately because it is newer and still evolving. - Historical records keep their original parser version and chain hashes.