Markdown content negotiation is a server feature—not a promise made by every AI crawler.
A site can return Markdown when a client sends Accept: text/markdown. That proves the server can negotiate a representation. It does not prove GPTBot, OAI-SearchBot, ClaudeBot, or another named crawler will send that header.
Cloudflare documents how a requesting client can receive Markdown, while the first-party crawler pages reviewed by CrawlLedger do not currently document an Accept: text/markdown request header for the tracked OpenAI, Anthropic, or Perplexity tokens.
What the HTTP exchange looks like
Content negotiation starts with a request header. On a compatible server, the response changes representation and advertises the selected media type:
curl -sS -D - -o /dev/null -H 'Accept: text/markdown' https://example.com/page
# Look for:
# Content-Type: text/markdown; charset=utf-8
# Vary: AcceptA 200 response with text/html means the request succeeded, but no Markdown representation was selected. A Markdown response shows server behavior for that request—not the behavior of every crawler.
What Cloudflare Markdown for Agents changes
For an enabled zone, Cloudflare documents on-the-fly HTML-to-Markdown conversion when text/markdown is requested. Its response uses Content-Type: text/markdown, varies the cache on Accept, recalculates the body length, and can include token-count and Content Signal headers. Converted responses remove validators such as ETag and Last-Modified because they describe a transformed body.
Cloudflare currently lists this feature for Pro, Business, and Enterprise zones. CrawlLedger did not enable it, change a plan, or incur spend during this audit.
What the named crawler documentation establishes
The operator pages establish user-agent tokens, product purposes, robots.txt controls, and—in some cases—network verification details. At the Aug 25, 2026 review, those pages did not specify that the tracked tokens send Accept: text/markdown. The careful state is therefore not documented, not “supported” or “unsupported.”
- GPTBot and OAI-SearchBot are separate OpenAI controls for training and search.
- ClaudeBot and Claude-SearchBot are separate Anthropic controls.
- PerplexityBot and Perplexity-User have separate automatic and user-requested roles.
Do not confuse four different signals
| Mechanism | Question it answers | What it does not prove |
|---|---|---|
Accept: text/markdown | Can this request receive a Markdown representation? | That a named crawler sends the header |
robots.txt | What fetch preferences are published for a token and path? | Identity, enforcement, or downstream-use terms |
llms.txt | What optional guidance and links did a publisher place at a known URL? | Access permission, crawler adoption, or Google ranking benefit |
Content-Signal | What downstream-use preferences were declared? | Technical blocking, universal support, or a legal conclusion |
How CrawlLedger handles this boundary
The current archive fetches public policy files, not ordinary page content, and does not vary requests to infer Markdown support. A future content-negotiation observation would need its own versioned signal, request headers, response metadata, cost review, and explicit evidence states. Until then, CrawlLedger records no per-domain Markdown-for-Agents conclusion.