Free tool · source-labeled · no input storage

Build an AI crawler policy draft without guesswork.

Choose separate starting rules for documented training, search, archive, and user-requested tokens. CrawlLedger generates ordinary robots.txt groups plus optional Content Signals that you can review before publishing.

Draft, not a one-click legal policy

Replacing an existing robots.txt file can remove search-engine, sitemap, or application rules. Merge carefully, validate the complete file, and check each operator's current documentation.

Crawler access draftTraining and downstream-use controls

Includes documented training crawlers and non-fetcher controls such as Google-Extended.

AI search crawlersOpen-web archive crawlerUser-requested fetchers

Some operators say ordinary robots rules may not apply to a user-triggered fetcher. Omission avoids implying otherwise.

Optional Content SignalsInclude a Content-Signal in generated groups

Content Signals express downstream-use preferences. They do not technically block access and are not universally honored.

searchai-inputai-train

Exactly which tokens are included

The generator is built from CrawlLedger's versioned registry. A token is included only with a first-party operator source and review date.

Token	Operator	Purpose	Fetcher or control
`GPTBot`	OpenAI	Model training	Documented fetcher
`OAI-SearchBot`	OpenAI	AI search	Documented fetcher
`ChatGPT-User`	OpenAI	User-requested fetch	Documented fetcher
`ClaudeBot`	Anthropic	Model training	Documented fetcher
`Claude-SearchBot`	Anthropic	AI search	Documented fetcher
`Claude-User`	Anthropic	User-requested fetch	Documented fetcher
`Google-Extended`	Google	Training and grounding control	Control token; not a separate fetcher
`Applebot-Extended`	Apple	Model training	Control token; not a separate fetcher
`PerplexityBot`	Perplexity	AI search	Documented fetcher
`Perplexity-User`	Perplexity	User-requested fetch	Documented fetcher
`CCBot`	Common Crawl	Open web archive	Documented fetcher

A robots.txt line is a published instruction for cooperative systems. It does not authenticate a request, enforce behavior, establish consent, or decide legal permission.