Free tool · source-labeled · no input storage

Build an AI crawler policy draft without guesswork.

Choose separate starting rules for documented training, search, archive, and user-requested tokens. CrawlLedger generates ordinary robots.txt groups plus optional Content Signals that you can review before publishing.

Draft, not a one-click legal policy

Replacing an existing robots.txt file can remove search-engine, sitemap, or application rules. Merge carefully, validate the complete file, and check each operator's current documentation.

Crawler access draftTraining and downstream-use controls

Includes documented training crawlers and non-fetcher controls such as Google-Extended.

AI search crawlersOpen-web archive crawlerUser-requested fetchers

Some operators say ordinary robots rules may not apply to a user-triggered fetcher. Omission avoids implying otherwise.

Optional Content SignalsInclude a Content-Signal in generated groups

Content Signals express downstream-use preferences. They do not technically block access and are not universally honored.

searchai-inputai-train

Exactly which tokens are tracked

The generator and checker use CrawlLedger's versioned registry. A token is tracked only with a first-party operator source and review date.

Token	Operator	Purpose	Draft coverage
`GPTBot`	OpenAI	Model training	Included when its purpose group is selected
`OAI-SearchBot`	OpenAI	AI search	Included when its purpose group is selected
`OAI-AdsBot`	OpenAI	Ad landing-page validation	Checker only; configure after reviewing ad requirements
`ChatGPT-User`	OpenAI	User-requested fetch	Included when its purpose group is selected
`ClaudeBot`	Anthropic	Model training	Included when its purpose group is selected
`Claude-SearchBot`	Anthropic	AI search	Included when its purpose group is selected
`Claude-User`	Anthropic	User-requested fetch	Included when its purpose group is selected
`Google-Extended`	Google	Training and grounding control	Included as a documented control token
`Applebot-Extended`	Apple	Model training	Included as a documented control token
`PerplexityBot`	Perplexity	AI search	Included when its purpose group is selected
`Perplexity-User`	Perplexity	User-requested fetch	Included when its purpose group is selected
`CCBot`	Common Crawl	Open web archive	Included when its purpose group is selected

OAI-AdsBot is not inserted into generated drafts because OpenAI documents it for advertiser-submitted landing pages, not general search or training. Any robots.txt line is a published instruction for cooperative systems; it does not authenticate a request, enforce behavior, establish consent, or decide legal permission.