GPTBot, OAI-SearchBot, OAI-AdsBot, and ChatGPT-User do different jobs.
OpenAI documents separate user-agent tokens for model-training crawl, ChatGPT search discovery, submitted ad landing-page validation, and user-triggered page visits. A robots.txt decision about one token should not be copied to the others without understanding that purpose.
GPTBot concerns potential model-training use; OAI-SearchBot concerns ChatGPT search; OAI-AdsBot visits landing pages submitted as ads; and ChatGPT-User is triggered by a person's request. OpenAI says each setting is independent.
OpenAI crawler comparison
| Token | Documented trigger | Documented purpose | Robots.txt boundary | Published IP list |
|---|---|---|---|---|
GPTBot | Automatic crawl | Content that may improve or train generative AI foundation models | OpenAI documents this as the separate training control | gptbot.json |
OAI-SearchBot | Automatic crawl | Surface and link websites in ChatGPT search features | OpenAI documents this as the Search opt-out and automatic-crawl control | searchbot.json |
OAI-AdsBot | Advertiser submits a landing page | Validate ad-page safety and help determine ad relevance | Purpose-limited fetcher; not the documented Search or training control | adsbot.json |
ChatGPT-User | A user asks ChatGPT, a Custom GPT, or a GPT Action to visit | Retrieve content for that user action | OpenAI says robots.txt rules may not apply because the visit is user-initiated | chatgpt-user.json |
Can a site allow ChatGPT search but decline GPTBot training?
OpenAI's documentation says the settings are independent and gives this exact policy shape as an example. A minimal starting point is:
User-agent: OAI-SearchBot
Allow: /
User-agent: GPTBot
Disallow: /This is a technical example, not a complete site policy or legal recommendation. Existing search-engine, application, sitemap, path-specific, and wildcard rules must be preserved and reviewed as a whole.
Where OAI-AdsBot fits
OpenAI currently describes OAI-AdsBot as visiting only pages submitted as ChatGPT ads. Its collected landing-page data is documented as not being used to train generative AI foundation models. CrawlLedger therefore tracks it separately and does not silently group it with either GPTBot or OAI-SearchBot in the policy generator.
Why ChatGPT-User needs a separate decision
ChatGPT-User is not documented as an automatic web crawler and does not control whether content appears in ChatGPT Search. OpenAI says ordinary robots.txt rules may not apply to these user-initiated requests. Treating it as a synonym for OAI-SearchBot would merge two different product behaviors.
Verification is separate from policy
A user-agent string is self-declared and can be copied. OpenAI publishes separate IP-range JSON files for each token so operators can add network verification when that matters. A robots.txt observation alone does not authenticate the requester or prove that a system followed the published rule.
Check a real policy without a network crawl
Paste an existing robots.txt file into CrawlLedger's checker to compare all four tokens at a specific path. The input is parsed for the response and is not stored by application code.
Primary source and review boundary
OpenAI's crawler documentation was reviewed on Sep 20, 2026. Operator documentation can change; the CrawlLedger registry records its review version and the weekly freshness gate prevents old review dates from passing silently.
Who, how, and why: CrawlLedger publishes this comparison under its own name. It was assembled from the linked OpenAI source and the same versioned registry used by the public parser, so readers can move from an explanation to observed evidence without treating either as legal permission.