Sourced guide

robots.txt and AI search crawlers: which ones to allow if you want to be cited

Which crawler names the major AI companies document for search, which ones relate to model training instead, and why blocking the search crawlers can keep a business out of AI answers.

Search crawlers and training crawlers are different

Several AI companies now publish separate crawler names for search and for training. OpenAI documents OAI-SearchBot (search) and GPTBot (training) as independent settings. Anthropic documents Claude-SearchBot (search), Claude-User (pages fetched when a user asks) and ClaudeBot (training). Apple documents Applebot (search features) and Applebot-Extended, which publishers can disallow to opt out of their content being used to train Apple's foundation models.

This means a business can choose to stay out of training while still allowing the crawlers that decide whether it can appear, with a link, in AI search answers.

Check before you assume

Some website builders, security plugins and hosting firewalls block unfamiliar bots by default. If a site's robots.txt or firewall blocks OAI-SearchBot, PerplexityBot, Claude-SearchBot, Applebot, Bingbot or Googlebot, the matching AI tool has less or no ability to read it.

Perplexity notes that changes can take up to 24 hours to be reflected in its systems. Anthropic asks site owners to set rules for every subdomain they want to control.

Sources (checked Nov 18, 2026)

  1. OpenAI: Overview of OpenAI crawlers
  2. Anthropic: crawler help article
  3. Perplexity: Perplexity crawlers
  4. Apple: About Applebot