Resources / Directory
AI crawler directory
Documented crawler tokens and controls to review before changing robots.txt. A token is not proof of a request; verify logs and provider guidance.
GPTBot
OpenAI · GPTBot
Training-related collection. Documented robots.txt control.
Use OpenAI's current machine-readable IP-prefix feed; never treat the user-agent alone as identity. Verification reference
OAI-SearchBot
OpenAI · OAI-SearchBot
ChatGPT search. Documented robots.txt control.
Use OpenAI's current machine-readable IP-prefix feed; never treat the user-agent alone as identity. Verification reference
ChatGPT-User
OpenAI · ChatGPT-User
User-triggered fetch. Rules may not apply.
Use OpenAI's current machine-readable IP-prefix feed; do not grant privileged access from the user-agent alone. Verification reference
ClaudeBot
Anthropic · ClaudeBot
Training-related collection. Documented robots.txt control.
Anthropic says it does not publish bot IP ranges, so use robots.txt and ordinary request controls rather than an IP allowlist.
Claude-SearchBot
Anthropic · Claude-SearchBot
Claude search. Documented robots.txt control.
Anthropic says it does not publish bot IP ranges, so use robots.txt and ordinary request controls rather than an IP allowlist.
Claude-User
Anthropic · Claude-User
User-triggered fetch. Anthropic says its bots honor robots.txt.
Anthropic says it does not publish bot IP ranges, so use robots.txt and ordinary request controls rather than an IP allowlist.
PerplexityBot
Perplexity · PerplexityBot
Perplexity search. Documented robots.txt control.
No provider IP-range reference is included here; follow the current provider documentation and do not trust a user-agent string alone.
Perplexity-User
Perplexity · Perplexity-User
User-triggered fetch. Different documented behavior; verify provider policy.
No provider IP-range reference is included here; follow the current provider documentation and do not trust a user-agent string alone.
Google-Extended
Google · Google-Extended
Gemini training and specified grounding uses. Control token; not a separate request user agent.
Google-Extended is a policy token, not a distinct requester. Verify the underlying Google crawler through DNS or Google's current JSON IP feeds. Verification reference
bingbot
Microsoft Bing · bingbot
Bing crawling and related grounding supply. Documented robots.txt control.
Use reverse and forward DNS validation or Bing's verification tool. Bing advises against hardcoding a static IP range. Verification reference