Crawlers
Every crawler, fetcher, search bot, ads bot, and robots.txt policy token documented on this site.
| Vendor | Name | Kind | Purpose | Status |
|---|---|---|---|---|
| Anthropic | Claude-SearchBot | search_bot | search | active |
| Anthropic | Claude-User | fetcher | user_fetch | active |
| Anthropic | ClaudeBot | crawler | training | active |
| Apple | Applebot | crawler | search | active |
| Apple | Applebot-Extended | policy_token | training | active |
| Google-Extended | policy_token | training | active | |
| GoogleOther | crawler | mixed | active | |
| Googlebot | crawler | search | active | |
| Microsoft | Bingbot | crawler | search | active |
| OpenAI | ChatGPT-User | fetcher | user_fetch | active |
| OpenAI | GPTBot | crawler | training | active |
| OpenAI | OAI-AdsBot | ads_bot | ads | active |
| OpenAI | OAI-SearchBot | search_bot | search | active |
| Perplexity | Perplexity-User | fetcher | user_fetch | active |
| Perplexity | PerplexityBot | search_bot | search | active |