# Crawlers

Every crawler, fetcher, search bot, ads bot, and robots.txt policy token documented on this site.

| Vendor | Name | Kind | Purpose | Status |
| --- | --- | --- | --- | --- |
| Anthropic | Claude-SearchBot | search_bot | search | active |
| Anthropic | Claude-User | fetcher | user_fetch | active |
| Anthropic | ClaudeBot | crawler | training | active |
| Apple | Applebot | crawler | search | active |
| Apple | Applebot-Extended | policy_token | training | active |
| Google | Google-Extended | policy_token | training | active |
| Google | GoogleOther | crawler | mixed | active |
| Google | Googlebot | crawler | search | active |
| Microsoft | Bingbot | crawler | search | active |
| OpenAI | ChatGPT-User | fetcher | user_fetch | active |
| OpenAI | GPTBot | crawler | training | active |
| OpenAI | OAI-AdsBot | ads_bot | ads | active |
| OpenAI | OAI-SearchBot | search_bot | search | active |
| Perplexity | Perplexity-User | fetcher | user_fetch | active |
| Perplexity | PerplexityBot | search_bot | search | active |