what it controls across crawlers
what it controls across 2 crawlers from 2 vendors, from 2 claims, last verified 2026-09-13.
what it controls across 2 crawlers from 2 vendors, from 2 claims, last verified 2026-09-13.
| Crawler | Vendor | Value | Method | Verified | Claim |
|---|---|---|---|---|---|
| Applebot-Extended | Apple | AI training use of content, not search indexing | vendor_doc | 2026-09-13 | claim 7 |
| Google-Extended | Gemini model training and grounding, not Search inclusion | vendor_doc | 2026-09-13 | claim 61 |
Not documented
| Crawler | Vendor |
|---|---|
| Claude-SearchBot | Anthropic |
| Claude-User | Anthropic |
| ClaudeBot | Anthropic |
| Applebot | Apple |
| GoogleOther | |
| Googlebot | |
| Bingbot | Microsoft |
| ChatGPT-User | OpenAI |
| GPTBot | OpenAI |
| OAI-AdsBot | OpenAI |
| OAI-SearchBot | OpenAI |
| Perplexity-User | Perplexity |
| PerplexityBot | Perplexity |