opt out mechanism across crawlers
opt out mechanism across 1 crawlers from 1 vendors, from 1 claims, last verified 2026-09-14.
opt out mechanism across 1 crawlers from 1 vendors, from 1 claims, last verified 2026-09-14.
Not documented
| Crawler | Vendor |
|---|---|
| Claude-SearchBot | Anthropic |
| Claude-User | Anthropic |
| ClaudeBot | Anthropic |
| Applebot | Apple |
| Applebot-Extended | Apple |
| Google-Extended | |
| GoogleOther | |
| Googlebot | |
| ChatGPT-User | OpenAI |
| GPTBot | OpenAI |
| OAI-AdsBot | OpenAI |
| OAI-SearchBot | OpenAI |
| Perplexity-User | Perplexity |
| PerplexityBot | Perplexity |