respects robots txt across crawlers
respects robots txt across 15 crawlers from 6 vendors, from 15 claims, last verified 2026-09-14.
respects robots txt across 15 crawlers from 6 vendors, from 15 claims, last verified 2026-09-14.
| Crawler | Vendor | Value | Method | Verified | Claim |
|---|---|---|---|---|---|
| Applebot | Apple | yes | vendor_doc | 2026-09-13 | claim 9 |
| Applebot-Extended | Apple | n/a | vendor_doc | 2026-09-13 | claim 1 |
| Bingbot | Microsoft | yes | vendor_doc | 2026-09-14 | claim 121 |
| ChatGPT-User | OpenAI | partial | vendor_doc | 2026-09-13 | claim 19 |
| Claude-SearchBot | Anthropic | yes | vendor_doc | 2026-09-13 | claim 28 |
| Claude-User | Anthropic | yes | vendor_doc | 2026-09-13 | claim 37 |
| ClaudeBot | Anthropic | yes | vendor_doc | 2026-09-13 | claim 47 |
| Google-Extended | n/a | vendor_doc | 2026-09-13 | claim 55 | |
| Googlebot | yes | vendor_doc | 2026-09-13 | claim 63 | |
| GoogleOther | yes | vendor_doc | 2026-09-13 | claim 71 | |
| GPTBot | OpenAI | yes | vendor_doc | 2026-09-13 | claim 78 |
| OAI-AdsBot | OpenAI | not_documented | vendor_doc | 2026-09-13 | claim 86 |
| OAI-SearchBot | OpenAI | yes | vendor_doc | 2026-09-13 | claim 94 |
| Perplexity-User | Perplexity | no | vendor_doc | 2026-09-13 | claim 103 |
| PerplexityBot | Perplexity | yes | vendor_doc | 2026-09-13 | claim 112 |
Not documented
Every documented crawler has a claim for this field.