# honours crawl delay across crawlers

honours crawl delay across 15 crawlers from 6 vendors, from 15 claims, last verified 2026-09-14.

honours crawl delay across 15 crawlers from 6 vendors, from 15 claims, last verified 2026-09-14.

| Crawler | Vendor | Value | Method | Verified | Claim |
| --- | --- | --- | --- | --- | --- |
| Applebot | Apple | no | vendor_doc | 2026-09-13 | claim 10 |
| Applebot-Extended | Apple | n/a | vendor_doc | 2026-09-13 | claim 2 |
| Bingbot | Microsoft | yes | vendor_doc | 2026-09-14 | claim 122 |
| ChatGPT-User | OpenAI | not_documented | vendor_doc | 2026-09-13 | claim 20 |
| Claude-SearchBot | Anthropic | yes | vendor_doc | 2026-09-13 | claim 29 |
| Claude-User | Anthropic | yes | vendor_doc | 2026-09-13 | claim 39 |
| ClaudeBot | Anthropic | yes | vendor_doc | 2026-09-13 | claim 48 |
| Google-Extended | Google | n/a | vendor_doc | 2026-09-13 | claim 56 |
| Googlebot | Google | no | vendor_doc | 2026-09-13 | claim 64 |
| GoogleOther | Google | no | vendor_doc | 2026-09-13 | claim 72 |
| GPTBot | OpenAI | not_documented | vendor_doc | 2026-09-13 | claim 79 |
| OAI-AdsBot | OpenAI | not_documented | vendor_doc | 2026-09-13 | claim 87 |
| OAI-SearchBot | OpenAI | not_documented | vendor_doc | 2026-09-13 | claim 95 |
| Perplexity-User | Perplexity | not_documented | vendor_doc | 2026-09-13 | claim 104 |
| PerplexityBot | Perplexity | not_documented | vendor_doc | 2026-09-13 | claim 113 |

## Not documented

Every documented crawler has a claim for this field.

## Sources

| Source |
| --- |
| https://support.apple.com/en-us/119829 |
| https://developers.openai.com/api/docs/bots |
| https://support.claude.com/en/articles/8896518-does-anthropic-crawl-content-from-the-web-and-how-can-site-owners-block-the-crawler |
| https://developers.google.com/search/docs/crawling-indexing/google-common-crawlers |
| https://developers.google.com/search/blog/2019/07/a-note-on-unsupported-rules-in-robotstxt |
| https://docs.perplexity.ai/docs/resources/perplexity-crawlers |
| https://www.bing.com/webmasters/help/crawl-control-55a30303 |