Rattlesnakes By Mail

honours crawl delay across crawlers

honours crawl delay across 15 crawlers from 6 vendors, from 15 claims, last verified 2026-09-14.

honours crawl delay across 15 crawlers from 6 vendors, from 15 claims, last verified 2026-09-14.

CrawlerVendorValueMethodVerifiedClaim
ApplebotApplenovendor_doc2026-09-13claim 10
Applebot-ExtendedApplen/avendor_doc2026-09-13claim 2
BingbotMicrosoftyesvendor_doc2026-09-14claim 122
ChatGPT-UserOpenAInot_documentedvendor_doc2026-09-13claim 20
Claude-SearchBotAnthropicyesvendor_doc2026-09-13claim 29
Claude-UserAnthropicyesvendor_doc2026-09-13claim 39
ClaudeBotAnthropicyesvendor_doc2026-09-13claim 48
Google-ExtendedGooglen/avendor_doc2026-09-13claim 56
GooglebotGooglenovendor_doc2026-09-13claim 64
GoogleOtherGooglenovendor_doc2026-09-13claim 72
GPTBotOpenAInot_documentedvendor_doc2026-09-13claim 79
OAI-AdsBotOpenAInot_documentedvendor_doc2026-09-13claim 87
OAI-SearchBotOpenAInot_documentedvendor_doc2026-09-13claim 95
Perplexity-UserPerplexitynot_documentedvendor_doc2026-09-13claim 104
PerplexityBotPerplexitynot_documentedvendor_doc2026-09-13claim 113

Not documented

Every documented crawler has a claim for this field.

Sources

Source
https://support.apple.com/en-us/119829
https://developers.openai.com/api/docs/bots
https://support.claude.com/en/articles/8896518-does-anthropic-crawl-content-from-the-web-and-how-can-site-owners-block-the-crawler
https://developers.google.com/search/docs/crawling-indexing/google-common-crawlers
https://developers.google.com/search/blog/2019/07/a-note-on-unsupported-rules-in-robotstxt
https://docs.perplexity.ai/docs/resources/perplexity-crawlers
https://www.bing.com/webmasters/help/crawl-control-55a30303