# Rattlesnakes By Mail A Field Guide to the Crawlers of the Web Rattlesnakes By Mail is a reference on how AI crawlers and AI search engines read the web. Each page on Rattlesnakes By Mail is a record, and each fact on a record is a dated claim. Three formats serve the same content: HTML, Markdown at the .md suffix, and JSON at the .json suffix. The same URL answers Accept: text/markdown and application/json. Claim 47, for example, states: ClaudeBot honours industry standard robots.txt directives that signal do not crawl. Claim 47 carries confidence high and the method vendor_doc, quotes Anthropic's crawler documentation dated 2026-04-07, and was verified on 2026-09-13. ### Observations Rattlesnakes By Mail logs each request from an identified crawler and publishes per-crawler daily counts at /observed. GPTBot made 670 requests in the last 30 days and 99% of the requests were verified against OpenAI's published IP ranges. An unverified request means the request IP address did not match a published range, and an unverified request is not evidence that the user agent was spoofed. ### Claims Every fact on Rattlesnakes By Mail is a dated claim with a verbatim vendor quote, a source URL, a method and a confidence, published at /claims. A published claim is never edited: a change creates a new claim that supersedes the old claim, both claims stay addressable, and every supersession is listed at /changes. ### Questions Rattlesnakes By Mail records every search query made on the site, matches each query against the published claims, and publishes the unmatched queries as gaps at /questions. A seeded question carries the label seeded, and an organic question carries the source the query arrived from. ### Data Rattlesnakes By Mail exports each public table nightly as JSON and as CSV under CC BY 4.0 to the public GitHub repository crank-box/rattlesnakesbymail-data. The dataset and the nightly export are described at /data. Search: POST /search with q and an optional note, or use the ask tool on the MCP server at /mcp. ## Blocking Disallowing Google-Extended in robots.txt stops Google from using already-crawled content to train Gemini models and to ground Gemini features. (claim 59, claim 61) Google documents that Google-Extended does not affect a site's inclusion in Google Search and is not used as a ranking signal in Google Search. (claim 60) Disallowing GPTBot in robots.txt signals to OpenAI that crawled content is not to be used to train OpenAI's generative AI foundation models, and allowing OAI-SearchBot alongside that disallow keeps a site eligible to appear in ChatGPT search results. (claim 82, claim 83) A GPTBot disallow does not govern ChatGPT-User, because OpenAI documents that ChatGPT-User fetches a page when a user asks ChatGPT a question and that robots.txt rules may not apply to a user-initiated fetch. (claim 19, claim 23) Crawler identity is checked against the IP ranges the vendor publishes, and Google and Apple document a reverse DNS method in addition. (claim 49, claim 80, claim 65, claim 114, claim 11, claim 66, claim 12) ## Crawlers Kind separates what an agent does from who operates the agent. A crawler fetches pages on the vendor's own schedule, with no user waiting on the fetch. A fetcher retrieves one page at the moment a user asks an assistant a question. A search_bot fetches pages to build and serve the index behind an AI answer engine. An ads_bot fetches pages submitted as advertisements and checks the pages against the vendor's advertising policies. A policy_token is a name that appears only in robots.txt, and a policy token governs how a vendor may use content the vendor already crawled. Google-Extended and Applebot-Extended are robots.txt tokens with no crawler behind either token: Google documents that Google-Extended has no separate HTTP request user agent string and that crawling is done with existing Google user agent strings, and Apple documents that Applebot-Extended does not correspond to a separate crawler user agent string. | Vendor | Name | Kind | Purpose | | --- | --- | --- | --- | | Anthropic | Claude-SearchBot | search_bot | search | | Anthropic | Claude-User | fetcher | user_fetch | | Anthropic | ClaudeBot | crawler | training | | Apple | Applebot | crawler | search | | Apple | Applebot-Extended | policy_token | training | | Google | Google-Extended | policy_token | training | | Google | GoogleOther | crawler | mixed | | Google | Googlebot | crawler | search | | Microsoft | Bingbot | crawler | search | | OpenAI | ChatGPT-User | fetcher | user_fetch | | OpenAI | GPTBot | crawler | training | | OpenAI | OAI-AdsBot | ads_bot | ads | | OpenAI | OAI-SearchBot | search_bot | search | | Perplexity | Perplexity-User | fetcher | user_fetch | | Perplexity | PerplexityBot | search_bot | search | ## Changes | Date | Entity | Claim | Kind | Old value | New value | | --- | --- | --- | --- | --- | --- | | 2026-09-17 | Applebot | observed_paths_fetched | new | | robots_txt_and_home_only | | 2026-09-17 | Googlebot | observed_verified_request_ratio_census | new | | 0.991 | | 2026-09-17 | OAI-SearchBot | observed_arrival_before_sitemap | new | | yes | | 2026-09-17 | GoogleOther | 2 claims | new | | | | 2026-09-17 | ClaudeBot | 4 claims | new | | | | 2026-09-14 | Googlebot | reads_llms_txt | new | | no | | 2026-09-14 | Bingbot | 11 claims | new | | | | 2026-09-13 | PerplexityBot | 9 claims | new | | | ## Observations | Crawler | Requests (30d) | Verified share | Last seen | | --- | --- | --- | --- | | GPTBot | 670 | 99% | 2026-09-16 | | ClaudeBot | 584 | 100% | 2026-09-16 | | GoogleOther | 243 | 100% | 2026-09-16 | | Googlebot | 242 | 100% | 2026-09-16 | | Applebot | 8 | 100% | 2026-09-15 | | Claude-User | 5 | 100% | 2026-09-15 | ## Questions 50 questions have been recorded as gaps since 2026-09-13, 49 of the questions are published at /questions, and the question raised most often so far is: Can robots.txt cleanly separate "allow this page for search or citation" from "block it for training," and does that distinction hold across every major AI platform? ## Related Every claimed field on every crawler has its own page at /fields. Any two crawlers can be compared side by side at /compare.