# Rattlesnakes By Mail

A Field Guide to the Crawlers of the Web: a reference on how AI crawlers and AI search engines read the web, where every page exists as HTML, as Markdown at the .md suffix, and as JSON at the .json suffix.
Observations at /observed, claims at /claims, questions at /questions, and the open dataset at /data.

## Crawlers

- [Claude-SearchBot](https://rattlesnakesbymail.com/crawlers/claude-searchbot.md): Anthropic, search_bot, purpose search
- [Claude-User](https://rattlesnakesbymail.com/crawlers/claude-user.md): Anthropic, fetcher, purpose user_fetch
- [ClaudeBot](https://rattlesnakesbymail.com/crawlers/claudebot.md): Anthropic, crawler, purpose training
- [Applebot](https://rattlesnakesbymail.com/crawlers/applebot.md): Apple, crawler, purpose search
- [Applebot-Extended](https://rattlesnakesbymail.com/crawlers/applebot-extended.md): Apple, policy_token, purpose training
- [Google-Extended](https://rattlesnakesbymail.com/crawlers/google-extended.md): Google, policy_token, purpose training
- [GoogleOther](https://rattlesnakesbymail.com/crawlers/googleother.md): Google, crawler, purpose mixed
- [Googlebot](https://rattlesnakesbymail.com/crawlers/googlebot.md): Google, crawler, purpose search
- [Bingbot](https://rattlesnakesbymail.com/crawlers/bingbot.md): Microsoft, crawler, purpose search
- [ChatGPT-User](https://rattlesnakesbymail.com/crawlers/chatgpt-user.md): OpenAI, fetcher, purpose user_fetch
- [GPTBot](https://rattlesnakesbymail.com/crawlers/gptbot.md): OpenAI, crawler, purpose training
- [OAI-AdsBot](https://rattlesnakesbymail.com/crawlers/oai-adsbot.md): OpenAI, ads_bot, purpose ads
- [OAI-SearchBot](https://rattlesnakesbymail.com/crawlers/oai-searchbot.md): OpenAI, search_bot, purpose search
- [Perplexity-User](https://rattlesnakesbymail.com/crawlers/perplexity-user.md): Perplexity, fetcher, purpose user_fetch
- [PerplexityBot](https://rattlesnakesbymail.com/crawlers/perplexitybot.md): Perplexity, search_bot, purpose search

## Reference

- [/data.md](https://rattlesnakesbymail.com/data.md): dataset landing, schema, licence
- [/data/latest.json](https://rattlesnakesbymail.com/data/latest.json): manifest of the nightly export
- [/method.md](https://rattlesnakesbymail.com/method.md): verification method
- [/changes.xml](https://rattlesnakesbymail.com/changes.xml): changelog as Atom