> ## Documentation Index
> Fetch the complete documentation index at: https://docs.rankraft.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# AI crawlers we track

> Every crawler RankRaft AI recognises, with its provider, type and whether it can be verified.

RankRaft AI recognises crawlers by a case-insensitive match on the user agent. **Verifiable** means the provider publishes IP ranges, so genuine hits show as **Verified** and impersonators as **Spoofed**. See [Crawler verification](/agent-analytics/crawler-verification).

## AI crawlers

| Crawler | Provider | Type | Verifiable |
| - | - | - | - |
| GPTBot | OpenAI | AI training | Yes |
| OAI-SearchBot | OpenAI | Indexing | Yes |
| ChatGPT-User | OpenAI | AI assistant | Yes |
| ClaudeBot | Anthropic | AI training | No |
| Claude-SearchBot | Anthropic | Indexing | No |
| Claude-User | Anthropic | AI assistant | No |
| anthropic-ai | Anthropic | AI training | No |
| claude-web | Anthropic | AI assistant | No |
| PerplexityBot | Perplexity | Indexing | Yes |
| Perplexity-User | Perplexity | AI assistant | Yes |
| Google-Extended | Google | AI training | Yes |
| GoogleOther | Google | AI training | Yes |
| Google-CloudVertexBot | Google | AI training | Yes |
| MicrosoftPreview | Microsoft | AI assistant | No |
| meta-externalagent | Meta | AI training | No |
| meta-externalfetcher | Meta | AI assistant | No |
| FacebookBot | Meta | AI training | No |
| Applebot-Extended | Apple | AI training | No |
| Amazonbot | Amazon | AI training | No |
| Bytespider | ByteDance | AI training | No |
| TikTokSpider | ByteDance | AI training | No |
| CCBot | Common Crawl | AI training | No |
| DuckAssistBot | DuckDuckGo | AI assistant | No |
| YouBot | You.com | Indexing | No |
| MistralAI-User | Mistral | AI assistant | No |
| cohere-ai | Cohere | AI training | No |
| cohere-training-data-crawler | Cohere | AI training | No |
| AI2Bot | Allen AI | AI training | No |
| Diffbot | Diffbot | AI training | No |
| PanguBot | Huawei | AI training | No |
| omgili, Timpibot, ImagesiftBot, Webzio-Extended, Kangaroo Bot | Other | AI training | No |

## Search bots

Classic search indexers are tracked as **search bots**, separately from AI crawlers.

| Crawler | Provider | Verifiable |
| - | - | - |
| Googlebot, Googlebot-Image | Google | Yes |
| Google-InspectionTool, AdsBot-Google | Google | Yes |
| bingbot | Microsoft | Yes |
| BingPreview | Microsoft | No |
| Applebot | Apple | No |
| facebookexternalhit | Meta | No |
| DuckDuckBot | DuckDuckGo | No |
| PetalBot | Huawei | No |
| Baiduspider | Baidu | No |
| YandexBot | Yandex | No |

## Everything else

User agents that look automated but match no known crawler, such as uptime monitors, SEO tools, `curl`, `python-requests` and headless browsers, are counted as **other bots** in your daily totals and aren't stored individually.

<Tip>
  Missing a crawler you see in your own server logs? Email [hello@rankraft.ai](mailto:hello@rankraft.ai) with the user agent and we'll add it.
</Tip>


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.