> ## Documentation Index
> Fetch the complete documentation index at: https://docs.rankraft.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# AI Crawlers

> Which AI bots read your site, how often, whether they are verified, and whether anything is blocked.

**AI Crawlers** answers two questions: which AI bots read your site, and whether anything is blocked. Everything on this page comes from your [integration](/agent-analytics/integrations/overview), not from answer scans.

## Crawl volume

A time series of **AI crawler** and **search bot** requests over the selected range. It includes every bot request, verified or not. Use **Raw log** to jump to the matching rows in the [Request log](/agent-analytics/dashboard/logs).

## AI bots on your site

A table of every bot seen in the period. Switch the grouping with **By bot**, **By provider** or **By type**.

| Column | Meaning |
| - | - |
| Bot | The crawler's name, such as GPTBot or PerplexityBot |
| Provider | The company that runs it |
| Type | AI assistant, AI training, Indexing or AI agent |
| Hits | Requests in the period |
| Pages | Distinct paths requested |
| Verified | Share of hits whose IP matched the provider's published ranges. See [Crawler verification](/agent-analytics/crawler-verification) |
| Last seen | Most recent request |
| robots.txt | **Allowed** or **Blocked by robots.txt**, from the latest [AI access check](/agent-analytics/bots/ai-access-check) |

Click a row to filter the whole page to that bot, provider or type.

## Most-read pages

The six paths with the most AI crawler visits in the period. **All pages** opens [Website Audit](/agent-analytics/dashboard/pages).

## Crawl errors

Pages that answered an AI crawler with a 4xx or 5xx status. A crawler that keeps hitting a 404 or a 500 is reading a broken version of your site, and that broken version is what the answer engine learns from. If nothing failed, the card says *No errors served to AI crawlers in this period.*

## Am I blocking AI by accident?

A matrix of the [AI access check](/agent-analytics/bots/ai-access-check) results for each probed crawler, with three columns:

* **robots.txt**: whether your robots file disallows the bot.
* **Meta robots**: whether your homepage carries a `noai` or `noindex` meta tag.
* **Live probe**: what your server actually returned when asked as that bot.

Each cell reads **Allowed**, **Blocked**, **Challenged** or **Unclear**. Use **Re-check** to run the check again from **Integrations**. If the check hasn't run yet, the card says *Not checked yet*.

## Filters and date range

All Agent Analytics pages share the same controls:

* **Date range:** **24h** (hourly buckets), **7d** (the default, daily buckets), **30d** (daily) or **90d** (weekly).
* **Filters:** provider, bot type, bot, verification state, source, page type and path. Active filters appear as chips under **Active filters**. Remove a chip to clear that filter.
* **Time zone:** charts are bucketed in your site's time zone.


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.