Skip to main content
AI Crawlers answers two questions: which AI bots read your site, and whether anything is blocked. Everything on this page comes from your integration, not from answer scans.

​
Crawl volume

A time series of AI crawler and search bot requests over the selected range. It includes every bot request, verified or not. Use Raw log to jump to the matching rows in the Request log.

​
AI bots on your site

A table of every bot seen in the period. Switch the grouping with By bot, By provider or By type. Click a row to filter the whole page to that bot, provider or type.

​
Most-read pages

The six paths with the most AI crawler visits in the period. All pages opens Website Audit.

​
Crawl errors

Pages that answered an AI crawler with a 4xx or 5xx status. A crawler that keeps hitting a 404 or a 500 is reading a broken version of your site, and that broken version is what the answer engine learns from. If nothing failed, the card says No errors served to AI crawlers in this period.

​
Am I blocking AI by accident?

A matrix of the AI access check results for each probed crawler, with three columns:
  • robots.txt: whether your robots file disallows the bot.
  • Meta robots: whether your homepage carries a noai or noindex meta tag.
  • Live probe: what your server actually returned when asked as that bot.
Each cell reads Allowed, Blocked, Challenged or Unclear. Use Re-check to run the check again from Integrations. If the check hasn’t run yet, the card says Not checked yet.

​
Filters and date range

All Agent Analytics pages share the same controls:
  • Date range: 24h (hourly buckets), 7d (the default, daily buckets), 30d (daily) or 90d (weekly).
  • Filters: provider, bot type, bot, verification state, source, page type and path. Active filters appear as chips under Active filters. Remove a chip to clear that filter.
  • Time zone: charts are bucketed in your site’s time zone.