Skip to main content
If a site blocks GPTBot or ClaudeBot, AI answers can’t quote it, and your Visibility Score will stay low whatever else you do. The AI access check tells you whether that’s happening. It works without any integration installed.

​
What it checks

1

Homepage

Fetches your homepage as RankRaftBot, then as a normal browser. If only the browser is served, some bot protection is active: Your server refused our crawler but served a browser, so some bot protection is active. Allow-list RankRaftBot if you want us to read the site directly.
2

robots.txt

Reads your robots.txt and checks every AI assistant and AI training crawler against its own User-agent group, plus the User-agent: * group. A full-site Disallow is reported as a block; an explicit Allow: / cancels it.
3

Meta robots

Looks for noai, noimageai or noindex in the homepage’s robots meta tags.
4

X-Robots-Tag

Looks for a noindex response header that hides your pages from indexers.
5

Live probes

Requests a page as GPTBot, ClaudeBot, PerplexityBot and OAI-SearchBot, and compares the responses with a browser and an unrelated crawler. This catches CDN and firewall rules that robots.txt doesn’t show.
If your site is behind Cloudflare, the check also reminds you to make sure Block AI Scrapers and Crawlers is off under Security → Bots, because that setting isn’t visible from outside.

​
Results

The AI crawler access card on Integrations shows one of: Only published rules (robots.txt, meta tags, headers) are reported as definite blocks. Live probe results are shown as advisory, because a firewall may still let a verified crawler through. A dropped connection, a 429 or a 503 is never reported as a block. When something you fixed is now passing, the card says Fixed since the last check. AI crawlers are getting through now. Blocks also feed the Action Center: a definite robots.txt block becomes a high-impact, small-effort action.

​
When it runs

  • During onboarding, while RankRaft AI reads your site.
  • Monthly, with the scheduled page inventory.
  • Whenever you click Re-check now on Integrations. It takes about 15 seconds. You can re-check once every 10 minutes.

​
Allowing AI crawlers in robots.txt

To explicitly allow the main AI crawlers:
robots.txt
Blocking training crawlers (GPTBot, ClaudeBot, Google-Extended, CCBot) keeps your content out of future training data. Blocking search and user crawlers (OAI-SearchBot, ChatGPT-User, PerplexityBot, Claude-User) stops answer engines from reading and citing your pages today. Decide each one on purpose.