> ## Documentation Index
> Fetch the complete documentation index at: https://docs.rankraft.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# AI access check

> Find out whether your robots.txt, meta tags, headers or bot protection stop AI crawlers from reading your site.

If a site blocks GPTBot or ClaudeBot, AI answers can't quote it, and your Visibility Score will stay low whatever else you do. The AI access check tells you whether that's happening. It works without any integration installed.

## What it checks

<Steps>
  <Step title="Homepage">
    Fetches your homepage as [RankRaftBot](/agent-analytics/bots/rankraftbot), then as a normal browser. If only the browser is served, some bot protection is active: *Your server refused our crawler but served a browser, so some bot protection is active. Allow-list RankRaftBot if you want us to read the site directly.*
  </Step>

  <Step title="robots.txt">
    Reads your `robots.txt` and checks every AI assistant and AI training crawler against its own `User-agent` group, plus the `User-agent: *` group. A full-site `Disallow` is reported as a block; an explicit `Allow: /` cancels it.
  </Step>

  <Step title="Meta robots">
    Looks for `noai`, `noimageai` or `noindex` in the homepage's robots meta tags.
  </Step>

  <Step title="X-Robots-Tag">
    Looks for a `noindex` response header that hides your pages from indexers.
  </Step>

  <Step title="Live probes">
    Requests a page as GPTBot, ClaudeBot, PerplexityBot and OAI-SearchBot, and compares the responses with a browser and an unrelated crawler. This catches CDN and firewall rules that `robots.txt` doesn't show.
  </Step>
</Steps>

If your site is behind Cloudflare, the check also reminds you to make sure **Block AI Scrapers and Crawlers** is off under **Security → Bots**, because that setting isn't visible from outside.

## Results

The **AI crawler access** card on **Integrations** shows one of:

| Result | Meaning |
| - | - |
| **AI crawlers can read your site** | Nothing blocks the crawlers we check |
| **AI crawlers are blocked** | A published rule (for example `robots.txt`) blocks them. The card lists which |
| **Some AI crawlers may be blocked** | A probe or tag suggests a block, but there's no published rule |
| **We could not tell** | Your site rate-limited or dropped our checks. That isn't evidence of a block; we try again on the next scan |

Only published rules (`robots.txt`, meta tags, headers) are reported as definite blocks. Live probe results are shown as advisory, because a firewall may still let a *verified* crawler through. A dropped connection, a 429 or a 503 is never reported as a block.

When something you fixed is now passing, the card says *Fixed since the last check. AI crawlers are getting through now.*

Blocks also feed the [Action Center](/analytics/action-center/about): a definite `robots.txt` block becomes a high-impact, small-effort action.

## When it runs

* During onboarding, while RankRaft AI reads your site.
* Monthly, with the scheduled page inventory.
* Whenever you click **Re-check now** on **Integrations**. It takes about 15 seconds. You can re-check once every 10 minutes.

## Allowing AI crawlers in robots.txt

To explicitly allow the main AI crawlers:

```txt robots.txt theme={"system"}
User-agent: GPTBot
User-agent: OAI-SearchBot
User-agent: ChatGPT-User
User-agent: ClaudeBot
User-agent: Claude-SearchBot
User-agent: Claude-User
User-agent: PerplexityBot
User-agent: Perplexity-User
User-agent: Google-Extended
Allow: /
```

<Note>
  Blocking training crawlers (GPTBot, ClaudeBot, Google-Extended, CCBot) keeps your content out of future training data. Blocking search and user crawlers (OAI-SearchBot, ChatGPT-User, PerplexityBot, Claude-User) stops answer engines from reading and citing your pages today. Decide each one on purpose.
</Note>


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.