> ## Documentation Index
> Fetch the complete documentation index at: https://docs.rankraft.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Diagnose AI visibility with logs and answers

> Use crawler logs and answer data together to find out why a page isn't cited, and what to fix first.

Answer scans tell you **what AI says**. Your server logs tell you **what AI reads**. Each on its own leaves you guessing. Together they separate a technical problem (AI can't reach or read the page) from a content problem (AI reads it and still prefers someone else).

This guide needs an [integration](https://docs.rankraft.ai/agent-analytics/integrations/overview.md) that sees crawlers: the Cloudflare Worker, the WordPress plugin or server logs. The website snippet only sees people, not crawlers.

## Work through it in order

Each step only makes sense once the one before it passes.

**1. Can AI crawlers reach the site?**

Open **AI Crawlers → Am I blocking AI by accident?** or the **AI crawler access** card on **Integrations**. A block in `robots.txt`, a `noai` or `noindex` tag, or a firewall rule stops everything downstream. Nothing else on this list matters until this passes. See [AI access check](https://docs.rankraft.ai/agent-analytics/bots/ai-access-check.md).

**2. Are all three kinds of crawler visiting?**

On **AI Crawlers**, group the table **By type**. The types do different jobs, and blocking or missing one doesn't affect the others. See the table under [Crawler types](#crawler-types) below. AI assistant visits are the closest thing in your logs to a live answer reading your site.

**3. Are they reading the pages that matter?**

Open [Website Audit](https://docs.rankraft.ai/agent-analytics/dashboard/pages.md). Check the page-type breakdown and look for your product, pricing and comparison pages in the pages table. If crawlers only ever reach your homepage and a few top-level pages, the deeper pages are hard to find: improve internal links and make sure they're in your sitemap.

**4. Are they getting errors?**

Check **AI Crawlers → Crawl errors**. A page that answers crawlers with a 404 or 500 is a broken page as far as the engine is concerned, even if it works in your browser. Pages that only render with JavaScript cause the same problem, because AI crawlers don't run JavaScript.

**5. Read, but not cited?**

Open [Citations & Sources → Your pages AI cites](https://docs.rankraft.ai/analytics/citations/citations-and-sources.md#your-pages-ai-cites) and compare it with the pages crawlers read. A page that's read regularly but never cited is usually a content problem: AI found it and preferred another source. Look at what the cited pages for the same prompts do differently.

**6. Cited, but no longer crawled?**

The same card shows the last verified AI crawl of each cited page, and how many cited pages had no verified crawl in the range. A cited page that crawlers have stopped visiting is often the next one to drop out of answers. Check it for errors, redirects or a new block.

## Crawler types

| Type | What it does | If it's missing |
| - | - | - |
| **AI training** | Collects pages for future model training, such as GPTBot or ClaudeBot | Your content may not shape what models know about you over time |
| **Indexing** | Builds the search index an engine answers from, such as OAI-SearchBot or PerplexityBot | The engine has little of your site to retrieve when it searches |
| **AI assistant** | Fetches a page live because a user's question led there, such as ChatGPT-User or Perplexity-User | No answers are currently fetching your pages in real time |

The full list is in [AI crawlers we track](https://docs.rankraft.ai/agent-analytics/bots/crawlers.md).

## Symptoms and likely causes

| What you see | Likely cause | Where to look |
| - | - | - |
| No AI crawler hits at all | A block, or the integration isn't sending | [AI access check](https://docs.rankraft.ai/agent-analytics/bots/ai-access-check.md), [Request log](https://docs.rankraft.ai/agent-analytics/dashboard/logs.md) |
| Plenty of training crawls, no AI assistant fetches | Your pages aren't being picked for live answers | Prompts where you're never named, in [Prompts & Answers](https://docs.rankraft.ai/analytics/prompts/prompts-and-answers.md) |
| Crawlers stop at the homepage | Weak internal linking or a missing sitemap | [Website Audit](https://docs.rankraft.ai/agent-analytics/dashboard/pages.md) |
| A page is crawled but never cited | AI prefers another source for the same question | [Where to get featured](https://docs.rankraft.ai/analytics/citations/citations-and-sources.md#where-to-get-featured) and the answers for that prompt |
| Cited pages with no recent crawl | Errors, redirects or a new block on those pages | [Crawl errors](https://docs.rankraft.ai/agent-analytics/dashboard/bots.md#crawl-errors) |
| Lots of crawler activity on a topic you don't track | A gap in your prompt set | Add prompts for it. See [Writing effective prompts](https://docs.rankraft.ai/analytics/prompts/writing-effective-prompts.md) |
| Genuine GPTBot or Googlebot hits marked **Spoofed** | Your integration reports a proxy's IP | [Crawler verification](https://docs.rankraft.ai/agent-analytics/crawler-verification.md#why-genuine-crawlers-can-show-as-spoofed) |

## Making a page easy to quote

Once AI can reach and read a page, how it's written decides whether it gets quoted:

* **Answer first.** Open each section with one sentence that directly answers the question the section is about. Put the detail after it.
* **Keep sections self-contained.** An engine may quote one passage on its own. A section that only makes sense after the previous one is hard to quote.
* **Match the shape of the question.** A "best X for Y" question wants a short list with reasons. A "how much does X cost" question wants a number.
* **Put the content in the HTML.** Text that only appears after JavaScript runs is invisible to AI crawlers.
* **Describe images.** Write meaningful `alt` text for images that carry information, such as charts or product photos.
* **Say who you are.** If AI cites your page without naming you, make the brand, product and what it's for explicit on the page.

## Measure the result

Many of these problems also appear as cards in the [Action Center](https://docs.rankraft.ai/analytics/action-center/about.md), such as a `robots.txt` block. When you fix one, mark its card done. RankRaft AI compares the prompts the card targeted before and after, against prompts it didn't target, and reports whether the metric moved. See [Measuring results](https://docs.rankraft.ai/analytics/action-center/measuring-results.md).


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.