Work through it in order
Each step only makes sense once the one before it passes.1
Can AI crawlers reach the site?
Open AI Crawlers → Am I blocking AI by accident? or the AI crawler access card on Integrations. A block in
robots.txt, a noai or noindex tag, or a firewall rule stops everything downstream. Nothing else on this list matters until this passes. See AI access check.2
Are all three kinds of crawler visiting?
On AI Crawlers, group the table By type. The types do different jobs, and blocking or missing one doesn’t affect the others. See the table under Crawler types below. AI assistant visits are the closest thing in your logs to a live answer reading your site.
3
Are they reading the pages that matter?
Open Website Audit. Check the page-type breakdown and look for your product, pricing and comparison pages in the pages table. If crawlers only ever reach your homepage and a few top-level pages, the deeper pages are hard to find: improve internal links and make sure they’re in your sitemap.
4
Are they getting errors?
Check AI Crawlers → Crawl errors. A page that answers crawlers with a 404 or 500 is a broken page as far as the engine is concerned, even if it works in your browser. Pages that only render with JavaScript cause the same problem, because AI crawlers don’t run JavaScript.
5
Read, but not cited?
Open Citations & Sources → Your pages AI cites and compare it with the pages crawlers read. A page that’s read regularly but never cited is usually a content problem: AI found it and preferred another source. Look at what the cited pages for the same prompts do differently.
6
Cited, but no longer crawled?
The same card shows the last verified AI crawl of each cited page, and how many cited pages had no verified crawl in the range. A cited page that crawlers have stopped visiting is often the next one to drop out of answers. Check it for errors, redirects or a new block.
Crawler types
The full list is in AI crawlers we track.
Symptoms and likely causes
Making a page easy to quote
Once AI can reach and read a page, how it’s written decides whether it gets quoted:- Answer first. Open each section with one sentence that directly answers the question the section is about. Put the detail after it.
- Keep sections self-contained. An engine may quote one passage on its own. A section that only makes sense after the previous one is hard to quote.
- Match the shape of the question. A “best X for Y” question wants a short list with reasons. A “how much does X cost” question wants a number.
- Put the content in the HTML. Text that only appears after JavaScript runs is invisible to AI crawlers.
- Describe images. Write meaningful
alttext for images that carry information, such as charts or product photos. - Say who you are. If AI cites your page without naming you, make the brand, product and what it’s for explicit on the page.
Measure the result
Many of these problems also appear as cards in the Action Center, such as arobots.txt block. When you fix one, mark its card done. RankRaft AI compares the prompts the card targeted before and after, against prompts it didn’t target, and reports whether the metric moved. See Measuring results.