What you will see in your logs
We send three shapes of user agent, all linking back to rankraft.ai/bot:The access check is the one that looks alarming
To tell you whether you are blocking GPTBot, we have to ask your server what it would do if GPTBot knocked. So that third user agent begins with the token of the bot being tested. A line likeGPTBot/1.0 (+https://rankraft.ai/bot) RankRaft-AccessCheck in your log is us, not OpenAI.
This is deliberate and we don’t hide it. Edge rules match on the bot token as a substring, so the prefix is what makes the test meaningful, while the suffix and the URL make it obvious who actually called.
Other RankRaft user agents you may see:
RankRaft-Test/1.0: test events and the cache check from the WordPress plugin, on your own server.RankRaft-Callback/1.0: a signed request telling your WordPress plugin to resume logging.
What we request
- Your
/robots.txt, your homepage and a small number of requests to a single page. - Requests are spaced 400 ms apart. We don’t crawl your whole site, and we don’t follow links.
- We send no cookies, submit no forms and run no JavaScript. Every request is a plain GET or HEAD.
- A check runs when a customer asks for one, during setup, and on a monthly schedule. Not continuously.
How to allow us
Nothing is required. If you want to be explicit,RankRaftBot is the token to allow. Our access check deliberately identifies as the bot under test, so a rule written against GPTBot will also stop the check. That’s the correct result, because it’s what we’re measuring.
How to block us
Add this to yourrobots.txt and we’ll stop:
robots.txt