Custom log integration

From a log drain (Vercel, Netlify), Fastly or your own server, POST batches of requests to RankRaft AI. Unlike the snippet, this sees crawlers as well as referrals. Sees: everything: AI crawlers, search bots and AI referrals.

Prerequisites

A RankRaft AI site key. Open Integrations, choose Any other site and click Create site key. See Your site key.
A process that can read your request logs and make outbound HTTPS requests.

Endpoint

POST https://ingest.rankraft.ai/v1/events

Authentication

Send your site key in the X-Site-Key header, with Content-Type: application/json. Store the key as a secret, not in source control.

Request body

{
  "source": "custom",
  "events": [
    {
      "ts": "2026-10-15T10:00:00Z",
      "host": "example.com",
      "method": "GET",
      "path": "/blog/post",
      "ip": "20.171.206.44",
      "user_agent": "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; GPTBot/1.2; +https://openai.com/gptbot",
      "status": 200,
      "bytes": 18234
    }
  ]
}

Fields

Field	Required	Description
`host`	Yes	Hostname requested. Must be one of your site’s allowed hostnames.
`path`	Yes	Path requested. Any query string is removed before storage.
`ip`	Recommended	The client IP address, not your load balancer’s. Used for crawler verification, then stored only as a hash.
`user_agent`	Recommended	The User-Agent header. Needed to recognise crawlers.
`referer`	Recommended	The Referer header. Needed to detect AI referrals. Leave the field out when there is none. Don’t send `null`.
`query`	Optional	The raw query string without `?`. Used to detect `utm_source` on AI referrals.
`status`	Optional	HTTP status your site returned (100–599). Powers Crawl errors.
`bytes`	Optional	Response size in bytes.
`ts`	Optional	ISO-8601 time of the request. Defaults to the time we receive it.
`method`	Optional	HTTP method. Defaults to `GET`.
`country`	Optional	ISO 3166-1 alpha-2 country code, such as `GB`.

Send every request, not just bot requests. RankRaft AI classifies them for you. Human traffic is counted toward your daily totals so that AI share of traffic is accurate, and it isn’t stored individually.

Examples

await fetch("https://ingest.rankraft.ai/v1/events", {
  method: "POST",
  headers: {
    "X-Site-Key": process.env.RANKRAFT_SITE_KEY,
    "Content-Type": "application/json",
  },
  body: JSON.stringify({ source: "custom", events }),
});

import os, requests

requests.post(
    "https://ingest.rankraft.ai/v1/events",
    headers={"X-Site-Key": os.environ["RANKRAFT_SITE_KEY"]},
    json={"source": "custom", "events": events},
    timeout=5,
)

A successful request returns 202 Accepted:

{ "accepted": 1, "rejected": 0 }

If some events are invalid, the valid ones are still stored. The response counts the rejected ones and lists up to five issues with the index of each failing event.

Best practices

Batch. Up to 1,000 events and 5 MB per request. Send a batch every few seconds to a minute, not one request per page view.
Never block the response. Send after the response has gone out, or from a background queue.
Retry sensibly. On 429 or any 5xx, keep the batch and retry with backoff. On 403 or 410 the site is paused or removed, so stop sending. On any other 4xx, drop the batch: it won’t succeed on retry.
Send the real client IP. If you send your proxy’s address, every genuine crawler shows as Spoofed.

Check it’s working

Press Check again at the top of Integrations. It turns to Connected within a minute of your first batch. Then open Integrations → Advanced → Delivery log: each batch shows how many requests were stored, counted, skipped or rejected. A high Skipped count means the hostnames you send aren’t on your allowed list. Need help? Email [email protected].

Overview

Dashboard

Integrations

WordPress plugin

Bots

Prerequisites

Endpoint

Authentication

Request body

Fields

Examples

Best practices

Check it’s working

​Prerequisites

​Endpoint

​Authentication

​Request body

​Fields

​Examples

​Best practices

​Check it’s working

Prerequisites

Endpoint

Authentication

Request body

Fields

Examples

Best practices

Check it’s working