> ## Documentation Index
> Fetch the complete documentation index at: https://docs.rankraft.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# AI crawler traffic to a warehouse

> Load yesterday's AI crawler and referral traffic into BigQuery, Snowflake or Postgres every day.

Run this once a day (cron, Airflow, a scheduled Worker) to append yesterday's traffic. Reports are aggregated, so a day of even heavy crawler traffic is a few hundred rows.

```python theme={"system"}
import datetime as dt, os, requests

BASE = "https://api.rankraft.ai"
HEADERS = {"X-API-Key": os.environ["RANKRAFT_API_KEY"]}
SITE = os.environ["RANKRAFT_SITE_ID"]
TZ = "Asia/Karachi"  # your reporting time zone

day = (dt.datetime.now(dt.timezone.utc) - dt.timedelta(days=1)).date().isoformat()

def fetch(path, query):
    rows, cursor = [], None
    while True:
        body = {**query, **({"cursor": cursor} if cursor else {})}
        res = requests.post(f"{BASE}{path}", json=body, headers=HEADERS, timeout=60)
        if res.status_code == 429:
            import time; time.sleep(int(res.headers.get("Retry-After", "60"))); continue
        res.raise_for_status()
        page = res.json()
        rows += page["data"]
        cursor = page["info"]["next_cursor"]
        if not cursor:
            return rows

common = {"site_id": SITE, "start_date": day, "end_date": day, "timezone": TZ, "limit": 500}

bots = fetch("/v1/reports/bots", {**common,
    "group_by": ["date", "bot_name", "page_type"],
    "metrics": ["hits", "pages", "verified_share", "error_hits"]})

referrals = fetch("/v1/reports/referrals", {**common,
    "group_by": ["date", "referral_source", "path"],
    "metrics": ["visits"]})

# load(bots, "rankraft_bot_daily"); load(referrals, "rankraft_referral_daily")
```

## Notes

* Make the load idempotent: delete the day's rows before inserting, so a re-run after a failure does not double-count.
* Late data exists. Integrations retry, so a few hits for a day can arrive after midnight. Re-loading the previous three days each run catches them.
* Spoofed crawler hits are excluded. To store them too, add `{"field": "verified", "op": "in", "value": ["verified", "failed", "partial", "unverified"]}` as the bots `filter`.


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.