See which AI crawlers reach your site, and what your site actually told them
AI crawlers do not execute JavaScript, so a browser analytics snippet can never record one. This is a small Cloudflare Worker that runs at the edge of your own Cloudflare account. It watches requests as they arrive, reports every AI crawler hit, and, uniquely, records the status code your site returned - so a page quietly serving 403 to an AI crawler stops being invisible.
Overview
The gap is architectural. Analytics snippets run in a browser after the page loads. AI crawlers fetch your HTML, read it, and leave without ever running a script, so they never appear in JavaScript-based reporting. A site relying only on a snippet sees a crawler count near zero and reasonably concludes AI is not reading it, when in fact nothing was watching.
A Cloudflare Worker closes that gap at the edge, before the request reaches your server. It matches the User-Agent against 16 published AI crawler signatures, passes the request through untouched, and reports the hit. Nothing about your site's behaviour changes: the Worker returns exactly the response your origin produced.
Two things it records that an origin-side integration cannot. First, the status code: because the Worker wraps the fetch, it knows whether the crawler received a 200 or a 404 or a 403. Second, cache hits: the Worker runs before Cloudflare's cache lookup, so a crawl is recorded even when Cloudflare answers from cache and your origin is never contacted at all.
Because the request arrives with a real client IP, identity can be checked rather than assumed. Anyone can put GPTBot in a header, so each visit is matched against the crawler operator's own published address ranges. A match confirms it; a non-match is reported as unproven rather than as forgery, because published ranges go stale and wrongly calling a real crawl fake is the worse error.
How it works
Records the crawlers JavaScript cannot see
Matches 16 AI crawler signatures at the edge, including every major index crawler: GPTBot, OAI-SearchBot, ClaudeBot, Claude-SearchBot, PerplexityBot, and Google-Extended. These build the indexes AI assistants answer from, and a browser snippet has never been able to record a single one.
The status code each crawler received
The Worker wraps the request to your site, so it sees the response. A page returning 403 or 404 to an AI index crawler is a citation problem that produces no other symptom: your site looks fine to you, and the crawler quietly gets nothing. This is the one signal an origin-side integration structurally cannot capture, because it runs before the response exists.
Crawls that never touch your origin
Workers run before Cloudflare's cache lookup. When a crawler requests a page Cloudflare already has cached, your server is never contacted and origin-side tracking sees nothing at all. The Worker still records it, so cached pages do not become blind spots.
IP-verified crawler identity
A User-Agent is only a claim. Cloudflare passes the real client IP, so each visit is checked against the operator's own published address ranges. Verified visits are marked as such, and unverified ones are labelled not proven rather than accused of spoofing.
Adds no latency, and cannot break your site
Reporting runs inside waitUntil, so the response is returned without waiting for us, and the network call swallows its own errors. If our collector were slow or offline your site would be unaffected. Requests that are not AI crawlers make no extra network call at all.
Non-AI traffic never leaves your infrastructure
Filtering happens inside your own Worker, before anything is sent. Human visitors, their IP addresses, and the pages they read are never transmitted to us. Only requests matching a known AI crawler signature are reported.
Framework independent
It runs at Cloudflare, not in your application, so it does not care what your site is built with. WordPress, Next.js, Webflow, a static bucket, or a bespoke stack all work identically as long as the domain is on Cloudflare.
Set up
You deploy the Worker into your own Cloudflare account. We never ask for access to it, and there is no OAuth or API key exchange with us.
Add your domain to Cloudflare if it is not already there, and point its nameservers at the two Cloudflare provides. The free plan is sufficient; nothing in this integration needs a paid feature.
Clone the Worker from our repository, set the route to your domain in wrangler.jsonc, and add your website key as a Worker secret. The key comes from Settings > Tracking Setup in AI-Advisors.
Run wrangler deploy. AI crawler visits appear in AI Analytics within minutes of the next crawl, and Settings > Tracking Setup shows Cloudflare as reporting once the first one arrives.
Takes about 15 minutes
Data and permissions
The Worker runs in your Cloudflare account, deployed by you. We have no access to your Cloudflare account, your DNS, or your configuration. It authenticates to us with the same per-website key used by the tracking snippet, and only AI crawler requests are ever sent.
- AI crawler requests only: URL path, user agent, referrer, and client IP address
- The status code and response size your site returned to that crawler
- The time of each request
- Nothing. The Worker is read-only and returns exactly the response your origin produced, unmodified
- Human visitor traffic - filtering happens inside your own Worker, so non-AI requests are never transmitted
- Request or response bodies, form contents, cookies, and session data
- Your Cloudflare account, DNS records, firewall rules, and cache configuration
- The response your visitors receive - it is passed through unchanged
No OAuth scopes are requested because there is no OAuth and no connection to your Cloudflare account. You deploy the Worker yourself with your own credentials; it reports to us using the per-website key from Settings > Tracking Setup.
Good to know
- It works on every plan. There is no feature gate on collection, and Cloudflare's free plan is sufficient.
- It reports AI crawlers only. Human visits from AI assistants and form-submit conversions stay with the JavaScript snippet, so running both records each thing once and never double counts.
- It does not see requests Cloudflare itself blocked. Workers run after Cloudflare's firewall and bot management, so a crawler your own rules block never reaches the Worker. If you suspect that is happening, Cloudflare's AI Crawl Control dashboard shows blocked crawler traffic directly.
- Cloudflare's free Workers plan covers 100,000 requests a day. The Worker runs on every request matching its route, so invocations track your total traffic rather than your AI traffic; a busy site will need the paid Workers plan, which starts at five dollars a month.
- The route pattern should cover your whole site, not just the homepage. Crawlers fetch deep pages, and robots.txt is usually the first thing they request.
- Not on Cloudflare and running Next.js? The Next.js middleware integration records crawlers from your server instead, with the same IP verification.
Frequently asked questions
Why can't the JavaScript snippet track AI crawlers?
Because AI crawlers do not run JavaScript. GPTBot, ClaudeBot, OAI-SearchBot and PerplexityBot fetch your HTML, read it, and leave. A tracking snippet only executes in a browser after the page loads, so it is never given the chance to run. This is architectural rather than a misconfiguration, and no amount of snippet tuning changes it. A Worker runs at Cloudflare's edge where the request actually arrives, which is why it can see them.
Does this let me see AI crawlers my firewall is blocking?
No, and it is worth being precise about why. Cloudflare processes a request through its own security layers before handing it to a Worker, and a blocking rule ends the request there. So a crawler blocked by your WAF or bot management settings never reaches the Worker and cannot be reported by it. If you want to see blocked crawler traffic, Cloudflare's own AI Crawl Control dashboard shows it. What the Worker adds is what happened to crawlers that were allowed through: which pages they fetched, and what status code your site returned.
What does it see that the WordPress plugin or Next.js middleware doesn't?
Two things. The status code your site returned, because the Worker wraps the request and sees the response, whereas origin-side integrations run before the response exists. And crawls served from Cloudflare's cache, which never reach your origin at all, so origin-side tracking has no way to know they happened. It is also framework independent, so it works on sites where neither of the others can be installed.
Will this slow down my site or risk breaking it?
Requests that are not AI crawlers pass straight through with no extra network call. For those that are, the report is sent inside waitUntil, which returns the response without waiting for us, and the call swallows its own errors so a problem on our side cannot surface on yours. The Worker returns exactly the response your origin produced and modifies nothing.
What data leaves my infrastructure?
Only AI crawler requests: the URL path, user agent, referrer, client IP address, status code, and response size. Filtering happens inside your own Worker before anything is sent, so human visitors are never transmitted. Request bodies, form contents, cookies, and session data are never read.
Do you get access to my Cloudflare account?
No. You deploy the Worker yourself using your own credentials, and it lives in your account under your control. We never request access to your Cloudflare account, DNS, firewall rules, or cache settings. The Worker reports to us with the per-website key from Settings > Tracking Setup, the same one the tracking snippet uses.
What does it cost?
Nothing from us on any plan, and Cloudflare's free plan is enough to run it. The one thing to watch is Cloudflare's own Workers pricing: the free tier covers 100,000 requests a day, and because the Worker runs on every request matching its route, that limit tracks your total site traffic rather than your AI traffic. Busy sites will need Cloudflare's paid Workers plan, which starts at five dollars a month.
