One file in your Next.js app, and the AI crawlers stop being invisible
AI crawlers do not execute JavaScript. That single fact means a browser tracking snippet, however well installed, can never record a visit from GPTBot or ClaudeBot. This integration is one file you add to your Next.js project. It runs on your server, reads the request before your page renders, and reports every AI crawler hit - the traffic your analytics has been silently missing.
Overview
The gap is architectural, not a configuration mistake. Analytics snippets run in a browser after the page loads. AI crawlers fetch your HTML, read it, and leave without ever running a script, so they never appear in JavaScript-based reporting. Sites relying only on a snippet see a crawler count near zero and reasonably conclude that AI is not reading them, when the truth is that nothing was watching.
Middleware closes that gap where the request actually arrives. It inspects the User-Agent server-side, matches it against 16 published AI crawler signatures - GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-User, Claude-SearchBot, PerplexityBot, Perplexity-User, Google-Extended, Bytespider, CCBot, and more - and reports the path, referrer, and client IP for each hit. Everything else passes through untouched.
Because the request arrives with a real client IP, we can go one step further than user-agent matching. Anyone can claim to be GPTBot in a header. We check the source IP against the crawler operator's own published address ranges, so a verified visit is genuine, not merely self-declared. A match proves identity; a non-match is reported as unproven rather than as forgery, because published ranges go stale and wrongly calling a real crawl fake would be the worse error.
How it works
Records the crawlers JavaScript cannot see
Matches 16 AI crawler signatures at the server, including every major index crawler: GPTBot, OAI-SearchBot, ClaudeBot, Claude-SearchBot, PerplexityBot, and Google-Extended. These are the bots that build the indexes AI assistants answer from, and a browser snippet has never been able to record a single one of them.
IP-verified crawler identity
A User-Agent is only a claim. Because middleware sees the real client IP, each visit is checked against the operator's own published address ranges. Verified visits are marked as such in your dashboard, and unverified ones are labelled not proven rather than accused of spoofing.
Captures robots.txt and llms.txt requests
The supplied matcher deliberately does not exclude robots.txt, sitemap.xml or llms.txt. Those are the first files an AI crawler fetches and the clearest signal that a crawler has arrived at all, so excluding them would discard the most useful data on the site.
Adds no latency, and cannot break your site
Reporting runs inside waitUntil, so your page is returned to the visitor without waiting for us, and the network call swallows its own errors. If our collector were slow or offline, your site would be unaffected. Requests that are not AI traffic make no network call at all.
Non-AI traffic never leaves your infrastructure
Filtering happens in your own middleware, before anything is sent. Human visitors, their IP addresses, and the pages they read are never transmitted to us. Only requests matching a known AI crawler signature are reported.
Works on any host, not just Vercel
This is a Next.js capability, not a hosting one. It runs the same on Vercel, a self-hosted Node server, or a Docker container. The one exception is a fully static export, which has no server to run middleware on.
Set up
One file, copied from your dashboard with your website key already filled in. There is no OAuth, no account linking, and nothing to configure on our side.
Open Settings > Tracking Setup in AI-Advisors and copy the Next.js middleware block. The code arrives with your website's key already in place, so there is nothing to substitute.
Save it as middleware.ts in your project root, next to your app directory. On Next.js 16 or later the convention was renamed, so name the file proxy.ts and rename the exported function to proxy - everything else is identical. If you already have a middleware file, merge the matcher into yours rather than replacing it.
Deploy. AI crawler visits appear in AI Analytics within minutes of the next crawl, and Settings > Tracking Setup shows Next.js middleware as reporting once the first one arrives.
Takes a few minutes
Data and permissions
There is no OAuth, no API key exchange, and no access to your hosting account. The file authenticates with the same per-website key used by the tracking snippet, and only AI crawler requests are ever sent.
- AI crawler requests only: URL path, user agent, referrer, and client IP address
- The time of each request
- Nothing. This integration is read-only and never modifies your site, your content, or your configuration
- Human visitor traffic - filtering happens in your own middleware, so non-AI requests are never transmitted
- Request or response bodies, form contents, cookies, and session data
- Your hosting provider account, deployment settings, or environment
- The response itself - the file observes requests and changes nothing about what your visitors receive
No OAuth scopes are requested because there is no OAuth. The file authenticates with the per-website key from Settings > Tracking Setup, sent over HTTPS, and the platform scopes every write to that one website.
Good to know
- It works on every plan. The code is generated from your dashboard and there is no feature gate on collection.
- It reports AI crawlers only. Human visits from AI assistants and form-submit conversions stay with the JavaScript snippet, so running both records each thing once and never double counts.
- Middleware runs before your page renders and does not wait for the response, so it cannot report the status code a crawler received. If you need that, or you serve pages from a CDN cache that middleware never sees, the Cloudflare integration covers both.
- The matcher skips build assets and static media, so images, stylesheets, and scripts do not generate reports.
- If your repository is public, move the key out of the file into an environment variable. The generated code notes where.
- Static exports have no server and therefore no middleware. Any rendered Next.js deployment works.
Frequently asked questions
Why can't the JavaScript snippet track AI crawlers?
Because AI crawlers do not run JavaScript. GPTBot, ClaudeBot, OAI-SearchBot and PerplexityBot fetch your HTML, read it, and leave. A tracking snippet only executes in a browser after the page loads, so it is never given the chance to run. This is architectural, not a bug or a misconfiguration, and no amount of snippet tuning changes it. Middleware runs on the server where the request actually arrives, which is why it can see them.
Is this a Vercel integration?
No, it is a Next.js one. The file runs anywhere Next.js runs a server: Vercel, a self-hosted Node process, or a Docker container. We deliberately avoid calling it a Vercel integration because that would wrongly exclude Next.js sites hosted elsewhere and wrongly imply support for non-Next.js sites that happen to be on Vercel. The only Next.js deployment it cannot work with is a fully static export, which has no server.
Will this slow down my site?
No. Requests that are not AI crawlers make no network call at all and pass straight through. For requests that are, the report is sent inside waitUntil, which returns your page to the visitor without waiting for us to finish. The call also swallows its own errors, so even if our collector were offline your site would serve normally.
What data leaves my server?
Only AI crawler requests, and only four fields: the URL path, the user agent, the referrer, and the client IP address. Filtering happens inside your own middleware before anything is sent, so human visitors are never transmitted. Request bodies, form contents, cookies, and session data are never read.
What does IP verification actually prove?
That a visit claiming to be a given crawler really came from that operator. Anyone can put GPTBot in a User-Agent header, so we check the request's source IP against the crawler operator's own published address ranges. A match confirms identity. A non-match is recorded as not proven rather than as spoofing, because published ranges go stale and operators add addresses before updating their lists - wrongly labelling a genuine crawl as fake would corrupt your data in the direction that matters most.
I already have a middleware file. Will this conflict?
Merge rather than replace. Next.js supports one middleware file per project, so combine the AI bot check into your existing function and merge the matcher patterns. The important part to preserve is that the matcher must not exclude robots.txt, sitemap.xml or llms.txt, since those are the requests most worth recording.
Do I still need the JavaScript snippet?
Yes, for a different job. The snippet records human visits arriving from AI assistants and detects form-submit conversions, which requires inspecting the page in a browser and is impossible from the server. Middleware records the crawlers. They cover different halves and do not overlap, so running both records each thing exactly once.
