Mostly no. The few that added blocks aimed almost all of them at AI training crawlers, never at AI search. We scanned the robots.txt of the same 390 B2B SaaS websites on June 27 and again on September 15, 2026, the date Cloudflare set for its new AI crawler defaults. In the eleven weeks between the two scans, B2B SaaS companies wrote AI crawlers into their robots.txt files by name, and most of those new rules let the crawlers in.
- More rules. Sites naming AI crawlers rose from 55 to 76 of the 358 we could read both times
- Mostly yes. 23 of the 24 companies writing AI-crawler rules for the first time let at least one crawler in
- No new search blocks. Seven companies added blocks, and none of the new blocks hit an AI search crawler
- The edge decides. 32 of the 62 sites that let GPTBot in are served through Cloudflare, whose settings can overrule robots.txt
What changed since June: more rules, mostly letting crawlers in
B2B SaaS companies spent the summer writing AI crawlers into robots.txt by name, and most of the new rules say yes. This is the September 2026 edition of our AI Bot Blocking Index: the same panel of 390 companies and the same 17 crawlers as the June scan, so every change below is a change on the same site.
Rules that let an AI crawler in grew by 203 between the two scans. Rules that block one grew by 11. Here, a rule lets a crawler in when the file names it and does not block the whole site. Some of those rules still fence off individual pages: GitHub names GPTBot, OAI-SearchBot, ClaudeBot and PerplexityBot in one group with page-level allow and disallow lines.
Naming a crawler did not unlock anything. Under RFC 9309, the robots.txt standard, a crawler a file does not name follows the file's wildcard rules, and no site in the panel blocks everything with a wildcard. These rules record a decision the companies chose to write down.
Twenty-four companies wrote AI-crawler rules for the first time, and 20 of them only let crawlers in. Hootsuite, Unbounce, Zscaler, Palo Alto Networks, Recurly and ChurnZero are among them. The change was even across both cohorts: 21.6% of established brands and 21.5% of Y Combinator companies with a readable robots.txt now have an explicit AI-crawler rule, up from 16.1% and 14.6% in June. One site, suger.io, wrote the panel's first rule for OAI-AdsBot, OpenAI's ChatGPT Ads crawler, and it lets it in.
B2B SaaS did not spend the quarter before Cloudflare's new defaults locking AI crawlers out. It spent it writing them in by name.
The seven companies that added blocks, and what they left open
Seven companies added a block this quarter, and six of them blocked only AI training crawlers. Five of the seven name AI search crawlers in the same file without blocking them.
Bytespider, ByteDance's crawler, was the most common new target. GitHub, Postman and Amplitude each added a site-wide block for it and no other new block. It is now the most-blocked crawler in the panel, at 12 of 362 readable sites, overtaking Common Crawl's CCBot at 8.
Three Y Combinator companies went wider. getthematic.com and Zuddl each blocked seven training crawlers, and Sendbird, which already blocked GPTBot in June, added five more. Figma is the one site that fits neither pattern. It added blocks for ClaudeBot and Claude-User, Anthropic's crawler that fetches a page when a Claude user asks for it, and it has blocked three AI search crawlers since at least June.
No company newly blocked an AI search crawler. Only two sites block one at all: Figma, and ZoomInfo, which blocks DuckDuckGo's DuckAssistBot. The count fell from three to two because panoramaed.com removed all 16 of its AI-crawler rules, including an old block on Applebot.
What Cloudflare announced for September 15, 2026
Cloudflare announced new defaults that allow search crawlers but block AI training and agent bots on pages that show ads. According to Cloudflare's July 1 press release, they apply to new customers, to new sites added by existing customers, and to existing free-plan customers who have not changed their settings by September 15.
Three details matter more than the headline:
- Googlebot is included. Cloudflare's documentation says crawlers that combine search and training are blocked by every setting that blocks AI training, including the older Block AI bots option. That matters because Google's AI Overviews run on the ordinary Googlebot crawl.
- The old toggle retires. The same documentation marks Block AI bots as deprecating on September 15, 2026, while the separate Search, Agent and Training settings each offer three options: block on all pages, block only on pages with ads, or allow.
- Announced, not confirmed. Cloudflare said it would test and finalize the defaults before September 15. We found no Cloudflare page confirming they went live, so this post describes what was announced.
On the default settings, only pages that Cloudflare's automated detection finds showing ads are affected. The riskier case is a site that chose to block AI training on every page, because Cloudflare says that choice also blocks Googlebot on every page.
On Cloudflare, a choice to block AI training on every page is also a choice to block Googlebot on every page.
The companies that blocked drew the line Cloudflare now draws
The B2B SaaS companies that blocked AI crawlers separated training from search, the same split Cloudflare's new settings are built on. Of the 16 sites in the panel that block at least one training crawler, 7 let AI search crawlers in and block none of them: Amplitude, getthematic.com, GitHub, Postman, Secoda, Sendbird and Volopay.
Our categories are close to Cloudflare's but not identical. We class Applebot as a search crawler, while Cloudflare treats it as one crawler doing both jobs. Cloudflare's Agent group includes ChatGPT-User, which our index classes as user-initiated, and only one site blocks a crawler of that kind: Figma, which blocks ChatGPT-User and Claude-User.
Far more sites went the other way from Cloudflare's training default. 62 now name GPTBot, OpenAI's training crawler, without blocking it, up from 44 in June. For the full case on which OpenAI crawler to keep, see GPTBot vs OAI-SearchBot.
Half the sites that let GPTBot in sit behind Cloudflare
A robots.txt rule is a request, and Cloudflare's settings are enforcement that runs before the file is ever read. Of the 62 sites that let GPTBot in by name, 32 are served through Cloudflare. Of the 70 sites that let any AI crawler in by name, 35 are.
Our scan cannot see which Cloudflare plan those sites are on, what their dashboard settings say, or whether any of their pages show ads. So we cannot say how many of them block the crawlers their robots.txt invites. What the data does show is that for half of the companies writing yes to AI crawlers, the final answer depends on a setting outside that file. The standard is blunt about it: robots.txt rules “are not a form of access authorization.”
That gap between the file and the edge is the two-layer problem laid out in our guide to which AI crawlers to allow, and it can quietly cost you the AI citations the file was meant to earn. The quickest outside signal is your server logs: if AI bot traffic from the crawlers you allow never shows up, something in front of your origin may be turning them away. Our bot activity guide shows how to check.
What to check this week if your site is on Cloudflare
Four checks, in order, cover the changes Cloudflare announced.
- AI bot policies. In the Cloudflare dashboard, go to Security Settings, then Configure AI bot policies. If you want AI search engines to cite you, Search should be set to allow.
- Training and Googlebot. If Training blocks on all pages, crawlers that combine search and training are blocked on all pages too, and Cloudflare names Googlebot, Applebot and Bingbot. Decide whether keeping AI training out is worth that.
- The legacy setting. Cloudflare marks Block AI bots as deprecating on September 15, 2026. If your team ever switched it on, confirm the new Search, Agent and Training settings say what you intend.
- Blocked crawlers. Under Security, then Analytics, the Events tab shows blocked AI bot traffic. Bot Fight Mode can challenge crawlers as well, and Cloudflare says custom rules cannot skip it, so if it is stopping a crawler you need, switch it off.
Then confirm the file itself says what you think. A crawler is blocked from your whole site only when its group, or the wildcard group it falls back to, contains a bare Disallow: /. An empty Disallow: line allows everything.
The free AI Bot Access Checker reads your live robots.txt against 25 AI crawlers, using the same parser as this index, and flags whether your site is served through Cloudflare. No signup.
Run the free AI Bot Access Checker →What this scan cannot tell you
The scan reads robots.txt and its response headers. It does not test whether a crawler actually gets through.
- Enforcement. robots.txt shows what a company asks crawlers to do. A Cloudflare setting or firewall rule can override it, and the scan tests neither.
- Cloudflare plans and settings. We detect Cloudflare from response headers. Plan, dashboard settings and ad detection are invisible to us, so we make no estimate of how many sites Cloudflare's defaults affect.
- Motives. A robots.txt change shows what changed, never why.
- The wider web. This is a fixed panel of 390 B2B SaaS companies, not a sample of all websites, so its rates should not be read as web-wide.
- Timing. The September scan ran on the date Cloudflare set. It measures the quarter before the new defaults, not their effect.
How we measured
We fetched the robots.txt of the same 390 B2B SaaS websites on June 27 and September 15, 2026, and checked each file against the same 17 crawlers. The panel is fixed: 153 established B2B SaaS brands and 237 active B2B SaaS companies from the Y Combinator directory.
Each file is parsed per crawler with the RFC 9309 parser behind our AI Bot Access Checker. A crawler counts as blocked when its group, or the wildcard group it falls back to, disallows the whole site. It counts as let in when the file names it without blocking the whole site, and as unnamed otherwise. Change figures use only the 358 sites whose robots.txt was readable in both scans. Four sites became unreachable and four came back, and none of those count as changes. Rates for a single edition use that scan's 362 readable sites.
Every site, crawler and figure is downloadable under CC BY 4.0 in the September 2026 edition, and the June 2026 edition stays at its own URL, so both sets of numbers remain citable.
Frequently Asked Questions
#Are B2B SaaS companies blocking more AI crawlers in 2026?
Slightly more blocks, and far more rules that let crawlers in. Across 358 B2B SaaS sites scanned in both June and September 2026, rules blocking an AI crawler rose from 44 to 55, while rules letting one in rose from 349 to 552. No site newly blocked an AI search crawler, and only 2 of 362 readable sites, 0.6%, block one at all.
#Does Cloudflare block AI crawlers by default?
Not on every site. Cloudflare announced defaults for September 15, 2026 that allow search crawlers but block AI training and agent bots on pages that show ads, for new customers, new sites, and existing free-plan customers who have not changed their settings. Every Cloudflare site can set Search, Agent and Training separately. Cloudflare described September 15 as a deadline to finalize the defaults, and we found no Cloudflare page confirming they went live.
#Can Cloudflare block Googlebot?
Yes, if you block AI training. Cloudflare's documentation says crawlers that combine search and training are blocked by any setting that blocks AI training, including the older Block AI bots option, and it names Googlebot, Applebot and Bingbot as examples. A Training setting that blocks on all pages therefore blocks Googlebot on all pages.
#What percentage of B2B SaaS websites block GPTBot?
1.9%. In our September 15, 2026 scan of 390 B2B SaaS websites, 7 of the 362 sites with a readable robots.txt block GPTBot, OpenAI's training crawler. Far more go the other way: 62 sites name GPTBot in robots.txt without blocking it, up from 44 in June.
#Can you block AI training crawlers but still allow AI search crawlers?
Yes. robots.txt applies rules per crawler, so a file can disallow a training crawler such as GPTBot while allowing OAI-SearchBot and PerplexityBot. Seven sites in our panel block at least one training crawler while letting AI search crawlers in, including GitHub, Postman and Amplitude. On Cloudflare the same split is a setting, with one catch: blocking training also blocks crawlers that combine search and training, such as Googlebot.
#Does robots.txt stop AI crawlers?
Only crawlers that choose to follow it. RFC 9309, the robots.txt standard, states that its rules are not a form of access authorization. To enforce a block, or to confirm a block is not happening, check the layer in front of your site, such as Cloudflare's AI bot settings or your firewall rules.
#Which AI crawler do B2B SaaS companies block most?
Bytespider, ByteDance's crawler. In our September 15, 2026 scan it is blocked by 12 of 362 readable B2B SaaS sites, 3.3%, overtaking Common Crawl's CCBot at 8. GitHub, Postman and Amplitude all added Bytespider blocks between June and September 2026.
