Original research

AI Bot Blocking Index: How 390 B2B SaaS Sites Treat AI Crawlers

Almost no B2B SaaS company blocks AI search crawlers in robots.txt. We fetched the robots.txt of 390 B2B SaaS websites and checked how each one treats 17 known AI crawlers, from ChatGPT and Perplexity's retrieval bots to training crawlers like GPTBot and CCBot. To avoid a big-brand bias, the sample spans two cohorts: 153 recognizable established brands and 237 active growth-stage B2B SaaS companies from the Y Combinator directory. Across the 362 sites with a readable robots.txt, only 3 (0.8%) block an AI search crawler. The real gatekeeper sits one layer up: 48.2% of these sites are served behind Cloudflare, whose default bot rules can block AI crawlers regardless of what robots.txt says.

390 B2B SaaS sites · 17 AI crawlers · scanned 2026-06-27 · 362 reachable, 28 unreachable

Key findings

Block rate by crawler

Across the 362 reachable sites, every AI crawler is blocked by a low single-digit share or fewer. The most-blocked crawler is Common Crawl's CCBot at 2.2%; the search and user-initiated retrieval bots that actually feed AI answers sit at or below 1.7%.

Established brands vs growth-stage startups

Splitting the sample by maturity rules out a big-brand bias: both cohorts overwhelmingly leave AI crawlers open. Established brands block an AI search crawler at 1.3% and growth-stage YC companies at 0.5%. The one place they diverge is the edge: 50.2% of the YC companies sit behind Cloudflare versus 45.1% of the established brands, so the silent edge-block exposure is actually higher among younger companies.

What this means for AI visibility

If you want AI engines to read your pages, robots.txt is rarely the problem. The data shows B2B SaaS sites overwhelmingly leave AI crawlers alone in robots.txt: only 55 of 362 reachable sites have any explicit AI rule, and not one uses a blanket Disallow: /. The blocking that does happen is concentrated on training crawlers (GPTBot, CCBot, Bytespider), not the search and retrieval bots that decide whether you get cited.

The bigger exposure is the edge. With 48.2% of sites behind Cloudflare, a default bot-management rule can challenge or block an AI crawler before it ever reaches your robots.txt. A page can be perfectly permissive in robots.txt and still earn zero citations because the WAF turned the crawler away. For the full explainer on which bots to allow and how to fix the Cloudflare default, read AI Crawlers in 2026: which bots to allow and which to block.

Methodology

The sample combines two documented cohorts, deduplicated: 153 recognizable, established B2B SaaS brands (a curated convenience sample across CRM, marketing, dev tooling, analytics, collaboration, HR/finance, security, and support), and 237 active B2B SaaS companies from the Y Combinator company directory (the public dataset, filtered reproducibly to companies tagged both B2B and SaaS, with status Active, Public, or Acquired, a resolvable website, and a team of 10 or more). The two cohorts let us check whether company maturity changes the result. It does not, materially.

Each robots.txt was fetched on 2026-06-27 and evaluated against 17 known AI crawler user-agents using the same RFC 9309 parser that powers our free AI Bot Access Checker. 28 of the 390 sites were unreachable, returned no robots.txt, or blocked our request at scan time; the remaining 362 were parsed. A site is counted as effectively blocking a crawler when its robots.txt has an explicit Disallow: / for that user-agent, or a wildcard (User-agent: *) Disallow: / with no overriding Allow. Cloudflare edge behavior is reported as a separate exposure signal, not counted as a robots.txt block. The full per-site, per-crawler results are below and downloadable.

The full dataset (390 sites)

Every scanned site, sorted by domain, tagged by cohort. Rows that block an AI search crawler are highlighted. Click any column header to sort. Download the complete per-crawler matrix as JSON or CSV.

AI-Advisors AI Bot Blocking Index, 390 B2B SaaS sites, scanned 2026-06-27. Free to cite and reuse with attribution (CC BY 4.0).

Check your own site

Run your domain through the free AI Bot Access Checker to see, in seconds, which AI crawlers your robots.txt allows and blocks, then confirm your edge layer is not turning them away. For the strategy behind the data, read AI Crawlers in 2026 and how to optimize a website for AI search.