GPTBot vs OAI-SearchBot: Why Blocking the Wrong One Kills Your ChatGPT Visibility

GPTBot collects training data. OAI-SearchBot powers ChatGPT Search citations. Most "block OpenAI bots" advice on the public web treats them as a single decision. They are not. They are two separate crawlers with two separate jobs, and blocking the wrong one quietly removes you from ChatGPT visibility while accomplishing nothing about training data.

This post is the binary decision: which of OpenAI's two main crawlers should your robots.txt block, which should it allow, and what changes when you get it wrong. Four worked robots.txt scenarios, a server-log signal map, the Cloudflare default-block that silently breaks both, and the AI Visibility Lift bridge that extends the same discipline to Anthropic, Google, and Perplexity crawlers.

What GPTBot and OAI-SearchBot actually do

OpenAI's bot documentation describes each crawler plainly. The short version:

GPTBot

GPTBot is OpenAI's training-data crawler. OpenAI's own description: "GPTBot is used to make our generative AI foundation models more useful and safe." The bot collects publicly accessible content for training future versions of models behind ChatGPT and OpenAI's other products. The user-agent prefix is GPTBot/1.3. The IP range file lives at openai.com/gptbot.json. The robots.txt token is GPTBot.

The training-data signal is explicit. From OpenAI: "Disallowing GPTBot indicates a site's content should not be used in training generative AI foundation models." This is the unambiguous opt-out lever for OpenAI training.

OAI-SearchBot

OAI-SearchBot is OpenAI's retrieval crawler. From the same documentation: "OAI-SearchBot is used to surface websites in search results in ChatGPT's search features." The bot indexes pages so ChatGPT can cite them in real-time conversation answers. The user-agent prefix is OAI-SearchBot/1.3 (note the full Chrome 131 macOS prefix, the only OpenAI bot using browser-like emulation). The IP range file lives at openai.com/searchbot.json. The robots.txt token is OAI-SearchBot.

OpenAI documents the two bots as mutually independent. Allowing or blocking GPTBot has no effect on OAI-SearchBot's access, and vice versa. This is the architectural fact that the rest of the post depends on.

The asymmetric consequence: what happens when you block each one

The two bots have radically different costs of blocking. Get the asymmetry right and you control the trade-off precisely. Get it wrong and you pay a high price for the wrong decision.

Block GPTBot: low-cost opt-out

Blocking GPTBot opts your content out of future foundation-model training. That's it. Your existing ChatGPT Search citations are unaffected. Your eligibility for future ChatGPT Search citations is unaffected. If OpenAI's next model adds a separate retrieval crawler, the GPTBot block does not extend to it. The blast radius is narrow and predictable.

Block OAI-SearchBot: high-cost exit from ChatGPT visibility

Blocking OAI-SearchBot exits you from ChatGPT Search citations entirely. ChatGPT cannot cite a page it cannot crawl. Your content can still be in OpenAI's training data (if you didn't also block GPTBot), but ChatGPT's real-time citation pipeline is the surface that drives clicks, leads, and pipeline. Block this and you've removed yourself from the citation graph.

The most common mistake is conflating them. Teams that want to protect their content from "AI scraping" often block both, then wonder why their AI citations dropped. The protection they wanted came from blocking GPTBot. Blocking OAI-SearchBot was collateral damage with no upside.

GPTBot vs OAI-SearchBot side-by-side

Eight fields where the two bots differ. Read across each row to see why they need separate decisions.

Two patterns to notice. First, the user-agent strings differ structurally: GPTBot's prefix is minimal, OAI-SearchBot's prefix includes a full Chrome 131 macOS string suggesting fuller browser-like rendering capabilities. Second, the IP range files are different (gptbot.json vs searchbot.json), which means firewall allowlists by IP must be configured for each bot independently. They cannot share a rule.

The decision: which one should you block?

Two independent questions produce four combinations. Each combination is appropriate for a specific kind of site. Find your row in the matrix below.

The B2B default (block GPTBot, allow OAI-SearchBot) is the right starting point for most mid-market SaaS, professional services, and content-marketing sites. It opts you out of training while keeping you eligible for the citations that actually drive pipeline. The training-only opt-in quadrant is rare and almost never strategically correct. The strict-privacy quadrant fits paywalled publishers and regulated industries where every form of AI access needs to be restricted.

robots.txt configurations for 4 common scenarios

Each block below corresponds to one quadrant of the decision matrix. Copy the one that matches your stance, paste it into your /robots.txt file at the site root, and confirm it serves at https://yourdomain.com/robots.txt with HTTP 200.

Scenario 1: B2B default (recommended for most)

Block GPTBot, allow OAI-SearchBot. Eligible for ChatGPT Search citations, opted out of training.

# B2B default
# Block training crawler
User-agent: GPTBot
Disallow: /

# Allow retrieval crawler (for ChatGPT Search citations)
User-agent: OAI-SearchBot
Allow: /

# Allow user-initiated fetches
User-agent: ChatGPT-User
Allow: /

Scenario 2: Training-friendly (publishers, max reach)

Allow both. Maximum exposure across training and citation surfaces.

# Training-friendly
User-agent: GPTBot
Allow: /

User-agent: OAI-SearchBot
Allow: /

User-agent: ChatGPT-User
Allow: /

Scenario 3: Strict privacy (paywalled, regulated)

Block both. No training, no citations.

# Strict privacy
User-agent: GPTBot
Disallow: /

User-agent: OAI-SearchBot
Disallow: /

# Note: ChatGPT-User fires when a user explicitly fetches a URL.
# OpenAI notes robots.txt rules may not apply for this bot.

Scenario 4: Selective (public pages allowed, private blocked)

Block GPTBot universally. Allow OAI-SearchBot on public paths, block on members-only or sensitive paths.

# Selective: public allowed, private blocked
User-agent: GPTBot
Disallow: /

User-agent: OAI-SearchBot
Disallow: /members/
Disallow: /account/
Disallow: /api/
Disallow: /internal/
Allow: /

User-agent: ChatGPT-User
Allow: /

After publishing your robots.txt and llms.txt files, verify reachability at the WAF and CDN layer. Robots.txt is advisory at the origin only. The next two sections cover why that matters.

How to identify each crawler in your server logs

Once your robots.txt is live, the next move is checking your server access logs for actual bot visits. Six log signatures cover the OpenAI quartet, the major Anthropic equivalent, and the spoofer pattern to watch for.

If your logs show GPTBot visits but no OAI-SearchBot visits, the most likely cause is a Cloudflare or CDN edge rule blocking the retrieval crawler before it reaches your origin. The next section covers that failure mode in detail. For broader tracking that captures all 16+ AI crawlers automatically, see the bot activity tracking guide.

The Cloudflare default-block angle (the silent killer)

Since July 2025, Cloudflare's account abuse protection has blocked AI bots by default for many customers. This is the single most common reason a perfectly correct robots.txt fails to produce ChatGPT Search citations. Your origin says allow OAI-SearchBot. Your Cloudflare edge says block all unknown bots. Cloudflare wins, the request never reaches your origin, and your ChatGPT visibility quietly dies.

The same problem repeats across Akamai Bot Manager, Fastly, Imperva, and DataDome. Each treats unknown bots as suspicious by default and may issue challenges, rate-limits, or outright blocks. The fix is the same at every edge layer: add an explicit allowlist rule for the retrieval crawlers you want through.

How to fix it in Cloudflare

Open the Cloudflare dashboard, navigate to Security → Events, filter the user-agent column for OAI-SearchBot and GPTBot. If you see blocked in the action column, the default rule is the cause. Add a WAF custom rule:

# Allow OpenAI's retrieval crawler for ChatGPT Search citations
(http.user_agent contains "OAI-SearchBot/1.3")

Action: Skip
Skip:
  - All remaining custom rules
  - Bot Fight Mode
  - Super Bot Fight Mode

# Optional: log matched requests for verification
Log: Yes

If you also want training inclusion (Scenario 2 from the matrix above), repeat the rule with GPTBot/1.3 as the match pattern. The principle is the same: edge-layer rules must mirror your robots.txt intent or robots.txt is silently overruled.

Common mistakes when blocking OpenAI bots

Four anti-patterns surface repeatedly when teams configure crawler access for the first time. Each one breaks the binary decision in a different way.

Mistake 1: Blocking OAI-SearchBot thinking it's a training bot

OAI-SearchBot is the retrieval bot, not the training bot. Its job is to fetch pages so ChatGPT can cite them. Block this and you exit citations. If you wanted to block training data collection, you wanted to block GPTBot. The names sound similar; the consequences are opposite.

Mistake 2: Blocking GPTBot to "protect content" but leaving CCBot allowed

CCBot is Common Crawl's crawler. Common Crawl is one of the largest sources of training data for most foundation models, not just OpenAI's. Blocking GPTBot while leaving CCBot allowed is a half-measure. If your goal is to keep your content out of LLM training pipelines, block CCBot, Google-Extended, ClaudeBot, and Meta-ExternalAgent alongside GPTBot.

Mistake 3: Allowing OAI-SearchBot in robots.txt but not at the WAF layer

Robots.txt is advisory at the origin. Cloudflare, Akamai, Fastly, Imperva, and DataDome can still block bots before they reach the origin. If your robots.txt allows OAI-SearchBot but your edge layer doesn't, the edge layer wins and your content is invisible. Mirror the allowlist at every edge that fronts your origin.

Mistake 4: Treating ChatGPT-User the same as the other two

ChatGPT-User fires only when a ChatGPT user explicitly asks the model to fetch a specific URL. OpenAI's documentation notes that robots.txt rules may not apply for this bot, because the user initiated the request directly. Blocking ChatGPT-User as if it were an automated crawler breaks user-initiated browsing inside ChatGPT and accomplishes nothing about training or general citations.

The AI Visibility Lift: same discipline across every AI vendor

OpenAI's two-bot split (training vs retrieval) is not unique. Every major AI vendor separates training crawlers from retrieval crawlers from user-initiated crawlers. Once you understand the OpenAI decision, the same framework extends to Anthropic, Google, Perplexity, and Microsoft.

• OpenAI: GPTBot (training) and OAI-SearchBot (retrieval), plus ChatGPT-User (user-initiated) and OAI-AdsBot (ad validation, see the 4-bot taxonomy post)
• Anthropic: ClaudeBot (training) and Claude-SearchBot (retrieval), plus Claude-User (user-initiated)
• Google: Google-Extended (training opt-out) and GoogleOther (used for AI Overviews retrieval), plus Googlebot (traditional search)
• Microsoft: Bingbot powers Copilot retrieval through Bing's index; no separate training bot is publicly documented
• Perplexity: PerplexityBot (retrieval) and Perplexity-User (user-initiated)

The default discipline is the same at each vendor. Block training crawlers if you want to opt out of model training. Allow retrieval crawlers if you want to be cited in AI search answers. Allow user-initiated crawlers because the user explicitly asked. For the full allow-or-block reference across all 17 major AI crawlers, see our 2026 AI crawler guide. The mid-market B2B teams that apply this discipline consistently across vendors compound an AI Visibility Lift advantage that takes competitors a year to close once they realize the pattern exists.

Frequently Asked Questions

Related Reading

• What Is OAI-AdsBot? (OpenAI's ChatGPT Ads Crawler)
• How to Track AI Bot Activity on Your Website
• What Is llms.txt and Does Your Website Need One?
• How to Get Cited by Microsoft Copilot: A 2026 Bing-Era SEO Playbook
• How to Get Cited by Google AI Overviews: A 2026 Schema + Topical Authority Playbook