oai searchbot robots.txt guide for ChatGPT visibility

What does oai searchbot robots.txt actually control?

OAI-SearchBot controls search inclusion, not training consent. OpenAI describes OAI-SearchBot as the crawler used to surface websites in ChatGPT's search features. The same documentation says sites opted out of OAI-SearchBot will not be shown in ChatGPT search answers, though they can still appear as navigational links. That makes the robots rule a practical gate for summaries, snippets, citations, and normal source-link visibility in ChatGPT search experiences.

The important word is "search." If a publisher blocks every OpenAI user agent because an old policy wanted to avoid training collection, that publisher may also block the crawler that could have made the page visible in ChatGPT search. The outcome looks like an AI-search content problem, but the cause is often a crawler-policy problem. Before rewriting an article, confirm the page is eligible to be discovered.

That last row is where many real failures happen. A site may not mention `OAI-SearchBot` at all, but a broad `User-agent: *` disallow, CDN bot rule, security plugin, or legacy AI-blocking snippet can still prevent access. The safest audit checks the matched rule, not just whether the exact user-agent string appears in the file.

How is OAI-SearchBot different from GPTBot and ChatGPT-User?

The crawler names are easy to group together and dangerous to treat as one policy. OpenAI documents each setting as independent. OAI-SearchBot is for search. GPTBot is associated with crawling content that may be used to train generative AI foundation models. ChatGPT-User is used for certain user actions, such as a ChatGPT or Custom GPT request that visits a web page. OAI-AdsBot is a separate ads landing-page validation crawler. Four names, four jobs, four risk profiles.

OAI-SearchBot is the ChatGPT search visibility crawler

If you care about being discovered through ChatGPT search, this is the crawler to review first. OpenAI recommends allowing OAI-SearchBot in robots.txt and allowing requests from its published IP ranges to help ensure the site can appear in search results. That does not guarantee ranking or citation, but blocking access is a clear self-inflicted eligibility problem.

GPTBot is the training policy decision

GPTBot should be handled by a different stakeholder conversation. Legal, editorial, and executive teams may decide that future site content should not be used for potential training. If that is the policy, disallow GPTBot directly. Do not turn that policy into a blanket OpenAI block unless the business also wants to opt out of ChatGPT search visibility.

ChatGPT-User is user-triggered, not the search inclusion lever

OpenAI says ChatGPT-User is not used to determine whether content may appear in Search and tells publishers to use OAI-SearchBot for Search opt-outs and automatic crawl. That distinction is central for troubleshooting. A log line from ChatGPT-User can show that a user-triggered action happened, but allowing ChatGPT-User alone is not the same as making a page eligible for ChatGPT search answers.

Should you allow OAI-SearchBot but block GPTBot?

For many public sites, yes. The balanced policy is to allow OAI-SearchBot for ChatGPT search while disallowing GPTBot if the organization wants a training opt-out. This is the same split that appears in other AI-search access decisions: search visibility and model training are related conversations, but they are not the same control. The clean policy names the crawler that matches each intent.

The strongest candidates for this split are SaaS documentation hubs, ecommerce catalogs, publisher articles, help centers, product comparisons, and citation-ready resources. If a page is already built for public discovery, internal links, structured data, and conversion paths, blocking OAI-SearchBot can undermine the work described in our ChatGPT Shopping SEO guide and broader answer engine optimization checklist.

Blocking GPTBot answers a training preference. Blocking OAI-SearchBot answers a ChatGPT search visibility preference. Do not let one decision silently make the other.

There are legitimate reasons to block both. Private communities, sensitive archives, staging hosts, thin internal tools, and pages that should not be summarized publicly may choose a strict block. The key is deliberate scope. A full OpenAI block on a public product guide is very different from a full block on a staging subdomain or account dashboard.

What robots.txt patterns work for OpenAI crawler policy?

The most maintainable robots.txt file uses explicit user-agent sections and avoids broad rules that hide intent. Google's robots.txt introduction remains a useful baseline: robots.txt controls crawler access to URLs, not whether a URL can be indexed from other discovery signals. For OpenAI search visibility, use robots.txt for crawl permission and use noindex when the goal is to prevent search surfacing from other discovered links.

Pattern 1: public visibility with training opt-out

This is the default pattern for many marketing and content teams: keep ChatGPT search access available, keep live user actions available, and block GPTBot for potential training collection.

User-agent: OAI-SearchBot
Allow: /

User-agent: GPTBot
Disallow: /

User-agent: ChatGPT-User
Allow: /

Pattern 2: maximum public OpenAI access

Some publishers are comfortable allowing search, training, and user-triggered fetches. If that is the policy, make it explicit so future security plugins or wildcard blocks do not change the practical result by accident.

User-agent: OAI-SearchBot
Allow: /

User-agent: GPTBot
Allow: /

User-agent: ChatGPT-User
Allow: /

Pattern 3: strict OpenAI block

Use this only when the organization accepts the visibility tradeoff. If you block OAI-SearchBot, the page should not be a priority for ChatGPT search citations, product discovery, or AI referral growth.

User-agent: OAI-SearchBot
Disallow: /

User-agent: GPTBot
Disallow: /

User-agent: ChatGPT-User
Disallow: /

How do noindex, firewalls, and IP ranges change the decision?

Robots.txt is necessary but not sufficient. OpenAI's current publisher FAQ says public websites can appear in ChatGPT search, but it also notes that if OpenAI gets the URL of a disallowed page from a third-party search provider or by crawling other pages, it may surface only the link and page title in ChatGPT Atlas when relevance signals exist. If you do not want that, the FAQ points to noindex, with an important caveat: the crawler must be allowed to crawl the relevant page to read the meta tag.

That creates a two-step policy. Robots.txt is the door. Noindex is the search-surfacing instruction. If you block crawl before the crawler can read noindex, you may not communicate the richer indexing preference you intended. This is the same concept behind our guides to meta robots tags and X-Robots-Tag headers: crawl permission and index permission are different layers.

IP ranges belong in the firewall review

OpenAI publishes IP range JSON files for OAI-SearchBot, GPTBot, and ChatGPT-User. Use those files as verification inputs for CDN, firewall, and bot-management rules rather than relying on user-agent strings alone. A correct robots.txt file does not help if the WAF blocks OAI-SearchBot before it can fetch the file or the page. Conversely, a permissive firewall does not help if the matched robots rule still disallows the crawler.

How do you audit OAI-SearchBot access after a release?

Audit from upstream to downstream. Start with the production robots.txt file on the exact protocol, host, and subdomain that serves the page. A staging policy on `staging.example.com` tells you nothing about `www.example.com`, and a file generated by a Next.js route can differ from the repository text after middleware, hosting, or environment variables are applied.

Step 1: fetch the live file and find the matched rule

Do not only search for `OAI-SearchBot`. Identify the rule group that actually applies. A broad `User-agent: *` block can still match if there is no more specific group. Old WordPress plugins, CDN templates, and copied AI-crawler lists often introduce this problem because they were written before OpenAI split search, training, ads, and user actions into separate agents.

Step 2: verify page fetches through the security stack

Use server logs, CDN logs, or bot-management dashboards to confirm OAI-SearchBot reaches the priority pages. If logs show only robots.txt checks and no page fetches, investigate rate limits, cache rules, JavaScript rendering barriers, and page-level blocks. The same troubleshooting sequence appears in our log file analysis workflow for classic SEO, but the user-agent list has changed.

Step 3: wait for adjustment before reading traffic

OpenAI says search systems can take about 24 hours to adjust after a robots.txt update. Use that as a minimum observation window, not as a promise of instant citations. Then compare logs, ChatGPT referral sessions, and page engagement. If your site gets `utm_source=chatgpt.com` referrals, connect that analysis to the same reporting rules in the ChatGPT GA4 attribution guide.

Which mistakes make sites invisible in ChatGPT search?

The first mistake is inherited policy. A site copied an AI crawler block in 2024, never revisited it, and now treats ChatGPT search as a content-quality problem. The second is plugin drift. A security or robots.txt plugin adds a broad AI-blocking section and the SEO team notices only after referral traffic falls or a manual ChatGPT search fails to cite priority pages. The third is treating OAI-SearchBot and GPTBot as interchangeable names for one OpenAI crawler.

The fourth mistake is measuring before eligibility. If OAI-SearchBot cannot fetch a product guide, an analytics dashboard will not fix the absence of ChatGPT search referrals. Start with access, then page quality, then attribution. That same order keeps your AI-search workflow aligned with our technical SEO checklist for AI-ready sites and llms.txt SEO guide.

Use a change log for every crawler-policy edit

Crawler rules should be annotated like redirects, canonicals, and index controls. Record the date, exact rule, affected host, owner, and intended outcome. Without a change log, future teams will not know whether a block was a deliberate risk decision, a temporary incident response, or a forgotten experiment. With a change log, ChatGPT visibility troubleshooting becomes much faster.

Keep crawler policy connected to page strategy

Allowing OAI-SearchBot is not a ranking tactic by itself. It is an eligibility prerequisite. The page still needs clear answers, crawlable HTML, helpful headings, verifiable source links, descriptive images, and internal context. Robots.txt gets the page into the possible set. Content quality decides whether that access was worth anything.

FAQ: oai searchbot robots.txt