By Scott Sumner | AI Search

Editorial illustration of a shop owner holding separate search and learning permission tickets, illustrating the distinction between AI search access and model training.

Blocking AI training does not always mean blocking AI search. Before you approve a blanket bot restriction, separate how your content gets reused from how potential customers discover it.

Answer: You can block OpenAI’s GPTBot for training while allowing OAI-SearchBot for ChatGPT search. These are independent controls. Other providers use different rules, and allowing search access does not guarantee a citation or recommendation.

The useful question is not simply whether you are comfortable with AI. It is which uses of your public website you want to permit, and which visitors a proposed setting will actually turn away.

Jump ahead

Separate discovery from training

Imagine a consultancy with a public guide that explains a difficult buying decision. The owner wants prospective clients to find that guide. They do not want its contents used to train another company’s foundation model.

Those preferences are not inherently contradictory. They describe two different uses of the same material.

Training helps shape a model. Search retrieves information to help answer a query. A third activity happens when a person asks an assistant to open a specific page. These activities can involve different agents and different controls.

My starting point is to write down the intended outcome before changing the website. “Keep our public expertise discoverable, but decline this provider’s training crawl” is something a technical team can investigate. “Block AI” leaves too much to interpretation.

This is a practical part of AI Findability: making a deliberate choice about how systems can reach your public business information. It is not a promise that permission creates visibility.

What OpenAI’s controls do

OpenAI’s crawler documentation, checked September 19, 2026, explicitly says its search and training settings are independent. It gives the example of allowing OAI-SearchBot while disallowing GPTBot.

  • OAI-SearchBot is for search. It helps surface websites in ChatGPT’s search features. OpenAI says sites that opt out will not be shown in ChatGPT search answers, although they can still appear as navigational links.
  • GPTBot is for potential training use. Disallowing it indicates that a site’s content should not be used to train OpenAI’s generative AI foundation models.
  • ChatGPT-User serves certain user-initiated actions. It is not OpenAI’s automatic search crawler. OpenAI says robots.txt rules may not apply because these visits are initiated by a user. It is not the control for search inclusion.

For a site that has chosen to allow search crawling of its public content and decline GPTBot, the relevant groups can look like this:

User-agent: OAI-SearchBot
Allow: /

User-agent: GPTBot
Disallow: /

This is an illustrative configuration, not a replacement robots.txt file. Have your developer reconcile it with existing path restrictions and bot-specific rules. A broad allow rule should not accidentally undo crawl restrictions you still want for a particular agent.

OpenAI says search systems can take approximately 24 hours to adjust to a robots.txt update. That is not an indexing deadline, and certainly not a promise of a recommendation within a day.

It also recommends allowing requests from its published search-bot IP ranges. A permission in robots.txt will not help if your firewall rejects the request before the crawler reaches the page.

Why Google’s controls differ

Do not copy OpenAI’s distinction onto every other provider. The names and product boundaries are different.

In Google’s common-crawler documentation, Google-Extended is a product-control token, not a separate HTTP crawler identity. Google says it controls use for training future Gemini models and for grounding in specified Gemini and Vertex AI products. Grounding means providing retrieved information to a model when it answers.

That second part matters. Describing Google-Extended as “only a training switch” would leave out a documented use.

Google also states that Google-Extended does not affect inclusion in Google Search and is not a Google Search ranking signal. Googlebot is a different control and affects Google Search, including its search features.

So the business discussion should name the product. Google Search and Gemini are not interchangeable labels for one setting. Ask which surface you want to remain accessible before agreeing to a change.

What to check before changing a setting

I would ask for a short written review, not a screenshot of an “AI protection” toggle. The review should answer four questions.

1. Which uses are we allowing or declining?

List search discovery, model training, and user-requested page visits separately. Match each decision to the provider’s current documentation. If a control covers more than one use, make that tradeoff explicit.

2. What does the website actually enforce?

Check robots.txt alongside the CDN, firewall, hosting controls, and relevant security plugins. A friendly page in your own browser does not prove a permitted crawler can retrieve it. Conversely, a declared bot name alone is not proof of identity. Use the provider’s published verification information rather than trusting a user-agent label.

A useful handoff records the exact rule changed, the affected agent or product, a representative public URL, the test result, and how to restore the previous setting. Keep the original configuration before editing.

3. Are we confusing public-content preferences with security?

Google’s robots.txt guide is clear: the file cannot force crawlers to obey. It also is not a dependable way to keep a URL out of search results.

Private customer files belong behind authentication and appropriate access controls. Putting a path in robots.txt is not a lock. And changing a crawl preference is not a way to remove information already learned by an existing model.

4. What evidence will show whether the change worked?

Verify that the published rules match the decision and that intended requests are not being rejected. Recheck after the provider’s stated adjustment period where one exists. Keep search visibility and business outcomes as separate follow-up measures.

A successful crawler request proves access, not endorsement. A citation proves a particular appearance, not reliable inclusion in every buyer’s answer. Keep those claims separate when reviewing results with your marketing team.

Make a deliberate business decision

There are sensible reasons to restrict content reuse. There are also sensible reasons for a service business to keep its public expertise available to search systems. You do not need to pretend either concern is foolish.

The mistake is approving a broad restriction without checking whether it matches the actual goal.

My recommendation: start with one important public service page. Ask your web team to document which search and training controls apply to it, including any firewall restrictions. Then decide what you want changed. Do not begin by disabling every protection, or by assuming every bot serves the same purpose.

If you need the wider picture, a website findability audit can examine accessibility alongside clarity, proof, and the route to an inquiry. The Findability OS Audit is the strategic diagnostic for prioritizing those issues. Get in touch if you want help deciding what your website should permit, rather than just flipping a switch.

Sources

This is an evergreen operating guide, not a new platform announcement. Documentation was checked September 19, 2026. Provider controls can change.