Should You Stop AI Training on Your Website?

You can ask the big AI companies to stop training on your website, and for most small businesses it costs little either way. What you want to avoid is blocking the wrong thing. AI tools visit websites for three different jobs, and only one of them is training. Block the other two and you can drop out of the answers people get when they ask an assistant who to call.

Should I stop AI tools training on my website?

Only if your content is the thing you sell. A photographer, an illustrator, a writer or a maker whose product descriptions took weeks to get right has a real reason to object to their work being learned from for free. A plumber or a salon whose site says what they do, where, and how to book has much less to protect, because those facts are on their Google listing anyway.

Either way, you are not choosing between AI and no AI. You are choosing which of the jobs below you're happy with.

What are AI crawlers actually doing on my site?

They are doing one of three jobs, and the main providers name a separate crawler for each. A crawler is a program that reads web pages automatically. Your website's robots.txt file, a short text file at yourbusiness.co.uk/robots.txt, is where you tell each one by name whether it is welcome.

Anthropic's own help page on how site owners can block its crawlers lists ClaudeBot for training, Claude-SearchBot for search and Claude-User for fetching a page someone asked about, and says its bots honour robots.txt. OpenAI's crawler documentation separates GPTBot, for training, from OAI-SearchBot, for ChatGPT's search results, and says each setting is independent, so you can block one and allow the other.

Does blocking Google's AI training hurt my Google ranking?

No. Google's crawler documentation says Google-Extended "does not impact a site's inclusion in Google Search nor is it used as a ranking signal". Google-Extended is not a separate visitor at all. It is a name you use in robots.txt to say whether Google may use what it already crawls for training future Gemini models and for grounding, which means Gemini apps drawing on your pages when they answer.

Blocking it does cost you one thing: your pages stop being eligible for those Gemini app answers. AI Overviews in Google Search are a different matter, handled by the ordinary search controls rather than Google-Extended.

How do I block AI training and nothing else?

Add a few lines to your robots.txt file that name the training crawlers and leave everything else alone. If you want to opt out of training by the three providers above, these are the lines:

On a website builder, search its help centre for "robots.txt" before you touch anything, because builders handle the file differently and some write it for you. If someone else looks after your site, send them those lines and ask them to check nothing else in the file changed.

It won't stop everyone. Robots.txt is a request, and while the providers named here say they follow it, the file can't force a crawler that chooses not to. There are plenty of smaller AI companies too, so three names is a start rather than a wall.

Is there one switch coming for every AI company?

One has been announced, but it is not available yet. On 30 October 2026 Google announced plans to support a shared AI training control in robots.txt, built on standards work with other providers, and said it aims to support it "along with other AI providers in 2027". The same post says existing company-specific controls will keep working alongside it.

Nothing needs doing now because of it. If you block the named training crawlers today, those lines keep working, and when the shared control arrives it will be one more line you can add.

What should I do next?

If you sell your words or pictures, add the training lines and leave the search crawlers alone. If you run a service business, the bigger risk is the opposite one: an old robots.txt that blocks the search crawlers by accident. Our guide to whether ChatGPT knows your business shows how to check that, and more on getting the basics right is in your first website.

At ByRender we build your whole website for free, you look round it, and you only pay if you want to keep it, with setup from £495 and then from £69 a month. If you'd like a look, the contact page is the place to start.

Frequently asked questions

Does blocking AI training take my site out of AI Overviews?

No. Google says Google-Extended does not affect inclusion in Google Search, and its page on AI features in Search points to the ordinary search controls, such as nosnippet and noindex, for limiting what appears in AI Overviews. Those controls limit you in normal search results too, so they are rarely worth using on a small business site.

Will blocking now remove what AI models have already learned?

Do not count on it. Anthropic describes blocking ClaudeBot as a signal that your future material should be left out of training. Nothing in robots.txt reaches back into a model that has already been trained.

Can I ask my web designer to do this?

Yes, and it is a few minutes' work for them. Tell them which jobs you want to block, using the table in this guide, and ask them to confirm the search crawlers are still allowed once it is done.