Optimizing a website for AI search without contributing to model training.
How I make my websites discoverable, understandable and easy for AI search engines to cite, while stopping my content from becoming part of large language models.Using search models like Google’s AI Summary has become extremely mainstream, over 60% of searches are now performed without ever clicking a website, which means websites that don’t optimize for AI, are likely left behind compared to competitors that do allow LLM’s to read their content. While some of us might dislike AI, it’s hard to give up on an ever increasing group of people that might find your content through such search features.
It is important to note that allowing LLM’s to read your content, does not have to mean that they also use it for training. In this post I’ll explain how I’ve optimized my website so that it does not allow AI models to train on my content, while I’ll still show up in AI search queries.
Being found
All major AI models fetch and train information using your content automatically, unless specified otherwise. Fortunately, at some point the robots.txt standard was introduced, which specifies what bots will and won’t have access to your website. While this is not respected by all, generally speaking, a lot of the AI labs will respect these permissions.
ChatGPT, makes use of OAI-SearchBot, which automatically assumes it has permission. By default, this opts into model training, which means all content on your website can be used to improve their products. To avoid this, block GPTBot in the robots.txt file, which will allow you to benefit from search without providing all your content for data training. For Claude, search access is given through Claude-User and Claude-SearchBot, while ClaudeBot should be blocked if you want to avoid training using your content.
Google requires GoogleBot access to provide users with content, while Google-Extended allows users to manipulate this data, however, Google-Extended allows Google to train models on this data. A fourth crawler to target is Perplexity, which does not train on user data and requires PerplexityBot and Perplexity‑User access, once again working by default.
Therefore, if you intend on being shown to AI but don’t want models to train on your content, blocking GPTBot, ClaudeBot and Google-Extended should get you well on your way to avoid training models by common providers.
As a more general advice, LLM’s are more biased towards pages including external links. I personally link my sources, but it can also be accompanied by other pages
Also read
Sustainable AI for data quality at Triodos Bank
Designing a value-driven data quality platform with AI assistants for a bank that supports 750,000+ customers in Europe, improving data steward workflows and legal compliance.
Checklist
Below is a simple checklist you can use to ensure you are found in this “new way of searching”, while not giving away all your information for models to be trained on freely. Alternatively, you could take the info from my own robots.txt and copy them over, this ensures that AI training is blocked but you’re still findable in search queries.