As a website owner, you decide whether to give these bots access. If you want to be visible in AI answers, it's usually wise to allow them.
What does this mean in practice?
AI crawlers are automated programs from companies that build AI models and AI search engines. They visit websites. Well-known examples are GPTBot and OAI-SearchBot (OpenAI), ClaudeBot (Anthropic), Google-Extended and PerplexityBot. Some collect data to train models, others fetch content for up-to-date answers.
Why does this matter for your business?
If you block these crawlers in your robots.txt, your website may not appear in AI answers. For local businesses that want to be recommended by ChatGPT or Perplexity, it usually pays to allow them.
Examples and best practices
- Check whether your robots.txt blocks AI crawlers.
- Distinguish between crawlers for training and crawlers for search, as you can decide per type.
- Keep your website fast, so crawlers can easily fetch your pages.
- Publish content you want AI to cite.
How we do it at Ikoon
At Ikoon, an AI-proof structure is standard. AI search crawlers can access your content, so your business can appear in answers from ChatGPT, Perplexity and Google AI. Ikoon's fast hosting also ensures crawlers can fetch your pages quickly.