With robots.txt you can, for example, shield the admin area of your site from crawlers. A wrong setting can accidentally block your entire site from Google, so handle it with care.
What does this mean in practice?
Robots.txt is a simple text file in the root folder of your website (yoursite.com/robots.txt). It tells crawlers from search engines and AI services which parts of your site they may visit and which they should skip. It usually also contains the address of your sitemap.
Why does this matter for your business?
A properly configured robots.txt guides crawlers to your important content and, for example, shields your admin area. But one wrong rule ('Disallow: /') can make your entire website invisible in Google.
Examples and best practices
- Only block technical areas, such as /wp-admin/.
- Include the address of your XML sitemap.
- Don't block CSS and JavaScript files needed to display the page.
- Make a deliberate choice about giving AI crawlers access.
How we do it at Ikoon
A properly configured robots.txt, with a sitemap and access for search engines and AI crawlers, is part of the SEO and AI-proof foundation of every Ikoon website. We keep test versions out of Google, so only your real site is found.