Robots.txt Generator for WordPress & Custom Sites

Generator Features

• AI Bot Protection: One-click blocking for ChatGPT (GPTBot) and Common Crawl (CCBot).
• Sitemap Integration: Automatically includes your XML sitemap for better indexing.
• Crawl Delay Support: Prevent server overload from aggressive crawlers.
• syntax Validation: Generates RFC-compliant robots.txt syntax ready for production.

Managed your crawlers? Now find high-volume long-tail keywords using our Keyword Mixer.

Why you must block AI bots (GPTBot) in 2026

AI crawlers like GPTBot and CCBot scrape your content to train their models, often without providing any traffic back to your site. Blocking them preserves your server resources and protects your intellectual property from being used in Large Language Models (LLMs) without accreditation.

Explanation of User-agent, Disallow, and Allow directives

• User-agent: *: Applies to all robots.
• Disallow: /admin/: Prevents crawlers from accessing your admin area.
• Allow: /admin/ajax.php: Exception to the rule, allowing access to specific files.

Where to upload your robots.txt file

This file must live at the root of your domain (e.g., https://yourdomain.com/robots.txt). If you use WordPress, you can upload it via FTP up to the `public_html` folder or use an SEO plugin like RankMath or Yoast to edit it virtually.