Robots.txt Generator
Create production-ready robots.txt files with easy controls for blocking AI scrapers (GPTBot, Claude, Bard). Manage crawl budget, protect sensitive content, and specify sitemaps.
Robots.txt Generator
Paths bots should NOT visit.
Exceptions to Disallow rules.
Download this content and save it as
robots.txt in your root directory.What is Robots.txt?
A robots.txt file is a text file placed in your website's root directory that tells web crawlers and bots which pages they can and cannot access. It is a critical component of SEO that helps you manage crawl budget, protect sensitive information, and control how your site is indexed.
Key Directives
- User-agent: Specifies which crawler the rules apply to (e.g., Googlebot, GPTBot, *)
- Disallow: Paths that crawlers should NOT visit
- Allow: Exceptions to disallow rules for specific paths
- Crawl-delay: Time in seconds the crawler should wait between requests
- Sitemap: URL of your XML sitemap to help search engines discover content
AI Scraper Protection
With the rise of AI training data collection, protecting your content has become increasingly important. Our tool includes options to block known AI scrapers:
- GPTBot: Used by OpenAI to train ChatGPT and other models
- CCBot: Common Crawl's bot used for internet-wide crawling
- Google-Extended: Google's bot for training generative AI models
- Anthropic-ai: Anthropic's bot for Claude model training
Best Practices
- Set appropriate crawl delays to prevent server overload
- Block admin, staging, and temporary directories
- Always include your sitemap URL for proper indexing
- Use specific user-agents for targeted rules
- Test your robots.txt file with Google Search Console
- Review and update regularly as your site structure changes