Robots.txt Generator — Build Rules for Search & AI Crawlers
Robots.txt Generator
Build a robots.txt file with rules for crawlers. Add user-agent directives, allow/disallow paths, and sitemap references.
#What this tool does
Build a robots.txt file with rules for search engines and AI crawlers. Add user-agent directives, allow/disallow paths, set crawl delays, and include sitemap references. The output updates as you edit.
#Managing AI crawlers
The rise of AI crawlers makes robots.txt more relevant than ever. Allow crawlers that cite your content in AI answers while blocking training-only scrapers:
# AI assistants that cite sources — allow
User-agent: GPTBot
Allow: /
User-agent: ClaudeBot
Allow: /
User-agent: PerplexityBot
Allow: /
# Training-only scrapers — block
User-agent: CCBot
Disallow: /
User-agent: Bytespider
Disallow: /Known AI crawler user-agents (as of February 2026): GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, PerplexityBot, GoogleOther, Bytespider, CCBot, Amazonbot, FacebookBot, Applebot-Extended.
#Important limitations
robots.txt is a request, not access control. Well-behaved crawlers respect it; malicious scrapers ignore it. For actual protection, use authentication or rate limiting.
Search engines may still index a blocked URL (without content). To prevent indexing entirely, use a noindex meta tag or X-Robots-Tag header.
#Open source
This tool is powered by @azin-tech/mini-tools, an open-source developer toolkit. View source on GitHub.
Auto-deploy into your own cloud
Push code, AZIN handles the rest. Auto-detected builds, your cloud account, no vendor lock-in.