Robots.txt Generator

Customize crawler access rules and block AI scrapers to preserve crawl budget and secure your sensitive directory paths.

Toggle AI Crawler Access

Allow all standard search bots (User-agent: *)Block GPTBot (OpenAI)Block PerplexityBot (Perplexity AI)Block ClaudeBot (Anthropic)Block Google-Extended (AI Training)Block Bingbot (Microsoft Bing)

Disallowed Paths (Comma Separated)

Sitemap URL

Generated robots.txt File Output