AI Robots.txt Generator

Configure your robots.txt to control AI crawlers, LLM search engines, and training bots.

Quick Presets

Paths to Disallow (Comma separated)

Leave empty to disallow the entire website (/) for blocked bots.

Google & OpenAI

Googlebot

User-agent: Googlebot

OpenAI SearchBot

User-agent: OAI-SearchBot

ChatGPT User Browsing

User-agent: ChatGPT-User

GPTBot (Training)

Training

User-agent: GPTBot

Anthropic

Claude SearchBot

User-agent: claude-searchbot

Claude User Browsing

User-agent: claude-user

AI Search & Tech

Perplexity AI

User-agent: PerplexityBot

Bingbot (Copilot)

User-agent: Bingbot

Applebot (Apple Intelligence)

User-agent: Applebot

Meta AI Agent

User-agent: Meta-ExternalAgent

xAI GrokBot

User-agent: GrokBot

Sitemap & Advanced Rules

Include Sitemap Directive

Crawl-delay (seconds)

Host Directive

General Crawlers (User-agent: *)

Default fallback rule for all other search engines

Want full RAG & Schema analysis for your site?

Try Full AEO Audit Pro →

robots.txt Output

AI Readiness Score:11/11 Crawlers Ready

# AI Crawler Directives generated by AEO Audit Pro

# --- Google & OpenAI ---
User-agent: Googlebot
Disallow:
User-agent: OAI-SearchBot
Disallow:
User-agent: ChatGPT-User
Disallow:
User-agent: GPTBot
Disallow:

# --- Anthropic ---
User-agent: claude-searchbot
Disallow:
User-agent: claude-user
Disallow:

# --- AI Search & Tech ---
User-agent: PerplexityBot
Disallow:
User-agent: Bingbot
Disallow:
User-agent: Applebot
Disallow:
User-agent: Meta-ExternalAgent
Disallow:
User-agent: GrokBot
Disallow:

# --- General Search Crawlers ---
User-agent: *
Allow: /

Sitemap: https://example.com/sitemap.xml

💡 How to use: Download or copy this content and save it as robots.txt in the root directory of your domain (e.g., yoursite.com/robots.txt).