AI Crawler Manager
Decide which AI assistants can crawl your site — and generate the exactrobots.txt rules to enforce it. Allow the bots that cite you in answers; block the ones that only scrape for training, if you choose.
Choose your bots
Your robots.txt
robots.txt. For hard enforcement, block at your firewall / Cloudflare WAF as well.How to use it
Where robots.txt goes
Save the generated rules as a plain-text file named robots.txt at the root of your domain, e.g. https://yoursite.com/robots.txt. It takes effect as soon as it's live.
Should I block AI crawlers or not?
For most sites, allow search and citation bots (OAI-SearchBot, PerplexityBot, Claude's search bot) — they're how you get cited in AI answers, which is free visibility. The choice to make is on training-only bots (GPTBot, ClaudeBot, CCBot): block them if you don't want your content in model training. Note that research from late 2025 found sites blocking crawlers lost traffic without reliably reducing citations — so block deliberately, not by reflex.
What's "Google-Extended"?
It controls whether Google may use your content for AI/Gemini training — separately from normal search. Blocking Google-Extended does not affect your Google Search ranking, so it's a safe opt-out of AI training if you want one.