Free tool

Decide which AI bots may read your site

Pick which AI crawlers to allow or block, for training, AI search and fetches a person asks for, and get a robots.txt you can copy or download. Add the rules to your existing file without breaking it.

The usual choice

Block model training, keep AI search. Your pages can still be cited and linked in ChatGPT, Claude and Perplexity answers without being used to train the next model.

More free tools.

1 · Start from

2 · Choose crawler by crawler

Model training

Collect pages to train AI models. Blocking these does not remove you from any search results.

  • GPTBot · OpenAI

    Trains OpenAI's GPT models.

  • ClaudeBot · Anthropic

    Trains Anthropic's Claude models.

  • Google-Extended · Google

    Not a separate crawler: tells Google whether Gemini may train on or ground answers in your pages. Google Search and AI Overviews are unaffected.

  • Applebot-Extended · Apple

    Tells Apple whether pages Applebot crawls may train Apple Intelligence. Siri and Spotlight search are unaffected.

  • Meta-ExternalAgent · Meta

    Trains Meta's Llama models and AI features.

  • CCBot · Common Crawl

    Open web archive that many AI companies train on.

  • Bytespider · ByteDance

    Trains ByteDance models (Doubao). Widely reported to ignore robots.txt.

  • Amazonbot · Amazon

    Used by Amazon for Alexa answers and AI models.

AI search and answers

Index pages so AI assistants can cite and link to you. Blocking these can drop you from those answers.

  • OAI-SearchBot · OpenAI

    Indexes pages for ChatGPT search results and links.

  • Claude-SearchBot · Anthropic

    Indexes pages so Claude can cite them in search answers.

  • PerplexityBot · Perplexity

    Indexes pages for Perplexity answers and links.

  • DuckAssistBot · DuckDuckGo

    Fetches pages for DuckDuckGo's AI-assisted answers.

Fetches a person asked for

Open one page when someone pastes your link or asks about it. Some vendors say these may not follow robots.txt.

  • ChatGPT-User · OpenAI

    Opens a page when a ChatGPT user asks it to.

  • Claude-User · Anthropic

    Opens a page when a Claude user asks it to.

  • Perplexity-User · Perplexity

    Opens a page when a Perplexity user asks a question.

  • Meta-ExternalFetcher · Meta

    Opens a link a person shares with Meta AI.

  • MistralAI-User · Mistral

    Opens a page when a Le Chat user asks it to.

3 · Your site

Block the chosen crawlers from

Your robots.txt

8 of 17 AI crawlers blocked
# robots.txt

User-agent: *
Disallow:

# --- AI crawlers (built with dueclix.com/tools/ai-robots-txt/) ---
# Blocked from the whole site:
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
User-agent: Applebot-Extended
User-agent: Meta-ExternalAgent
User-agent: CCBot
User-agent: Bytespider
User-agent: Amazonbot
Disallow: /

# Allowed (they follow the User-agent: * rules): OAI-SearchBot, Claude-SearchBot, PerplexityBot, DuckAssistBot, ChatGPT-User, Claude-User, Perplexity-User, Meta-ExternalFetcher, MistralAI-User
  • Bytespider is widely reported to ignore robots.txt. Blocking it at the server or firewall is the only sure way.

Upload the file to the top folder of your site so it opens at yoursite.com/robots.txt. On WordPress, an SEO plugin such as Yoast or Rank Math has a robots.txt editor.

robots.txt is a request, not a lock: well-known crawlers follow it, bad ones don't. Crawler names checked October 2026. Want this and the rest of your search setup checked? See the SEO Sprint.

AI crawlers, briefly

Training, search and fetches are three different questions.

Each AI company now runs separate bots for training models, for indexing pages for AI search, and for opening a page when a person asks. Blocking one doesn't block the others, so decide each on its own.

Will blocking AI crawlers hurt my Google ranking?
No. Googlebot is not on this list and is never blocked here. Google-Extended only controls whether Gemini may train on or ground answers in your pages; Google Search and AI Overviews are unaffected.
Should I block AI search bots like OAI-SearchBot or PerplexityBot?
Usually not. They are how ChatGPT search, Claude and Perplexity find and link to you. Most businesses block model training and keep the search bots allowed, which is the tool's default.
Does robots.txt actually stop AI companies?
The major vendors say their crawlers follow it. It is still a request, not a lock: some crawlers are reported to ignore it, and fetches a person asks for may skip it. To be sure, block those at your server or firewall too.
Why does the file not list the bots I allowed?
A crawler follows only the group that names it and then ignores the User-agent: * rules. Giving allowed bots their own group would let them into folders you closed for everyone, so they are left to follow the general rules.
Is anything I enter sent anywhere?
No. The file is built in your browser and nothing you enter leaves this page.

Want to show up in AI answers, not just block them?

Tell us your site. We'll check what search engines and AI assistants can see and what stops them citing you.

Ask about the SEO SprintFixed-price checks