Guide · 9 min read

robots.txt for AI bots: which crawlers to allow and which to block

Your robots.txt AI bots rules decide which assistants can read your site, and many sites block the wrong ones without noticing. This guide lists the crawlers each AI company documents, explains which ones train models and which ones feed answers, and gives you three robots.txt files you can copy, with what each one costs you in visibility.

Technical access, including robots.txt and CDN rules, is part of every RecommendedGEO plan, from $449 a month with a three-month minimum.

Robots.txt AI bots: robots.txt for AI bots: which crawlers to allow and which to block

What robots.txt can and cannot do

robots.txt is a plain text file at the root of your domain. It tells crawlers which paths they may fetch. Each block of rules starts with a User-agent line naming a crawler, followed by Allow and Disallow lines.

Three limits matter before you write anything:

  • It is a request, not a lock. Reputable crawlers follow it; a scraper that ignores it is not stopped by it.
  • It controls fetching, not memory. Blocking a crawler today does not remove what it collected before.
  • A crawler follows only the group that names it most specifically. If you add a group for GPTBot, GPTBot ignores everything under User-agent: *. This is the most common source of mistakes.

Changes are not instant either. OpenAI says search changes can take about 24 hours to apply, and Meta says its crawlers may cache robots.txt for up to a day.

Training crawlers, search crawlers and user fetchers

AI companies now run several bots with different jobs. Mixing them up is how sites end up invisible in ChatGPT while believing they only opted out of training.

  • Training crawlers collect pages that may be used to train future models. Blocking them is an opt-out from training. It does not, by itself, remove you from answers.
  • Search crawlers index pages so the assistant can show and cite them in search-style answers. Blocking them is what makes you disappear from those answers.
  • User fetchers visit a page because a person asked the assistant to look at it. Several vendors say robots.txt may not apply to them, because a human triggered the visit.
  • Control tokens are names you can use in robots.txt that do not crawl anything. They tell a company how it may use what its main crawler already fetched.

robots.txt AI bots: the list by vendor

These are the names the vendors publish in their own documentation. Use them exactly as written; matching is not case-sensitive, but spelling is.

  • OpenAI. GPTBot crawls content that may be used to train its models. OAI-SearchBot surfaces sites in ChatGPT search. ChatGPT-User handles visits triggered by users, and OpenAI notes that robots.txt rules may not apply to it. OpenAI states that each setting is independent, so you can allow OAI-SearchBot and block GPTBot. Source: OpenAI crawlers overview.
  • Anthropic. ClaudeBot collects content that could be used for training. Claude-SearchBot indexes pages to improve search results. Claude-User fetches pages when someone asks Claude a question; blocking it stops Claude from retrieving your pages for users. Source: Anthropic help center.
  • Perplexity. PerplexityBot surfaces and links sites in Perplexity results, and Perplexity says it is not used to train foundation models. Perplexity-User fetches pages during user actions and, in Perplexity's words, generally ignores robots.txt.
  • Google. Google-Extended is a control token with no crawler of its own. It governs whether content Googlebot crawled may be used to train Gemini models and for grounding in Gemini Apps and Vertex AI. It does not affect Google Search. AI Overviews and AI Mode use the normal Search index, so Googlebot is what counts there. Source: Google AI features and your website.
  • Apple. Applebot crawls for Siri, Spotlight and Safari features. Applebot-Extended does not crawl; disallowing it opts your content out of training Apple's foundation models, and Apple says such pages can still appear in its search results.
  • Microsoft. There is no separate Copilot crawler token. Copilot's answers with sources draw on Bing, which is crawled by Bingbot, and Bing Webmaster Tools now reports citations in Copilot. Block Bingbot and you lose both.
  • Common Crawl. CCBot builds an open web archive that many AI datasets have used. Common Crawl documents a Disallow group for CCBot as the way to opt out.
  • Meta. Meta-ExternalAgent crawls for uses such as training foundation models or indexing content for its products. Meta-ExternalFetcher fetches links at a user's request and, per Meta, may bypass robots.txt.

Other bots appear in server logs, and lists of them circulate online. We only include names we could confirm in a vendor's own documentation.

Three robots.txt files you can copy

Each list item below is one group. In the real file, put each directive on its own line and leave a blank line between groups. Replace the sitemap URL with yours.

Option 1, allow everything. This is what we use on our own site:

  • User-agent: * · Allow: /
  • Sitemap: https://www.example.com/sitemap.xml

You do not need a group per AI bot to allow it; the wildcard covers them. Some sites still list them explicitly so the intent is visible to anyone reading the file.

Option 2, allow search and answers, block training:

  • User-agent: GPTBot · Disallow: /
  • User-agent: ClaudeBot · Disallow: /
  • User-agent: Google-Extended · Disallow: /
  • User-agent: Applebot-Extended · Disallow: /
  • User-agent: CCBot · Disallow: /
  • User-agent: Meta-ExternalAgent · Disallow: /
  • User-agent: * · Allow: /

OAI-SearchBot, Claude-SearchBot, PerplexityBot, Googlebot, Bingbot and Applebot fall under the wildcard and stay allowed.

Option 3, block AI crawlers but keep classic search:

  • User-agent: GPTBot · Disallow: /
  • User-agent: OAI-SearchBot · Disallow: /
  • User-agent: ChatGPT-User · Disallow: /
  • User-agent: ClaudeBot · Disallow: /
  • User-agent: Claude-SearchBot · Disallow: /
  • User-agent: Claude-User · Disallow: /
  • User-agent: PerplexityBot · Disallow: /
  • User-agent: Perplexity-User · Disallow: /
  • User-agent: Google-Extended · Disallow: /
  • User-agent: Applebot-Extended · Disallow: /
  • User-agent: CCBot · Disallow: /
  • User-agent: Meta-ExternalAgent · Disallow: /
  • User-agent: Meta-ExternalFetcher · Disallow: /
  • User-agent: * · Allow: /

What each choice costs you in visibility

Option 1 keeps every door open. Your pages can be indexed by AI search engines, fetched when users ask about you and included in training data. For a business that wants to be recommended, this is usually the right default.

Option 2 is a reasonable middle ground. You stay citable in ChatGPT search, Claude, Perplexity, Copilot and AI Overviews. The trade-offs: future models learn less about you from your own site, and blocking Google-Extended also limits grounding in Gemini Apps, which is an answer surface, not only training.

Option 3 removes you from most assistant answers that cite sources, and user-triggered fetchers may still visit because of how vendors treat them. It does not remove you from AI Overviews: those rely on Googlebot, and blocking Googlebot removes you from Google Search too. To limit what Google shows, use nosnippet or noindex on specific pages instead.

If you sell anything that people research by asking an assistant, blocking search crawlers rarely makes sense. Your competitors will be the ones cited.

How to check what your robots.txt does today

  1. Open yourdomain.com/robots.txt in a browser. Check that it loads with a 200 status and is not an HTML error page.
  2. Find every group that names an AI bot. Remember that a named group replaces the wildcard for that bot, so a stray Disallow there wins.
  3. Look for broad rules such as Disallow: / under User-agent: *, often left over from a staging site.
  4. Use the robots.txt report in Google Search Console to see the version Google fetched and any parsing errors.
  5. Check your server or CDN logs for GPTBot, OAI-SearchBot, ClaudeBot, PerplexityBot and Bingbot, and note the status codes. Repeated 403 or 429 responses mean something other than robots.txt is blocking them.
  6. Ask ChatGPT, Claude or Perplexity to open a specific URL from your site and summarize it. If they cannot, investigate before assuming the problem is content.

The silent blocker: CDN and firewall settings

A clean robots.txt AI bots setup can still fail if your CDN or firewall turns the crawlers away before they reach the file. The bot sees an error or a challenge page, and nothing in robots.txt tells you.

Cloudflare, for example, has AI Crawler and Agent access policies under Security settings. You set an action separately for search, agent and training behaviour: block, block only on pages with ads, allow, or, for training, a disallow signal. Cloudflare's documentation says new domains default to allow, but a setting chosen months ago, a custom WAF rule or a hosting provider's bot protection can still block the crawlers you want.

  • Review every bot-related rule in your CDN, WAF and hosting panel, not only robots.txt.
  • Make sure search behaviour is allowed if you want to appear in AI answers.
  • After any change, check the logs again for 200 responses from the crawlers you care about.

Where this fits in GEO

Access is the first condition of generative engine optimization. If an assistant cannot read your pages, nothing you publish there can be cited. Getting your robots.txt AI bots rules right is quick, and it is the first thing we check in a free GEO audit.

Access is necessary but not sufficient. After it come pages that answer real questions, covered in how to write content AI cites, an optional llms.txt file, and a way to see whether anything changed, which we explain in how to measure AI visibility.

Frequently asked questions

Should I block GPTBot?

Only if you do not want your content used to train OpenAI's models. Blocking GPTBot does not remove you from ChatGPT search; that depends on OAI-SearchBot, which OpenAI treats as a separate setting.

Does blocking Google-Extended remove me from AI Overviews?

No. Google says Google-Extended does not affect Google Search, and AI Overviews use the Search index crawled by Googlebot. To limit what appears there, use page-level controls like nosnippet. More in how to appear in AI Overviews.

Do I need a separate group for each AI bot to allow it?

No. A User-agent: * group with Allow: / covers every crawler that has no group of its own. Separate groups are useful when you want to treat a bot differently from the rest.

Why do some AI fetchers ignore robots.txt?

OpenAI, Perplexity and Meta describe their user-triggered fetchers as acting on behalf of a person who asked for a specific page, so they say robots.txt may not apply. If you need to stop them, that has to happen at the server or CDN level.

How often should I review these rules?

Whenever a vendor announces a new crawler and at least a few times a year. The robots.txt AI bots landscape changes, and a name you did not list falls under your wildcard rules.

The four workstreams of GEO: measuring AI answers, citable content, outside presence and technical GEO
AI assistants tracked every month: ChatGPT, Gemini, Perplexity, Claude, Copilot and Google AI Overviews
Monthly GEO process: measure, fix, publish and report

Pay with Stripe

On the live site this button takes you straight to secure card payment (Stripe). This prototype doesn't charge yet.

Scroll to Top