AI Crawlers

Four crawler types decide which AI crawlers to block, and which to leave alone

Deciding AI crawlers to block or allow means telling four different jobs apart first, and the ones collecting training data are not the ones deciding whether an assistant can find you. Block the wrong one and you are not ranked lower, you are absent.

yoursite.com/robots.txt

User-agent: *Allow: /# Training crawlerUser-agent: GPTBotDisallow: /# Search crawler, same operatorUser-agent: OAI-SearchBotAllow: /Sitemap: yoursite.com/sitemap.xml

Key Takeaways

AI Crawlers to Block or Allow, in Short

Training, search-index, retrieval, and opt-out-token crawlers each do a different job, and blocking one has a different cost than blocking another.
Blocking a training crawler like GPTBot or ClaudeBot costs nothing in visibility; blocking a search-index crawler like OAI-SearchBot removes you from that surface entirely.
Google-Extended is a control name, not a crawler, and never appears in a request log.
Several operators, including OpenAI, Perplexity, and Meta, document that a retrieval fetch triggered by a person may not honour robots.txt at all.
Saving a rule and having it actually served are two different events; only fetching the live file from outside your own network confirms the second.

The Distinction

Four kinds of AI crawler, and why deciding AI crawlers to block or allow starts here

Nearly every mistake on this subject comes from treating four different things as one thing.

Training

Collects pages to train future models. Blocking it changes nothing about whether you appear in search or in an assistant’s answer.

GPTBot

Search index

Builds the index an AI search surface answers from. Blocking it takes you out of that surface.

OAI-SearchBot

Retrieval

Fetches a page live because a person asked about it. Several operators document that robots.txt may not apply here.

Perplexity-User

Opt-out token

Not a crawler. A name that exists only to be addressed in robots.txt, governing how already-collected data may be used.

Google-Extended

A fifth kind is forming: autonomous agents that act across several pages on a person's behalf. Almost nobody documents it, and no crawler taxonomy yet models it as a tier of its own, including ours.

The Reference

Every major AI crawler, and what it does

Every row below was checked against the source shown beside its group, on the date shown beside its group.

OpenAI4 crawlers

Checked 2026-09-13 · Default source for this group: developers.openai.com/api/docs/bots . Rows with different or qualifying evidence are linked individually. A blank Source cell means the row is covered by this default.

OpenAI AI crawlers: token, category, whether each respects robots.txt, what blocking costs, and source
Token	Category	Respects robots.txt	What blocking costs
`GPTBot`	Training	Yes	OpenAI takes it as a signal that your content should not be used to train its generative AI foundation models. No documented effect on search or on ChatGPT answers.
`OAI-SearchBot`	Search index	Yes	You will not be shown in ChatGPT search answers. OpenAI states you can still appear as a navigational link.
`ChatGPT-User`	Retrieval	Not reliably	A stated preference only. OpenAI’s documentation says that because these actions are initiated by a user, robots.txt rules may not apply.
`OAI-AdsBot`	Ads safety	Yes	It only fetches ad landing pages that were submitted to OpenAI. OpenAI states this data is not used to train its foundation models.

Anthropic4 crawlers

Checked 2026-09-13 · Default source for this group: support.claude.com, crawling and blocking . Rows with different or qualifying evidence are linked individually. A blank Source cell means the row is covered by this default.

Anthropic AI crawlers: token, category, whether each respects robots.txt, what blocking costs, and source
Token	Category	Respects robots.txt	What blocking costs	Source
`ClaudeBot`	Training	Yes	Anthropic states that restricting it signals your site’s future materials should be excluded from its model training datasets. Content already collected is not withdrawn.
`Claude-SearchBot`	Search index	Yes	Your content is not indexed for Claude’s search.
`Claude-User`	Retrieval	Yes	Claude cannot retrieve your content in response to a user’s question.
`anthropic-ai` `Claude-Web`	Legacy	Not documented	Anthropic’s current documentation lists neither token. They survive in copied robots.txt templates.	Absence. The page that does not list them

Google5 crawlers

Checked 2026-09-13 · Default source for this group: developers.google.com, Google common crawlers . Rows with different or qualifying evidence are linked individually. A blank Source cell means the row is covered by this default.

Google AI crawlers: token, category, whether each respects robots.txt, what blocking costs, and source
Token	Category	Respects robots.txt	What blocking costs	Source
`Googlebot`	Search index	Yes	Removal from Google Search including Discover and every Search feature, and from Google Images, Video and News. Almost never the right choice.
`Google-Extended`	Opt-out token	Not a crawler	Excludes your content from training future Gemini models and from Vertex AI grounding. Google states it does not affect inclusion in Google Search and is not used as a ranking signal.	Google states this directly. Google crawling documentation
`GoogleOther`	Research	Yes	Google product teams stop fetching your publicly accessible pages for research and development.
`Google-CloudVertexBot`	Customer-directed	Yes	It only crawls sites at the request of a site owner building Vertex AI agents.
`Google-InspectionTool`	Testing	Yes	The Rich Results Test and URL inspection stop working on your pages.

Perplexity2 crawlers

Checked 2026-09-13 · Default source for this group: docs.perplexity.ai/guides/bots . Rows with different or qualifying evidence are linked individually. A blank Source cell means the row is covered by this default.

Perplexity AI crawlers: token, category, whether each respects robots.txt, what blocking costs, and source
Token	Category	Respects robots.txt	What blocking costs	Source
`PerplexityBot`	Search index	Yes	Removal from Perplexity’s search results. Perplexity states this crawler is not used for AI model training.
`Perplexity-User`	Retrieval	No	Blocking is not reliably effective. Perplexity’s documentation says that since a user requested the fetch, this fetcher generally ignores robots.txt rules.

Apple3 crawlers

Checked 2026-09-13 · Default source for this group: support.apple.com/en-us/119829 , published 8 June 2026. Apple’s page does not render to automated retrieval, so these rows were checked against independent secondary sources rather than read at the primary, re-attempted 2026-08-03 with the same result. Rows with different or qualifying evidence are linked individually. A blank Source cell means the row is covered by this default.

Apple AI crawlers: token, category, whether each respects robots.txt, what blocking costs, and source
Token	Category	Respects robots.txt	What blocking costs
`Applebot`	Search crawler. Its data may also support model training and current AI-answer context	Yes, with documented fallback to Googlebot rules	Blocking prevents future crawling of the affected paths and can reduce their availability or freshness across Spotlight, Siri, Safari and other Apple search experiences. Training and AI-answer use have narrower controls: `Applebot-Extended` and `nosnippet`.
`Applebot-Extended`	Opt-out token	Not a crawler	Disallowing it excludes Applebot-crawled content from foundation-model training while leaving the content eligible for Apple search.
`iTMS`	Podcasts	No	Apple states this agent does not follow robots.txt because it is not a general search crawler. It fetches only URLs for content registered with Apple Podcasts.

Microsoft2 crawlers

Checked 2026-09-13 · Default source for this group: blogs.bing.com, new options for webmasters . Rows with different or qualifying evidence are linked individually. A blank Source cell means the row is covered by this default.

Microsoft AI crawlers: token, category, whether each respects robots.txt, what blocking costs, and source
Token	Category	Respects robots.txt	What blocking costs	Source
`bingbot`	Search index	Yes	Removal from Bing search results over time. Copilot experiences draw on the Bing index, so a page absent from that index is absent from them, but blocking the crawler does not evict pages already in it.	Derived, not stated. Bing documents the index-to-Copilot link, not this of robots.txt
`msnbot`	Legacy	Yes	Superseded by `bingbot`. Current use could not be confirmed from Microsoft’s own documentation.	Aggregator only. No Microsoft page confirms current use

Amazon4 crawlers

Checked 2026-09-13 · Default source for this group: developer.amazon.com/amazonbot . Rows with different or qualifying evidence are linked individually. A blank Source cell means the row is covered by this default.

Amazon AI crawlers: token, category, whether each respects robots.txt, what blocking costs, and source
Token	Category	Respects robots.txt	What blocking costs	Source
`Amazonbot`	Search index, and may train	Yes, checked at host level	Removal from Alexa answers and Amazon services. Training use is controlled separately by the page-level `noarchive` tag rather than by a second token.
`Amzn-SearchBot`	Search index	Documented, not fully confirmed	Amazon states it does not train generative AI models.
`Amzn-User`	Retrieval	Documented, not fully confirmed	Amazon states it does not train generative AI models.
`bedrockbot`	Customer-directed	Yes	It crawls only the URLs an AWS Bedrock customer selected.	Aggregator only. Not on Amazon’s amazonbot page

Meta3 crawlers

Checked 2026-09-13 · Default source for this group: developers.facebook.com, web crawlers . Rows with different or qualifying evidence are linked individually. A blank Source cell means the row is covered by this default.

Meta AI crawlers: token, category, whether each respects robots.txt, what blocking costs, and source
Token	Category	Respects robots.txt	What blocking costs
`meta-externalagent`	Training and indexing	Yes	Excludes your content from training Meta’s foundation models and from the indexing that improves its products.
`meta-externalfetcher`	Retrieval	May bypass	Blocking is not reliably effective. Meta documents that this user-initiated fetcher may bypass robots.txt.
`facebookexternalhit`	Link preview	Generally, and may bypass for malware checks	Your links stop rendering a preview card on Facebook, Instagram and Messenger. It has nothing to do with AI training.

ByteDance1 crawler

Checked 2026-09-13 · This group has no single default source. ByteDance publishes no crawler documentation page equivalent to the other operators here. Every row is linked individually.

ByteDance AI crawlers: token, category, whether each respects robots.txt, what blocking costs, and source
Token	Category	Respects robots.txt	What blocking costs	Source
`Bytespider`	Training	Disputed	ByteDance is widely reported to say Bytespider respects robots.txt, but publishes no crawler documentation stating so. Site owners report continued crawling of disallowed paths. Nothing dated and authoritative settles it either way.	No primary source exists. ByteDance publishes no crawler documentation, and the counter-accounts are webmaster reports rather than a dated study

Common Crawl1 crawler

Checked 2026-09-13 · Default source for this group: commoncrawl.org/ccbot . Rows with different or qualifying evidence are linked individually. A blank Source cell means the row is covered by this default.

Common Crawl AI crawlers: token, category, whether each respects robots.txt, what blocking costs, and source
Token	Category	Respects robots.txt	What blocking costs	Source
`CCBot`	Training dataset	Yes	Removal from a public open dataset that many AI trainers use downstream. Content already collected stays in the published archives.

Smaller operators and the long tail9 crawlers

Checked 2026-09-13. The Source cell states the grade of the source.

Smaller AI-crawler operators and the long tail: token and operator, category, whether each respects robots.txt, what blocking costs, and source
Token	Category	Respects robots.txt	What blocking costs	Source
`DuckAssistBot` DuckDuckGo	Retrieval	Yes	Removal from DuckDuckGo’s assistant answers. The opt-out takes effect after 72 hours, and DuckDuckGo states it does not affect organic search rankings.	DuckDuckGo help, vendor
`Bravebot` Brave	Search index and retrieval	Yes	Removal from Brave Search’s AI answers.	Aggregator
`YouBot` You.com	Search index and training	Yes	One token covering both purposes, so there is no partial choice.	Aggregator
`AI2Bot` Ai2	Training	Yes	Exclusion from open research datasets.	Aggregator
`Diffbot` Diffbot	Data extraction	Configurable per customer	Removal from a knowledge graph whose output feeds many downstream AI products.	Aggregator
`webzio-extended`, replacing `omgili` and `omgilibot` Webz.io	Data aggregation for AI training	Reported yes	Most circulating templates still name only `omgili`, so a block written from one of those misses this.	Aggregator
`Timpibot` Timpi	Search index	Could not be confirmed	Removal from a decentralised index some assistants retrieve from.	Aggregator
`cohere-ai` Cohere	Training	Could not be confirmed	No vendor documentation page could be located.	Behavioural reports only
`MistralAI-User`, `Kimi-User`, `LinerBot`, `PhindBot`, `kagi-fetcher` Various	Retrieval	Varies by operator	A growing long tail, each with its own policy.	Aggregator

Anthropic is the only major operator that states all three of its crawlers honour robots.txt, including the one a person triggers. OpenAI, Perplexity and Meta each carve that case out. Anthropic also supports Crawl-delay and publishes its verification IP ranges at claude.com/crawling/bots.json.

Google-Extended is the most misread token on this page. It is a control name, not a user agent, and it never appears in a request log. It does not affect AI Overviews or AI Mode, which are built from the regular Search index that Googlebot populates. The control for those is the Search Console toggle in section five.

In August 2025 Cloudflare published findings that Perplexity had reached content through undeclared crawlers using rotating addresses and a generic browser user agent, after its declared bots were blocked, and removed Perplexity from its verified bots list. Perplexity denied the findings. That is the position as of August 2025; the current status of the delisting was not confirmed at this page's last review.

If your robots.txt does not name Applebot but does name Googlebot, Apple follows your Googlebot rules. That is Apple's documented behaviour as described by independent secondary sources; Apple's own page does not render to automated retrieval and we have not read it directly. It means a file written only for Google is silently governing Apple too. Applebot also ignores crawl-delay.

Apple documents three uses of one crawl, and each has its own control. Search is governed by Applebot in robots.txt. Foundation-model training is governed by Applebot-Extended. Use as context in AI-generated answers is governed by nosnippet, or by marking a page isAccessibleForFree: false. Blocking the crawler is the broadest of the three and the only one that costs you search. As above, checked against secondary sources rather than at Apple's own page.

Microsoft publishes no AI training token. There is no Bing-Extended. Opting out of Microsoft's AI training means tagging every page with noarchive, or blocking bingbot and losing Bing with it. It is the clearest case on this page of robots.txt not being the whole control surface.

Common Crawl also runs an opt-out registry and publishes its addresses at index.commoncrawl.org/ccbot.json, noting that other crawlers falsely identify themselves as CCBot.

The most complete public directory of these moved from darkvisitors.com to knownagents.com and the old address now redirects, so a block list built from an older article may be citing a page that no longer maintains itself.

The Cost

What does blocking AI crawlers actually cost?

Blocking is not one decision with one price, and the price is different in each of the four categories.

Blocking training crawlers: Costs nothing in visibility. No effect on Google rankings, no effect on whether an assistant can find you, no effect on AI Overviews.
Blocking search-index crawlers: Takes you out of that assistant’s answers. You are not ranked lower there, you are absent, and there is no partial state.
Blocking retrieval fetchers: Works reliably only where the operator says it does. Anthropic says it honours robots.txt here. OpenAI, Perplexity and Meta each document that they may not.
Blocking an opt-out token: Costs nothing, because nothing fetches. It is a statement about use, not about access.
Blocking everything with one template: Opts you out of training and removes you from AI search in the same edit. Usually only the first was intended.

5.89%

Share of sites blocking GPTBot in robots.txt.

Ahrefs, roughly 140 million websites, data through December 2024, published 2025-05-21.

robots.txt is a published request under RFC 9309 with no technical enforcement behind it. Where an operator documents that it may ignore the request, that is stated policy rather than rogue behaviour, and it is worth reading as such.

The internet is not blocking AI at scale. Almost every site you are comparing yourself against has made no decision at all.

Beyond Robots

Which controls are not robots.txt at all?

Some of the controls that matter most this year are not robots.txt directives.

Controls that are not robots.txt directives: where each lives, what it does, and who honours it
Control	Where it lives	What it does	Who honours it
Generative-AI toggle	Google Search Console, under Settings	Excludes the site from AI Overviews, AI Mode and generative features in Discover. Domain-level, three states. Google states it does not affect conventional search ranking or inclusion. In effect 2026-06-17, documented in Search Console Help 2026-07-20, page-level controls scheduled 2027-03-03. Introduced under a binding UK Competition and Markets Authority conduct requirement.	Google
`noarchive`	Page `<head>`, or `X-Robots-Tag`	Bing: the page is excluded from Copilot answers and from Microsoft’s generative AI training data. Amazon: the page is not used for model training.	Microsoft, Amazon
`nocache`	Page `<head>`	Bing: only the URL, title and snippet may appear in chat answers. The content may still be used for training.	Microsoft
`nosnippet`	Page `<head>`	Google: suppresses AI features on the page, and the organic search snippet along with them.	Google
`noai`, `noimageai`	Page `<head>`, or `X-Robots-Tag`	States that content should not be used for AI training or image training. A DeviantArt community initiative, not a web standard, with no formal specification. More than 88,000 domains carried a tag as of June 2026, per Originality.AI’s public dashboard.	No documented support from any major crawler
`nosnippet`, Apple	Page `<head>`, or `X-Robots-Tag: applebot: nosnippet`	Apple: the page is not used as additional context when its AI models generate output, and any suggestion to visit it shows the title only.	Apple
`isAccessibleForFree: false`	Page-level JSON-LD	Apple: a page marked as paywalled, metered or subscription stays eligible for search results but is not used as AI context. Page level only; `hasPart` section markup is not supported.	Apple
`X-Robots-Tag`	HTTP response header	Applies the same page-level directives to files with no `<head>`, such as PDFs and images. Apple supports a vendor-prefixed form.	The vendors above, per directive

Two of these are per page, which makes them the opposite of robots.txt in operational terms: precise about which content they cover, and only as good as your ability to apply them everywhere they belong. The two Apple rows above were checked against secondary sources, because Apple's own documentation page does not render to automated retrieval.

What Is Coming

Content Signals, licensing, and what is still only a draft

Three newer mechanisms are worth knowing about, and only one of them changes what happens on the network today.

Newer mechanisms worth knowing about: Cloudflare Content Signals, Cloudflare Pay Per Use and Cloudflare default blocking, their status, and whether each is enforceable
Mechanism	Status	Enforceable?
Cloudflare Content Signals	Launched September 2025. Extends robots.txt with three signals, `search`, `ai-input` and `ai-train`, each taking yes, no, or omitted. Cloudflare applied `search=yes, ai-train=no` to managed robots.txt across more than 3.8 million domains.	Only by a few. Cloudflare’s own framing is that these are preferences and not technical countermeasures. Its Browser Run crawl endpoint reads them and rejects a crawl for a purpose set to no, but no other major crawler is known to honour them as a standalone directive.
Cloudflare Pay Per Use	Replaced Pay Per Crawl on 2026-07-01. Pays publishers when their content appears in an answer, rather than when a bot fetches the page.	A commercial arrangement, not a standard.
Cloudflare default blocking	From 2026-09-15, new Cloudflare sites and customers default to allowing search crawling while blocking training and agent use on pages carrying ads. Mixed-use crawlers offering no per-purpose choice are blocked by default on those pages.	Yes, at the network layer. This one is enforcement rather than a signal.
RSL 1.0	Really Simple Licensing launched 2025-09-10 and was published as 1.0 on 2025-12-10, expressed as licensing terms inside robots.txt.	No enforcement. It works only where an AI company chooses to honour it.
IETF AIPREF	`draft-ietf-aipref-vocab` published 2026-04-28 with Standards Track intent. Still a draft.	Nothing to implement yet.

Content Signals sit inside an existing robots.txt group and are read alongside the rules already there.

User-agent: *
Content-Signal: search=yes, ai-input=yes, ai-train=no
Disallow: /wp-admin/

search covers building a search index, ai-input covers using the page to build an answer at the moment someone asks, and ai-train covers training a model. Setting one to no states a preference. It does not stop a request arriving.

A CDN or firewall decides before robots.txt is ever read. Bot-fighting features classify AI crawlers as hostile and block them at the network layer, which is a common reason a crawler you allowed is not reaching you.

The Decision

What should you actually do?

There is no single correct configuration, and the right one depends on how the business makes money.

You sell something and want to be found

Allow the search-index and retrieval crawlers. Block the training crawlers if you object to that use. This is the right default for most SaaS, most services businesses and most ecommerce.

Your content is the product

Block the training crawlers. Think hard about search and retrieval, because an assistant summarising your work can replace the visit. Some publishers take that trade for reach and attribution. Others do not.

You run documentation or developer tools

Allow nearly everything. Being the source an assistant quotes when someone asks how your product works is close to ideal.

You run a staging, internal or headless site

Block everything. This is the one case where blanket blocking is clearly right, and it is worth checking, because staging sites getting indexed is common.

You are not sure

Allow search and retrieval, block training, and revisit in six months. It is the reversible choice.

There is no consensus here, and you should not let anyone tell you there is.

Set It Up

Setting crawler rules on a WordPress site

The block below is the common configuration: out of training, still in AI search.

Paste this at the end of your robots.txt, or use it as the starting point in whichever editor owns the file.

User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: CCBot
Disallow: /

User-agent: Google-Extended
Disallow: /

User-agent: Applebot-Extended
Disallow: /

User-agent: meta-externalagent
Disallow: /

User-agent: Bytespider
Disallow: /

User-agent: *
Allow: /

Sitemap: https://example.com/sitemap.xml

Every group above names a training crawler or an opt-out token. Nothing in it touches OAI-SearchBot, Claude-SearchBot, PerplexityBot, Applebot, bingbot or Googlebot, so search and retrieval are untouched. Replace example.com with your own host. robots.txt is read per host, so every subdomain needs its own file at its own root.

Edit robots.txt directly

If a physical robots.txt sits at your web root, edit it there. Rules are per user agent, and a single Disallow: / under a named agent covers the whole site for that agent.

Through your SEO plugin

Yoast SEO, Rank Math and All in One SEO include a robots.txt editor in their free versions. You get a text field, which is fine if you already know which agents you want and how they are spelled. Rank Math's advanced editor and tester sit behind its PRO tier.

Through IndexMesh for WordPress

The free IndexMesh WordPress plugin groups the crawlers in its bundled taxonomy by what they do, so you choose a purpose instead of remembering user-agent strings. It renders Content Signals directives inside its own managed block, shows the exact robots.txt output on its Preview tab after you save, and checks afterwards that the file is really being served.

See A Free WordPress Plugin That Works As a Real Block AI Crawlers Alternative.

The Screens

What do the controls look like?

Two screens carry most of this module: the presets, and the per-crawler list underneath them.

The IndexMesh WordPress plugin's Robots screen, with crawler presets grouped by what each crawler does — Presets grouped by crawler purpose, with Allow and Block set per group. You apply them; nothing is blocked for you.

The IndexMesh WordPress plugin's granular crawler controls, listing WordPress defaults, social crawlers and sitemaps separately — Granular control over WordPress paths, social crawlers, sitemaps and search engines, grouped by purpose.

The IndexMesh WordPress plugin's AI crawler controls, listing GPTBot, OAI-SearchBot, ChatGPT-User, PerplexityBot, ClaudeBot and more, each tagged by bundled purpose — Named AI crawlers, not a blanket rule: GPTBot, ClaudeBot, PerplexityBot and more, each tagged by purpose.

The IndexMesh WordPress plugin's bad-bot blocklist and per-file-type crawl controls for images and documents — A curated bad-bot blocklist and per-file-type controls, alongside the AI crawler list above.

The IndexMesh WordPress plugin's robots.txt preview, showing the exact file the site will serve, generated from the saved rules — The exact robots.txt the site will serve, generated from the saved rules.

Alongside Yours

How this works alongside what you already run

Keep the SEO platform you already trust. IndexMesh is not another one: it is an AI-discovery governance layer that sits around it.

Yoast SEO

What it already provides: Creating and editing a physical robots.txt as raw text where the server allows it. If file editing is unavailable, it directs you to edit or upload the file at server level
How IndexMesh differs: Quick policy setups, controls grouped by what a crawler does, an Allow, Block or Off decision per group, the agent’s role and the consequence stated beside it, a preview of the composed output, and a check that the rule is actually being served
How they coexist: The plugin appends its managed block after existing output through the robots_txt filter rather than replacing it

Rank Math

What it already provides: Robots controls within its own settings
How IndexMesh differs: The same structured policy model, plus ownership detection and import
How they coexist: Rules from another plugin are read from the served output only, never from its private settings

All in One SEO

What it already provides: Robots controls within its own settings
How IndexMesh differs: As above
How they coexist: As above

SEOPress

What it already provides: Robots controls within its own settings
How IndexMesh differs: As above
How they coexist: As above

Plain WordPress

What it already provides: A virtual robots.txt with almost nothing in it
How IndexMesh differs: The whole module, with no SEO plugin required
How they coexist: No conflict to resolve

One is a text editor. The other is a policy manager.

Yoast gives an experienced operator a file editor, and for someone who knows every token by heart that is enough. The plugin gives grouped controls, a preview, ownership checks and a deliberate takeover path instead.

Verify

How do you check the rules are actually being served?

Saving a rule and serving a rule are two different events, and only the second one is checkable.

Fetch your robots.txt from outside your own network.
Request https://yourdomain.com/robots.txt with curl or from a device that is not logged in. A logged-in browser session and a local cache both lie.
Confirm the status code is 200 and the content type is text/plain.
A 404, a redirect or an HTML error page all mean no rules are in force at all.
Compare the served body with what you saved, line for line.
If they differ, something between WordPress and the reader is rewriting or replacing the file.
Check whether a physical robots.txt exists at your web root.
A physical file there is served in preference to WordPress’s virtual output, and a plugin that reports that state is telling you the truth about what it controls. The general version of this question, which thing owns a public route, is covered on the llms.txt page.
Check your CDN or firewall separately.
Bot-fighting rules block at the network layer and never consult robots.txt, so an allowed crawler can still be turned away before it reaches your site.
Verify a bot before you trust its user agent.
Google warns plainly that the user agent string can be spoofed. OpenAI, Anthropic, Perplexity and Common Crawl publish their crawler addresses as JSON at openai.com/gptbot.json, claude.com/crawling/bots.json, perplexity.com/perplexitybot.json and index.commoncrawl.org/ccbot.json. Apple publishes an IP CIDR list and documents reverse DNS in the *.applebot.apple.com domain, verified in both directions. Match the requesting address against the published list, or use a reverse DNS lookup.
Check the other public routes while you are there.
Your sitemap, your homepage schema, any /.well-known/ file you serve, and anything else a crawler is meant to find at a fixed address.

See A Route Check Alone Isn't AI Crawler Reading Proof.

Mistakes

Mistakes worth avoiding

Each of these is common, and each one leaves the site owner believing something happened that did not.

Blocking OAI-SearchBot while meaning to block GPTBot: One token of difference between opting out of training and leaving ChatGPT’s search answers.
Copying a block-everything template without reading it: Most of them include the search and retrieval crawlers, and several include tokens that have nothing to do with AI.
Blocking facebookexternalhit: It stops nothing being trained. It stops your links rendering a preview card on Facebook, Instagram and Messenger.
Assuming Google-Extended affects AI Overviews: It does not. Those are built from the Search index Googlebot populates, and the control is the Search Console toggle.
Writing rules for Googlebot and assuming Apple is unaffected: Apple is documented as following your Googlebot rules where robots.txt does not mention Applebot, so a file written for one search engine can end up governing two. Checked against secondary sources; see the reference section.
Blocking a retrieval fetcher and believing it worked: Several operators document in their own words that a fetch a person asked for may ignore robots.txt.
Assuming your CDN is not overriding you: It is a common reason a crawler you allowed is not reaching your site, and nothing in WordPress will tell you it happened.
Saving a rule and never checking it is served: Caching layers, security plugins and CDNs all interfere, and none of them tell you.
Treating robots.txt as security: It is a published statement of preference. If something must not be reached, it needs authentication, not a polite request.

See AI Crawler Blocking Mistakes: What Reddit's 86% Citation Drop Actually Teaches.

FAQ

Common questions

Twelve questions people ask before changing a robots.txt file they cannot easily test.

What is an AI crawler?

An AI crawler is an automated program an AI company runs to fetch web pages. There are three kinds of program: training crawlers such as GPTBot, search-index crawlers such as OAI-SearchBot, and retrieval fetchers that load a page when a person asks. A fourth item, the opt-out token such as Google-Extended, is a control name and not a program.

Are AI crawlers like GPTBot blocked by default?

Not by robots.txt. With no rule, a crawler is allowed. Two things can block one without a rule from you: a CDN or firewall bot-fighting feature, which decides before robots.txt is read, and Cloudflare’s default for new Cloudflare sites, which from 2026-09-15 allows search crawling while blocking training and agent use on pages carrying ads. Fetch your live robots.txt and check your host’s bot settings to see which applies.

Does blocking AI crawlers hurt my Google rankings?

Blocking training crawlers such as GPTBot, ClaudeBot and CCBot, or setting the Google-Extended token, has no effect on Google rankings. Blocking Googlebot removes you from Google Search entirely. They are separate crawlers with separate jobs.

What is the difference between GPTBot and OAI-SearchBot?

Both are OpenAI’s. GPTBot collects training data. OAI-SearchBot builds the index ChatGPT search answers from. Blocking GPTBot costs you no visibility. Blocking OAI-SearchBot means you will not be shown in ChatGPT search answers, though OpenAI states you can still appear as a navigational link.

Does Google-Extended affect AI Overviews?

No. Google-Extended governs training for future Gemini models and grounding in Vertex AI. AI Overviews and AI Mode are built from the regular Search index, which Googlebot populates, so the Search Console toggle is the control for those.

Do AI crawlers actually obey robots.txt?

The major operators state that their declared crawlers do, and there is no technical enforcement behind that statement. OpenAI, Perplexity and Meta each document that their user-initiated fetchers may ignore robots.txt because a person asked for the page. Anthropic states that all three of its crawlers honour it. ByteDance publishes no crawler documentation at all: it is widely reported to say Bytespider respects robots.txt, site owners report otherwise, and nothing dated settles it, so the table marks that one disputed.

Can I stay in Google Search but out of AI Overviews?

Yes. Google Search Console carries a toggle under Settings that excludes a site from AI Overviews, AI Mode and generative features in Discover. It took effect on 2026-06-17 and Google states it does not affect conventional search ranking. It applies to a whole domain today, and page-level controls are scheduled for 2027-03-03.

Does blocking ClaudeBot remove me from Claude's answers?

No. ClaudeBot is Anthropic’s training crawler, and Anthropic describes restricting it as a signal that your site’s future materials should be excluded from its training datasets. Claude’s search uses Claude-SearchBot and per-user fetches use Claude-User.

Which AI crawlers should most sites allow?

Most businesses that want to be found allow the search-index and retrieval crawlers, including OAI-SearchBot, Claude-SearchBot, PerplexityBot, Applebot and bingbot, and decide separately about the training crawlers based on how they feel about that use.

How do I check whether my robots.txt rules are working?

Request your own robots.txt from outside your network and read exactly what comes back. Confirm a 200 response and a text/plain content type, compare the body against what you saved, and check your CDN’s bot settings separately, because those never consult robots.txt.

Should I block AI crawlers to protect my content from training?

It is a legitimate choice and it carries no search cost, provided you block only the training crawlers and set the opt-out tokens. It is a preference rather than enforcement, and it applies going forward: content already collected is not withdrawn by a later block.

Does a plugin help manage this in WordPress?

Yes. The free IndexMesh WordPress plugin groups the crawlers in its bundled taxonomy by what they do, so you choose Allow, Block or Off per group instead of writing rules by hand. It does not block anything automatically: staging, local and headless sites get a prompt, not a rule.

What You Can Do Now

Setting crawler rules without editing robots.txt by hand

The distinctions on this page are easy to get wrong in a text file, and expensive when you do.

The IndexMesh WordPress plugin groups crawlers by what they actually do, shows the exact robots.txt output on its Preview tab after you save, and checks that it's really being served.

Free, and it works alongside your existing SEO plugin.

See the WordPress plugin

Set crawler rules in WordPress →

We will tell you when it is ready.

This form needs JavaScript to submit. Email hello@indexmesh.ai instead, and we will get it.

You're on the list. We'll let you know the moment Mesh AI is live.

One email when it is ready. After that, only when there is something genuinely worth reading. Unsubscribe in one click.

Where your brand gets found, beyond your own site