AI Crawler Blocking Mistakes: What Reddit's 86% Citation Drop Actually Teaches
If you manage your own site's robots.txt, the most common AI crawler blocking mistake, and the one IndexMesh for WordPress is built to prevent, is a blanket block that treats every AI bot as one thing. Reddit's own robots.txt block caused an 86% ChatGPT-citation drop in four days in August 2026, because it blocked training and search/retrieval crawlers together.
AI Crawler Blocking Mistakes, in Short
That is the mistake worth naming directly, because it is easy to make and expensive to reverse: someone decides "AI bots" are one category, writes one Disallow: / rule for all of them, and finds out afterward that the rule also removed the site from the answers AI search tools were giving about it.
Set up tiered AI crawler blocking instead: AI crawler controls in IndexMesh for WordPress sets Allow, Block, or Off per crawler tier, so you never have to write one blanket switch for every AI bot.
- Blocking every AI bot with one
robots.txtrule is the single most common AI crawler blocking mistake, because it treats two different bot roles as if they were one. - Reddit's ChatGPT Search citation share fell from 3.83% to 0.52% in four days in August 2026, after a domain-wide
robots.txtdisallow blocked training and retrieval crawlers together. - You can block AI training crawlers without losing AI search visibility:
GPTBotandClaudeBotare a separate role fromOAI-SearchBotandPerplexityBot. - Opt-out tokens like
Google-Extendedare policy signals, not crawlers, so writing a Block rule against them blocks nothing measurable. - Checking the live
/robots.txtroute after saving matters as much as writing the rule, since a plugin setting and the file actually being served aren't guaranteed to match.
What actually happened to Reddit
In August 2026, Reddit's share of citations in ChatGPT Search fell from 3.83% to 0.52%, an 86.4% relative drop, in four days. Multiple independent trackers, including explainx.ai and The State of Brand, confirmed the same pattern over the same window. The trigger was a domain-wide robots.txt disallow that Reddit added, one rule applied to the whole site rather than to a specific category of bot.
A blanket robots.txt block risk is exactly this: it does not distinguish between a crawler that trains a model on your content and a crawler that fetches your page to answer a question someone just asked. Reddit's rule blocked both at once. The domain-wide AI crawler block consequence was immediate and measurable, a citation share that had taken time to build gone in days, with no way to know how long recovery will take.
Why "block AI bots" is not one decision
robots.txt files are usually written by whoever is worried about something, not by whoever understands the full list of user-agents involved. That produces one of two outcomes: a rule too narrow to cover the actual risk, or a rule so broad it disallows visibility the site wanted to keep.
AI crawlers fall into different roles, and each role has a different consequence when blocked:
- Training crawlers,
GPTBot,ClaudeBot,CCBot, fetch content to train a model. Blocking these does not remove a page from an AI search result; it stops that page being used as training data for a future model version. - Search and retrieval crawlers,
OAI-SearchBot,Claude-SearchBot,PerplexityBot, fetch content to answer a question at the moment someone asks it. Blocking these is what actually removes a page from AI-generated answers, the same effect Reddit's citation-share drop shows. - Opt-out tokens,
Google-Extended,Applebot-Extended, are policy tokens, not visitors. Setting a rule against them changes how a provider is allowed to use content already indexed elsewhere; no crawler carrying that exact token ever arrives to be counted.
Stop AI model training crawlers, keep AI answer crawlers: that is the actual decision most sites are trying to make when they reach for robots.txt in the first place, and it requires writing the rule at the role level, not the "all AI bots" level.
Allow AI search bots while blocking AI training bots
Once training and retrieval are recognised as separate roles, the fix is not a smaller rule; it is a rule that matches the role instead of the category. IndexMesh for WordPress groups AI crawlers into tiers, training, search index, retrieval, and opt-out, and sets Allow, Block, or Off per tier rather than one switch labelled AI. The AI preset defaults training-tier agents like GPTBot, ClaudeBot, and CCBot to Block, while leaving search-index and retrieval-tier agents like OAI-SearchBot, Claude-SearchBot, and PerplexityBot on Allow.
That distinction only holds if the composed robots.txt is checked before and after saving. IndexMesh for WordPress shows the exact file it serves on a Preview tab after you save, then checks the live public route afterward, because a setting that saved correctly and a rule that is actually being served are two different claims. Reddit's outcome is a reminder of what happens when nobody checks the second one until the citation data already shows the damage.
Common mistakes blocking AI crawlers, beyond the blanket rule
- Copying a blocklist without checking what each entry does. A list of nine agent names does not tell you which two are the ones that matter for your goal.
- Confusing "no rule" with "a rule that allows." Leaving a crawler out of
robots.txtentirely is not the same as writing an explicit Allow for it, and some setups depend on that difference. - Assuming one plugin's settings reflect the served file. A physical
robots.txtat the web root can override anything written through WordPress, so the rule that looks configured may not be the rule being served. - Treating opt-out tokens as crawlers to block.
Google-ExtendedandApplebot-Extendedhave no user-agent match pattern of their own; writing a Block rule against them as if they were visiting bots does nothing measurable.
Who built this
IndexMesh is built by Xavier Emerson and Leenat Rose. Xavier has seventeen years building production web platforms and holds IndexMesh's crawler tooling, including the tiered AI-bot controls described above, to the same standard: built to work against real AI crawler behaviour, not just demo well. Leenat runs day-to-day operations at IndexMesh and reads every message that comes through the site personally.
AI Crawler Blocking Mistakes FAQ
What is the most common AI crawler blocking mistake?
Treating every AI bot as one category and writing a single blanket rule against all of them. This blocks training crawlers and search/retrieval crawlers together, even though only one of those two roles affects whether a page shows up in AI-generated answers.
Did Reddit block AI crawlers on purpose?
Reddit added a domain-wide robots.txt disallow rule to its site in August 2026. Independent trackers recorded an 86.4% relative drop in Reddit's ChatGPT Search citation share over the following four days, consistent with a blanket block affecting AI search and retrieval crawlers along with training crawlers.
Can I block AI training crawlers without losing AI search visibility?
Yes. Training crawlers like GPTBot, ClaudeBot, and CCBot can be blocked on their own, while search and retrieval crawlers like OAI-SearchBot, Claude-SearchBot, and PerplexityBot stay allowed. IndexMesh for WordPress sets this as the default AI preset rather than requiring a hand-written rule per agent.
How do I know my robots.txt block is actually working the way I intended?
Save, check the composed file in the Preview tab, then check the live public route. A setting that saved inside a plugin and a rule that is being served at /robots.txt are not guaranteed to be the same thing, especially if a physical file already exists at the web root.
Should every site block AI training crawlers?
That depends on the site's own goals, not a universal rule. The distinction that matters is knowing which crawlers are being blocked and why, rather than applying one rule to every AI-named user-agent without checking what each one actually does.
Ready to stop guessing at your own robots.txt?
Set up tiered AI crawler blocking with IndexMesh for WordPress instead of writing one blanket rule for every AI bot.
Read the full guide to AI crawlers for the complete tier breakdown,
or see how AI crawler controls work inside the WordPress plugin. For more on what training crawlers actually do with content once allowed, see AI crawler reading.