The AI block that also blocks Google

Published 1 September 2026. Updated 17 September 2026, after Cloudflare shipped a setting that removes the tradeoff this article described.

On 15 September 2026, Cloudflare changed what happens when a website says no to AI. It now sorts robots by the job they came to do instead of by whether they count as AI, and it offers three ways to refuse. Two of them, Block and "Block on pages with ads", now reach Googlebot, Bingbot and Applebot as well, because those three crawlers do two jobs at once. The third, Disallow AI Training, refuses the training and keeps the search. Which one you picked is the whole difference.

That setting is no longer a security decision. Which automated visitors your site admits now helps determine whether your products come up when somebody asks an assistant for a recommendation. The question belongs to whoever answers for traffic and revenue, and it currently lives in the infrastructure panel, not in the product.

Three jobs, not two kinds of robot

Cloudflare published the classification on 1 July 2026. Instead of asking whether a bot is AI, it asks what the bot came to do, and recognizes three answers. Search is "any behavior that collects or indexes your content, so it can answer questions about it later". Agent is automation "acting, usually in real time, on a person's behalf, to get something done right now": a chatbot fetching a page mid-conversation, an assistant driving a browser. Training is "a crawler taking your content to train or fine-tune a model".

Cloudflare's own measurement puts mixed-use crawlers, the ones blending search, agent work and training behind a single identifier, at more than 36% of crawler activity, and says 52% of crawler requests were for AI training as of June 2026, up from 22% in spring 2025. Cloudflare measures its own network and sells the products described here, which cuts both ways: best view of the problem, most to gain from you fixing it their way.

When one crawler does two of the three jobs, the blunt settings apply "the most restrictive applicable rules". Googlebot indexes for search and collects for training, so choosing Block, or Block on pages with ads, stops it reaching you at all. Cloudflare says so plainly: selecting Block "stops them entirely", search included.

Where you write the refusal decides what it costs

The same refusal, written in two different places, used to produce opposite results. That is the part that changed, and the way it changed is the interesting bit.

Google documents an opt-out token called Google-Extended, which controls whether crawled content trains Gemini and grounds its answers. The documentation states that Google-Extended "does not impact a site's inclusion in Google Search", and is equally explicit about why: the token "doesn't have a separate HTTP request user agent string. Crawling is done with existing Google user agent strings; the robots.txt user-agent token is used in a control capacity."

In robots.txt, the text file where a site states its preferences and trusts crawlers to honor them, a refusal is addressed to a purpose. At the edge of the network a refusal could only be addressed to a request, and a request carries a user agent, the name a program announces itself with, not an intention. Our read in September was that the polite mechanism was more precise than the enforced one. Cloudflare's fix is the evidence: Disallow AI Training works by publishing a Disallow directive in your robots.txt, and it is named after it. The network did not replace the request. It learned to write one.

There is no correct default

On 21 August 2026 Cloudflare shipped a feature that rewrites your robots.txt to match what you configured at the edge, and the example justifying it kills the idea of a right answer: "an e-commerce store may want everything crawled and trained on, so its products surface when a shopper asks a chatbot for 'the best sofa for a small apartment.' A publisher that monetizes pages with ads may want the opposite."

The obvious objection is about who holds the switch. Writing in Tech Policy Press in July 2025 about the previous round of this, Luke Hogg and Tim Hwang called it "a deeply troubling development" that one company controlling "over 82 percent of the global market for DDoS and bot protection software" was moving the default. Our position: right about the concentration, wrong about the alternative. Robots.txt was always a request, and a request enforced somewhere ends up enforced by whoever runs the network. What follows is choosing deliberately instead of inheriting.

What to do now

Nothing urgent, which is the correction to what this article first said. Cloudflare migrated existing customers on its own, and previous Training selections of Block or Block on pages with ads moved to Disallow AI Training. The work is to open the AI traffic configuration in Security and confirm the setting says what you mean, because the three options now cost different things. If you run a store, the honest answer may still be that you want everything crawling you, which lines up with what an assistant finds when it visits your site.

One limit belongs next to that, because it is the reason this is not finished. Cloudflare says Disallow AI Training does not yet convey a no-training preference to Bing through robots.txt: Microsoft is building that, targeted for early 2027. Apple, Google and Microsoft all state that refusing training does not affect search ranking, and Amazon, Anthropic, Meta and OpenAI already run separate search and training crawlers, so blocking their training never cost an index in the first place.

What we said to watch was whether Google would split Googlebot in two. It did not, and the dilemma dissolved anyway: rather than each operator splitting its crawler, Cloudflare built a designation, called it Accountable, and made honoring a purpose the price of keeping search access. The thing worth watching now is whether that holds without a standard behind it, or whether ai-prefs at the IETF turns a private designation into a public rule.

Sources

• Cloudflare, "Have it both ways: stay discoverable in search while disallowing AI training", 15 September 2026.
• Cloudflare, "Your site, your rules", 1 July 2026.
• Cloudflare, "Content Independence Day, one year on", 1 July 2026.
• Cloudflare, "Bot Preference Sync", 21 August 2026.
• Google Search Central, "Google crawlers", accessed 17 September 2026.
• Tech Policy Press, "Guardian to Gatekeeper", 9 July 2025.