On September 15, 2026, Cloudflare introduced “Disallow AI Training,” a new setting that enables website owners to block unauthorized content scraping by generative AI models while maintaining their visibility in search engine results. This feature resolves a long-standing dilemma where site operators were forced to sacrifice search discoverability to opt out of AI training due to traditional mixed-use crawlers.

Announcing “Disallow AI Training”: Rejecting AI Model Scrapes While Preserving Search Indexing

Cybersecurity leader Cloudflare officially rolled out its new “Disallow AI Training” capability on September 15, 2026 (US time), allowing website operators to opt out of further AI model training while preserving search engine indexing. Provided as part of Cloudflare’s Bot Management and AI Crawl Control suite, this setting addresses a critical problem: previously, when site owners blocked web crawlers to prevent their content from being absorbed as AI training data, they risked having their sites vanish from search engines like Google altogether. According to Cloudflare’s research, while fewer than 1% of websites actively block search engine crawlers, 17% have enabled controls to restrict or block AI model training. This new setting technically decouples these conflicting requirements, allowing publishers and creators to protect their intellectual property without sacrificing the organic search traffic essential for online discoverability.

The diagram below illustrates the new access control framework introduced by Cloudflare.

A New Access Control Model Categorizing Crawlers into Three Signals

Cloudflare’s new control framework replaces broad bot management by classifying web crawler traffic into three distinct signals based on behavior and purpose:

• Search: Crawling conducted to build and update search engine indexes.
• Training: Data collection conducted to train or fine-tune generative AI models.
• Agent: Human-directed or bot-assisted visits, such as chat fetch bots accessing pages on behalf of users.

When “Disallow AI Training” is toggled, Cloudflare automatically generates and synchronizes no-training directives in the site’s robots.txt file via Bot Preference Sync. This enables mixed-use crawlers—which perform both search indexing and model training—to continue indexing for search while selectively blocking model training scrapes. Furthermore, training-only crawlers operated by companies like Amazon, Anthropic, Meta, and OpenAI can be blocked individually without impacting search placement.

Tackling Mixed-Use Crawlers and Introducing the “Accountable” Industry Standard

The Dilemma of “Mixed-Use Crawlers” Troubling Website Operators

The primary obstacle preventing site owners from opting out of AI training while remaining indexed in search has been the presence of “mixed-use crawlers.” A mixed-use crawler is a single program that simultaneously gathers content for both search indexing and AI model training. According to Cloudflare’s network observations, mixed-use crawlers accounted for 36.6% of all crawler traffic, forming the single largest traffic category. Consequently, legacy IP blocking and standard bot mitigation tools (like “Block AI Bots”) caused search engine indexing to halt whenever AI scraping was blocked, dealing a severe blow to media outlets and business sites dependent on search traffic for ad revenue and user sign-ups.

Refer to the graph below for crawler traffic classification and the impact of blocking controls.

Requirements for the “Accountable” Standard Joined by Major Tech Players

To resolve the mixed-use crawler problem, Cloudflare introduced “Accountable,” a new designation requiring crawler operators to adhere to standards of transparency and responsibility. To qualify as Accountable, operators must satisfy or formally commit within a set timeframe to four key criteria:

• Respect explicit site preferences opting out of AI model training.
• Support options to opt out of content inclusion in AI Summaries.
• Provide URL-level transparency disclosing how crawled content is used for search versus training.
• Publicly declare that opting out of AI training will not penalize a site’s search rankings or visibility.

Currently, Apple (Applebot), Google (Googlebot), and Microsoft (Bingbot) have earned or qualified for the “Accountable” designation by meeting these criteria or presenting clear implementation roadmaps. Note that Microsoft’s full domain-level robots.txt support for Bingbot is slated for early 2027, during which time management continues via Bing Webmaster Tools and meta tags.

Impact on the Web Economy and the Future of AI Summary Controls

Significance for Ad Models and Media Outlets: Recommended Presets by Business Model

The rise of generative AI and zero-click searches has led to a steep decline in referral traffic from search engines. For instance, OpenAI’s scrape-to-visitor referral ratio escalated dramatically from 250:1 to 1,500:1 within just six months, highlighting a growing industry concern over uncompensated content extraction. Cloudflare’s granular controls offer practical guidance tailored to different web business models:

• Ad-supported websites: Allow Search, disallow Training, and block Agents on ad-serving pages.
• Non-ad, awareness-driven sites: Allow Search, Training, and Agents across all pages to maximize distribution.

Effective September 15, 2026, Cloudflare provides recommended preset configurations during onboarding for new domains based on monetization model, serving as an infrastructure-level solution to safeguard the value of digital content.

What’s Next: The Roadmap Toward AI Summary Opt-Outs

Cloudflare’s initiatives extend beyond AI model training to address “AI Summaries”—where AI engines answer user queries directly on search results pages, reducing click-throughs to original source websites. While opting out of AI summaries currently requires site-by-site configuration, Cloudflare aims to launch an integrated platform by early 2027 that will allow site owners to manage summary permissions in a single location. Additionally, Cloudflare is contributing to open international standards, such as the IETF’s “ai-prefs” initiative, enabling websites to express AI access preferences in a standardized, interoperable format.

The roadmap below outlines the upcoming rollout of these control features.

Moving from blunt bot blocking to granular choice and fair value exchange, Cloudflare’s latest initiative marks a pivotal milestone in preserving the sustainability of the open web and supporting the ecosystem for creators and publishers in the AI era.

Reference Pages

• [Have it both ways: stay discoverable in search while disallowing AI training | Cloudflare Blog] https://blog.cloudflare.com/accountable-mixed-use-ai-crawlers/

【#Cloudflare #AITrainingOptOut #SearchIndexing #WebCrawlers #DisallowAITraining #GenerativeAI #WebSecurity #robots】