Cloudflare AI Crawler Controls: Protect Content and Keep Search Access
A practical WebOps checklist for reviewing Cloudflare crawler settings, limiting AI training use and checking that important pages remain accessible to search.
A website can welcome search engines while setting limits on how its content is used for AI training. The difficulty is knowing which setting does which job. A broad crawler block can reach further than a business owner expects.
What changed in September 2026
On 15 September 2026, Cloudflare announced Disallow AI Training, a setting intended to keep search access available while expressing a refusal of training use. Its announcement also changed the meaning of broader blocking controls for mixed-use crawlers. These are crawlers that serve more than one purpose.
Cloudflare says selecting Block can now stop mixed-use crawlers including Googlebot, Bingbot and Applebot. Its Disallow AI Training option instead publishes applicable preferences through robots.txt while allowing qualifying mixed-use search crawlers. This is a policy control, not a promise of complete protection against every scraper.
Treat search, training and agents as separate decisions
For a business that relies on enquiries from organic search, our recommendation is to review crawler access as part of WebOps. Start with the commercial purpose of each page: should people discover it, should automated tools retrieve it, and are you comfortable with training use? Those answers need not be identical.
A public service page and a paid research library have different needs. For example, a hypothetical manufacturer may want its product specifications discoverable while restricting training use of detailed technical articles. That is a reason to examine granular controls rather than copy a blanket blocking rule.
Check these five things before changing access
1. Record the current configuration. Capture search, training and agent settings, relevant firewall rules, and the robots.txt response. Assign one person responsibility for the change and keep a rollback record.
2. Identify important public URLs. Include service pages, product pages, articles and landing pages that generate enquiries. Check representative URLs rather than assuming the homepage tells the whole story.
3. Review crawler activity. Cloudflare’s documentation describes a crawler table with request counts and outcomes. Unsuccessful requests can reflect other rules or response errors, so a failed request alone does not establish that the AI control caused it.
4. Apply the narrowest policy that meets the business goal. If search visibility matters, investigate Disallow AI Training before choosing a broad Training Block. Review the live dashboard and current documentation because older instructions may describe earlier behaviour.
5. Verify and monitor. Inspect the served robots.txt, test important pages with search-engine inspection tools, and review crawler failures after the change. Monitor indexing and qualified enquiries over time. A successful browser visit does not prove that a crawler can access the same page.
Know where the control stops
A robots.txt preference depends on the receiving operator’s support. Cloudflare’s September announcement explicitly says Bing’s domain-level robots.txt training support is targeted for early 2027; until then, this setting does not automatically convey that preference to Bing. Treat that date as a stated target, not a delivered feature.
Do not equate a training preference with removal from AI answers. Search retrieval, summaries and model training are separate uses, and controls vary by operator. Nor should you forecast licensing income merely because crawler charging tools exist. This review concerns access policy, not a revenue projection.
Make the review part of routine website care
A useful monthly review pairs technical evidence with business outcomes: crawler access to key URLs, indexing changes, organic landing-page performance and enquiry quality. Keep a dated change log so an unexpected traffic shift can be investigated against actual configuration changes.
Our analysis: the value of granular crawler controls is choice. Businesses can make a deliberate decision about training use without casually giving up search discovery. The operational work is checking that the chosen policy behaves as intended.
ElloSpaces can review crawler rules, indexing access and website maintenance as part of a WebOps engagement. If you need a clear record of what your site allows and why, contact manas@ellospaces.com.
Sources and verification
Researched on 24 October 2026. Product behaviour and operator commitments may change; check the linked primary sources before implementation.
Cloudflare: Disallow AI Training announcement, 15 September 2026