Cloudflare Moves to Block AI Crawlers by Default
Cloudflare, a major player in internet infrastructure and content delivery, has announced a bold new policy aimed at restricting the unregulated scraping of web content by AI bots.
As of July 2025, AI crawlers will be automatically blocked from accessing any new website hosted on Cloudflare’s network — unless site owners explicitly allow access.
This change builds upon a 2023 feature that let publishers block AI bots with a single click. Now, that protection is on by default for all new websites.
Why Cloudflare Is Blocking AI Bots
Cloudflare CEO Matthew Prince stated that the decision is about restoring balance between content creators and AI developers:
“AI crawlers have been scraping content without limits. Our goal is to put the power back in the hands of creators while still helping AI companies innovate.”
He emphasised that the new framework will preserve the open internet while also offering fairer compensation models for original content publishers.
Introducing Cloudflare’s Pay-Per-Crawl Model
In a landmark shift, Cloudflare is also rolling out a “pay-per-crawl” model, allowing website owners to charge AI companies for access to their content. This model is expected to reshape how AI firms like OpenAI, Google, and others collect training data, since 16% of global internet traffic already passes through Cloudflare’s Content Delivery Network (CDN), according to its 2023 report.
What Are AI Crawlers and Why Are They Controversial?
AI crawlers are automated bots that extract large volumes of text, images, and other content from the web. This data is then used to train large language models (LLMs).
Unlike search engine bots, AI crawlers don’t drive traffic back to the content’s source. Instead, they use the content to generate responses in tools like chatbots and AI search engines, often without attribution or payment.
Cloudflare argues this undermines the open web by:
- Reducing traffic to the original publishers
- Cutting off advertising revenue
- Exploiting content without permission
OpenAI Responds: Pushback on Middleman Role
OpenAI, backed by Microsoft, has pushed back against the initiative. The company reportedly refused to participate, arguing that Cloudflare is inserting itself as a middleman between content creators and AI developers.
OpenAI noted that it already respects robots.txt protocols, which allow websites to block web crawlers, and sees no need for further intervention.
Legal Experts Weigh In: Will It Impact AI Development?
Matthew Holman, a Partner at U.K.-based law firm Cripps, told CNBC that the move could have major implications for the development of AI models: “AI crawlers are typically seen as more invasive and selective… If Cloudflare’s system works, it could restrict AI data gathering significantly.”
He added that the AI industry may experience short-term disruptions as access to fresh training data becomes more regulated.
What This Means for the Future of AI and Content Ownership
Cloudflare’s new policy marks a critical turning point in how content ownership is enforced online. As AI companies face increasing scrutiny over how they collect and use public data, pay-per-crawl models could become a new industry standard.
This shift could:
- Force AI companies to negotiate access terms
- Lead to more equitable data-sharing practices
- Slow the pace of LLM development unless compensated access is granted
Conclusion
By blocking AI bots by default and launching a pay-per-crawl model, Cloudflare is taking a strong stance to protect content creators and reshape the economics of the open internet. As the battle over data access heats up, this move may prompt broader industry-wide changes in how AI systems are trained and deployed.
