Cloudflare Blocks Unpermitted AI Crawling, Introducing “Pay-Per-Crawl” Model

• New Default Blocking of AI Crawlers

  Starting July 2, 2025, all new domains registered with Cloudflare must declare whether they allow AI crawlers. The default is set to block them. This replaces the older reliance on robots.txt—a voluntary exclusion protocol often ignored by AI bots—with an enforceable infrastructure-level restriction. Cloudflare identifies scrapers using a dynamic database of known AI bots and applies these settings automatically for new customers and early adopters of its bot-blocking tools.

• Monetization Model: “Pay Per Crawl”

  Cloudflare’s “Pay Per Crawl” initiative allows participating publishers to set a monetary rate for AI firms to access their web content. Though still limited to a curated group of leading creators, this model introduces a framework for content licensing and financial compensation. AI companies can review pricing and determine whether to register or be denied access. Cloudflare says this model will help ensure quality content is used “with permission and compensation.”

• Global Internet Impact: 16–20% of Traffic

  Cloudflare now powers roughly one-fifth of the internet. With its infrastructure processing trillions of daily requests and managing more than 1 million sites with active bot protection, its enforcement of these rules could substantially alter AI companies’ access to training data across the web.

• AI Bots Generate 50+ Billion Requests Daily

  As of March 2025, Cloudflare reported that AI crawlers generate more than 50 billion requests per day across its network. This explosion in automated scraping—much of it unauthorized—has driven urgency for a more structured, enforceable access model.

• CEO Statement: Empowering Creators

  CEO Matthew Prince has reiterated that AI has broken the implicit contract between content creators and indexing bots. At the Axios Live event and in multiple statements, he stressed the need to protect creators and reinvent the web economy: “If the Internet is going to survive the age of AI, we need to give publishers the control they deserve.”

• Why AI Crawlers Are a Concern

  AI firms often ingest text, code, and images without compensation or attribution. This practice deprives creators of traffic, ad revenue, and control over how their work is reused. Unlike traditional search engines, generative AI tools serve content directly in answers—eliminating the need to visit original sites and creating a “zero-click” web environment.

• Support from Major Media and Tech Leaders

  The policy is strongly backed by top industry figures. Roger Lynch (Condé Nast) called it a “game-changer” that supports quality journalism. Leaders from TIME, BuzzFeed, The Atlantic, Pinterest, The Associated Press, and Reddit echoed that AI firms must be held accountable and publishers compensated for the content that powers AI systems.

• Permission-Based Crawling Goes Industry-Wide

  Over 1 million Cloudflare customers had already enabled the AI crawler blocker by mid-2025. This tool, once optional, is now automatically applied to new signups. AI bots must now authenticate themselves and declare their purpose—whether for training, inference, or search—allowing site owners to make informed decisions.

• AI Companies Can No Longer Stay Anonymous

  Cloudflare is co-developing a new open protocol to verify and authenticate crawlers. AI companies must disclose bot identity and intent, giving websites the ability to allow or deny access. For uncooperative bots, Cloudflare can redirect them into the “AI Labyrinth”—a web maze filled with meaningless AI-generated content, effectively neutralizing rogue crawlers.

• Backed by Leading Publishers and Platforms

  More than 50 content and tech organizations—including Sky News, The Associated Press, Quora, SourceForge, Universal Music Group, Fortune, and Stack Overflow—are supporting the new framework. They argue that this system strengthens the foundation for a sustainable web economy while encouraging AI innovation with ethical boundaries.

• Copyright and Legal Tensions Intensify

  The UK and US have seen escalating legal disputes over AI training. The BBC recently threatened legal action against Perplexity AI, while cultural figures like Sir Elton John and Baroness Beeban Kidron have called for systemic safeguards. Cloudflare’s infrastructure-led approach is praised as a major step—but experts like Ed Newton-Rex (Fairly Trained) note that legal frameworks remain essential for long-term enforcement.

• Parallel to Broader Content Battles

  Cloudflare’s initiative aligns with growing calls to defend digital ownership in an AI-driven economy. The BBC, TIME, and other publishers assert that AI firms are undermining traditional traffic-based monetization models. Cloudflare’s policy brings enforceability to an area that has lacked it—shifting the terms of data access from passive to active, consent-based interaction.

Why This Matters:

Cloudflare’s move represents a foundational shift in internet infrastructure policy: it repositions publishers as active gatekeepers of their own content. This default-blocking, monetization-enabled model changes the economic dynamics of AI training and enforces clear consent-based data collection. With backing from top media and tech entities, and mechanisms like bot identity verification and the “AI Labyrinth,” Cloudflare is creating a framework that promotes transparency, compensation, and sustainability in the age of AI-driven content extraction. While not a substitute for legal reform, it sets a powerful new industry baseline.