Cloudflare, the web infrastructure company, recently launched “Pay per Crawl“, an experimental platform that allows publishers to charge for each visit made by an artificial intelligence bot to their websites. This initiative is a response from publishers to the Google AI Mode and other search engines such as OpenAI’s ChatGPT Search.

ALSO READ. How does Google AI Mode affect SEO? A study reveals its real impact on SEO

“Pay per Crawl” proposes a new business model for websites. Instead of allowing free crawling, site owners can set custom policies for each bot and decide whether to allow, block, or charge for that automated access.

The context is clear: the advertising model based on clicks and visits is collapsing in the face of the zero-click era. Conversational agents and generative AI interfaces, like the ones Google includes in its AI mode, present complete answers without requiring the user to visit the source site. This directly affects content monetization, decreases the value of appearing in search results, and radically redefines how SEO is done.

ALSO READ. How is Google’s AI mode different? This is how it works

Why do publishers see Google as a threat to their revenue?

The relationship between Google and publishers is unequal—let’s face it. But AI Mode creates a gap that seems insurmountable. Since May 2025, Google began to monetize AI Overviews by inserting ads into these AI-generated summaries. However, the publishers whose content trains and feeds the system receive no compensation for the use of their work. Unlike traditional featured snippets, AI Overviews rephrase information, which dilutes attribution and reduces referral traffic.

According to a study by Cloudflare, Google’s crawler accessed websites 14 times for every actual visit generated in June 2025. The scenario is even more concerning with OpenAI (1,700 crawls per click) and Anthropic (73,000 accesses per redirected visit). It’s traffic that doesn’t convert for publishers, yet still exploits their content without offering anything in return.

How does Pay per Crawl work?

With Pay per Crawl, Cloudflare proposes a scenario where crawling becomes a commercial transaction. Instead of simply optimizing for ranking, publishers can now condition access on payment by AI bots. It’s a paywall for bots.

This introduces a new logic: indexed content may have a cost per use, and AI algorithms must account for this barrier when training or generating responses. The platform allows:

• Setting prices for bot access.
• Applying specific rules per bot (e.g., allow Google Search but charge OpenAI).
• Managing the collection and distribution of micropayments directly through Cloudflare.

Publishers and major media outlets support Cloudflare

Cloudflare’s proposal isn’t standing alone. Major media outlets like TIME, Condé Nast, Fortune, The Atlantic, and Associated Press already block AI crawlers through Cloudflare’s infrastructure. While not all have confirmed active participation in the micropayment marketplace, the adoption of an explicit control model may reshape the new digital ecosystem.

Roger Lynch, CEO of Condé Nast, stated, “Cloudflare’s innovative approach to blocking AI crawlers is a game changer for publishers and sets a new standard for how online content is respected.”

From the tech sector, companies such as Pinterest, Quora, Reddit, and Stack Overflow also support this permission-based model. Reddit co-founder Steve Huffman summed up the industry’s sentiment: “Any platform on the web should have a say in who is using its content and for what.”

According to a statement released by Cloudflare, the following companies support the Pay per Crawl platform:

📰 Media outlets and publishers

1. Condé Nast
2. TIME
3. The Associated Press
4. The Atlantic
5. Fortune
6. BuzzFeed, Inc.
7. ADWEEK
8. The Arena Group
9. Atlas Obscura
10. Snopes.com
11. Groundviews.org
12. Half Baked Newsletter
13. Digital Content Next
14. DOC (Daily Online Content)
15. Raptive
16. PMC (Penske Media Corporation)
17. News/Media Alliance
18. Third Door Media
19. The Independent & Independent Media
20. SustainableMedia.Center

💡 Tech platforms, developers, and digital communities

1. Quora
2. Reddit
3. Pinterest
4. SourceForge
5. Stack Overflow
6. SimpleFeed
7. Linkup
8. Drupal & Acquia
9. EngineEars
10. Hyperscience
11. Webflow
12. Sovrn, Inc.

🎶 Other creative and specialized editorial sectors

1. Universal Music Group (UMG)
2. O’Reilly Media
3. StockTwits
4. ProRata AI
5. Evolve Media
6. IAB Tech Lab (Interactive Advertising Bureau)
7. International Center for Journalists (ICFJ+)

What legal challenges is Google facing over AI Mode?

While Cloudflare promotes a voluntary monetization model, Google is facing increasing regulatory pressure. On July 4, the Independent Publishers Alliance filed a formal complaint with the European Commission over unauthorized use of content in its AI Overviews.

The complaint argues that media outlets wanting to prevent Google’s AI from using their content must also sacrifice their visibility in traditional search rankings. This forces publishers into a corner: lose traffic or be exploited without compensation.

The complaint requests a temporary suspension of AI Overviews while their legality is reviewed. The rationale is that the economic damage is immediate and potentially irreversible.

For now, Google’s AI Mode has only been fully rolled out in the United States and India, where media outlets have reported over a 30% drop in CTR.

A new business model that will reshape SEO

The battle between Pay per Crawl and Google’s AI Mode is not just a clash of business models—it’s a confrontation between two opposing visions for the future of the internet:

• Cloudflare is betting on a web with permissions and fair compensation.
• Google continues to operate under a model of mass extraction and commercial use without direct payment.

Caught in the middle are publishers, SEO professionals, users, and the sustainability of the digital ecosystem itself.

What’s the difference between Google AI Mode and AI Overviews?

AI Overviews have become a widely used feature within Google Search. According to the company, over a billion people have used these summaries, which combine web results with AI-generated explanations.

However, AI Mode goes a step further. While AI Overviews appear in selected queries to provide a quick summary, AI Mode is a full AI-powered search environment where users can:

• Ask complex or multifaceted questions.
• Get reasoning and detailed comparisons.
• Explore topics in depth with follow-up questions.
• Receive helpful links and verified content.

In essence, AI Mode functions more like OpenAI’s ChatGPT.

👉 Follow us on Google News.