Cloudflare Blocks AI Bots, Announces Pay Per Crawl

As anticipated by the CEO a few days ago, Cloudflare has announced new tools that allow publishers to prevent indiscriminate scraping of content by so-called crawlers or AI bots . Site owners can also use Pay per Crawl .

Cloudflare Blocks All AI Bots

Matthew Prince , CEO of Cloudflare, says that July 1 will be remembered as Content Independence Day . For nearly 30 years, there has been a reciprocal exchange between search engines and websites. The former index the latter's content and take users to the original source. Publishers then profit from advertising revenue based on the number of visitors.

This virtuous model no longer works. Today there are AI crawlers from OpenAI, Anthropic, Google and other companies that “scrape” text, images and videos to generate a response, but users do not visit the original source of the content. Revenue for the publisher decreases and must choose between closing the site or paywall (subscription).

Exactly one year ago, Cloudflare announced a tool that allows you to block all AI bots with a single click (at the end of March , a tool that tricks AI crawlers was introduced). Now the block is active by default .

For each new domain, you can decide whether to allow AI bots to access the content. In practice, AI companies must ask for explicit permission . Cloudflare also offers managed editing of the robots.txt file. It will be automatically created and updated by the Californian company. Blocking AI bots is also available only for parts of the site monetized with advertising.

The latest innovation is Pay per Crawl , available in private beta to a limited number of publishers. Publishers can set a price that AI companies must pay to access content. Publishers can agree on separate prices for each AI bot. In the future, it will be possible to use AI agents to choose dynamic prices (different based on the type of content or the number of users).