Free, unrestricted crawling is ending for a big chunk of the internet. Starting September 15, AI agent crawlers get blocked by default on ad-supported pages across Cloudflare’s network, and that covers a large share of the web’s traffic.
Cloudflare announced the shift on July 1. Most of the early coverage zeroed in on Google, but the real story is what this means for anyone building or relying on autonomous agents that fetch pages in real time.
What Changed With Cloudflare’s New Crawler Rules
Cloudflare used to offer one blunt switch: block AI bots or don’t. Now there are three categories, and each behaves differently.
Search bots index pages to answer questions later. Agent bots act in real time on behalf of a person, including ChatGPT’s fetch bot and browser-driving tools. Training bots pull content into a model’s weights. All three went live for every customer, including free-tier accounts, on July 1.
The defaults flip on September 15. Training and Agent traffic gets blocked on any page carrying ads. Search stays allowed. New domains, new sites under existing accounts, and every existing free-tier customer inherit these defaults automatically, whether they’ve thought about it or not.
Cloudflare’s reasoning is straightforward: an ad on a page signals it was built for a human visitor. A search bot that sends a reader back counts as a referral. An agent that reads the page and hands the answer to someone else, without a click, doesn’t.
The Google Complication
Googlebot crawls for search and training with the same bot. That means a site blocking Training under the strict setting also blocks Googlebot, and loses search visibility along with it. Cloudflare CEO Matthew Prince framed this as an incentive for crawlers to separate their search, agent, and training functions rather than lumping them together.
What Agent Builders and Publishers Should Actually Do
If you’re running agents, figure out now which of your bots will get classified as Agent-class. This isn’t something you opt into. A research agent browsing in real time gets caught by the rule whether its operator thinks of it as a “crawler” or not. Expect gaps in coverage rather than total failure, since only ad-supported pages get blocked.
Publishers face a different checklist. Confirm your account tier, because free-tier sites move to the new defaults automatically. Then weigh whether blocking Training is worth taking Googlebot down with it.
The Money Angle: Pay Per Use
Cloudflare’s Pay Per Crawl program is becoming Pay Per Use, with services like Ceramic.ai and You.com paying publishers when agents pull their content. Cloudflare estimates more than half of AI crawler traffic re-fetches pages that haven’t changed, so there’s waste worth pricing out on both sides.
Thirty years of open, unmetered web access just got a bill attached. Builders who sort out access before September have a manageable problem. Everyone else finds out from a 403 error.
For deeper context on how Cloudflare frames this shift, see Cloudflare’s official announcement. And if you’re auditing how your own site handles bot traffic and structured data, our SEO and AEO audit services can help you check where you stand before the September deadline hits.
