AI agent crawlers will require permission to access certain web pages starting September 15, 2023, according to an announcement from Cloudflare on July 1. This change affects automated bots that fetch real-time data for users, prominently impacting those who rely on these agents for competitive analysis and customer service.
Understanding the New AI Crawler Regulations
Cloudflare has introduced a new classification system for AI crawlers, replacing the previous single block-AI-bots switch with three categories: Search, Agent, and Training. The Search category allows bots to index pages for future inquiries, while Agent refers to real-time automated systems like ChatGPT's fetch bot. The Training category encompasses crawlers that gather content for AI model training.
From September 15, pages displaying ads will block access for both Agent and Training crawlers, while Search crawlers will remain permitted. This change applies to new Cloudflare domains and existing free-tier customers, who will automatically transition to the new defaults.
Implications for AI Agent Operators
Operators of AI agents must determine which of their Cloudflare accounts will be classified as Agent-class. The classification is behavioral, meaning that real-time browsing agents will be subject to the new rules regardless of the operator’s intentions. The primary concern for these operators is to avoid being blocked from accessing ad-supported pages, which are crucial for gathering news, pricing, and product information.
