Cloudflare 将全面封禁 AI 机器人改为按搜索、训练和代理爬虫的精细控制

Cloudflare gives all customers new tools to control how AI bots access their websites. Since July 2024, Cloudflare customers have been able to block all AI crawlers with a single click . Now, instead of a blanket block, site owners can choose between three categories. Search covers search engine indexing. Training covers data collection for AI model training. Agent covers bots acting on behalf of users, like ChatGPT. Starting September 15, 2026, Training and Agent bots will be blocked by default on pages that carry ads, since advertising signals that a site wants human visitors. Search crawlers will still be allowed. Multi-purpose crawlers like Googlebot that both search and train will be treated according to whichever rule is strictest. The new options are available to users on the free plan as well. Ad Cloudflare is pushing bot operators to split their crawlers by purpose. Cloudflare CEO Matthew Prince has previously called out Google for bundling its search crawler with its AI crawler. In June 2026, Prince noted that bot traffic had surpassed human traffic on the internet for the first time, a milestone he originally didn't expect until late 2027 . Ad DEC_D_Incontent-1 For enterprise customers, Cloudflare is also launching BotBase , a searchable database of all known bots built right into the dashboard. It shows how each bot is classified and how it uses content, whether it just links to pages or reproduces them in full. Cloudflare is also changing the rules for verified bots . Previously, all verified bots were allowed through automatically. Going forward, a bot's category determines whether it gets access. The "Verified" label alone won't be enough anymore: Bot operators will need to prove they identify themselves honestly and don't abuse the access they're granted. Ad Subscribe to THE DECODER for ad-free reading, a weekly AI newsletter, our exclusive "AI Radar" frontier report six times a year, full archive access, and access to our comment section.