Cloudflare announced Bot Preference Sync, a feature that updates robots.txt based on AI bot configurations to control Search, Agent, and Training crawlers.

Key Details

• Bot Preference Sync synchronizes zone-level AI bot preferences with the website's robots.txt file.
• Available to all Cloudflare customers from Free tier to Enterprise.
• Prepends directives to existing robots.txt files to maintain user-configured rules.
• Sets default Disallow Training for ad-supported publishers to protect ad revenue while preserving search visibility.
• Requires mixed-use crawlers to meet transparency standards tracked in Cloudflare Radar.

In This Story