AI bot & crawler traffic
Copy linkAI bots scan public websites to collect data used in search engines, AI model training, and other data processing tasks
Latest AI blog posts
Copy linkPosts tagged with "AI" from the Cloudflare blog
Precursor, our new continuous behavioral validation engine for bot management, offers visibility into how humans and bots actually interact across the full user journey. By turning session-level behavior into bot detection signals, it identifies advanced automation with higher precision — while reducing friction for legitimate users.
Marina Elmore, Benedikt Wolters
Cloudflare’s new Attribution Business Insights dashboard helps website owners understand crawler behavior, appetite, and potential value, fueling business-level conversations around crawl compensation.
Jin-Hee Lee, Oliver Payne
Search is how we find nearly everything on the web — creators, merchants, answers. AI is rewriting the rules, leaving creators caught between staying discoverable in an agentic era and getting paid for their work. Today we're launching two initiatives to help.
Matthew Conroy