AI Insights

AI bot & crawler traffic

AI bots scan public websites to collect data used in search engines, AI model training, and other data processing tasks

HTTP traffic by bot

HTTP request trends for top five most active AI bots

Learn more...

Crawl purpose

Traffic volume by crawl purpose

Learn more...

Content type

Distribution of content types returned to AI bots & crawlers, grouped by MIME type category

Learn more...

Markdown for agents savings

Median response size reduction when serving markdown instead of HTML to AI bots & crawlers

Learn more...View in Data Explorer

Response status

Distribution of HTTP response status codes returned to AI bots & crawlers

Learn more...

HTML page requests by client type

Share of HTML page requests by client type

Learn more...

Crawl-to-refer ratio

Ratio of HTML page crawl requests to HTML page referrals by platform. Change reflects comparison with the previous period

Learn more...
Read the blog post

AI bot transparency

Tracking best practices from bots of leading AI models and service providers

Read the blog post

Generative AI services popularity

Top services based on 1.1.1.1 DNS resolver traffic

Learn more...

Adoption of AI agent standards

Standards adoption across of the top 200,000 scanned domains

Learn more...
Scan your website for agent readiness
These checks measure adoption of AI agent standards across successfully scanned domains. Out of the top 200,000, we exclude categories not meaningful for agent readiness analysis — such as Search Engines, or Empty and Unreachable infrastructure. Not all standards are applicable to every website — for example, x402 is only relevant for sites with paid content, and WebMCP for sites exposing tools to AI agents.

Workers AI model popularity

Distribution of AI model usage on Cloudflare Workers AI

Learn more...

Workers AI task popularity

Distribution of AI tasks performed on Cloudflare Workers AI

Learn more...

Latest AI blog posts

Posts tagged with "AI" from the Cloudflare blog

Cloudflare Blog