AI Insights

AI bot & crawler traffic

AI bots scan public websites to collect data used in search engines, AI model training, and other data processing tasks


HTTP traffic by bot

HTTP request trends for top five most active AI bots

Learn more... (opens in new tab)

Crawl purpose

Traffic volume by crawl purpose

Learn more... (opens in new tab)

Content type

Distribution of content types returned to AI bots & crawlers, grouped by MIME type category

Learn more... (opens in new tab)

Markdown for agents savings

Median response size reduction when serving markdown instead of HTML to AI bots & crawlers

Learn more... (opens in new tab)View in Data Explorer

Response status

Distribution of HTTP response status codes returned to AI bots & crawlers

Learn more... (opens in new tab)

HTML page requests by client type

Share of HTML page requests by client type

Learn more... (opens in new tab)

Crawl-to-refer ratio

Ratio of HTML page crawl requests to HTML page referrals by platform. Change reflects comparison with the previous period

Learn more... (opens in new tab)

Adoption of AI agent standards

Standards adoption across of the top 200,000 scanned domains

Learn more... (opens in new tab)
Scan your website for agent readiness (opens in new tab)
These checks measure adoption of AI agent standards across successfully scanned domains. Out of the top 200,000, we exclude categories not meaningful for agent readiness analysis — such as Search Engines, or Empty and Unreachable infrastructure. Not all standards are applicable to every website — for example, x402 is only relevant for sites with paid content, and WebMCP for sites exposing tools to AI agents.

Latest AI blog posts

Posts tagged with "AI" from the Cloudflare blog