Cloudflare Radar

Search

Search for locations, autonomous systems, reports, domains and more...

Language

English

Overview
Traffic
Bots
- Traffic
- Directory
Security
Connectivity
Routing
DNS
Tools
Reports
Resources

AI Insights

AI bot & crawler traffic

AI bots scan public websites to collect data used in search engines, AI model training, and other data processing tasks

Industry setIndustry sets are manually curated collections of related industries.

Select...

HTTP traffic by bot

HTTP request trends for top five most active AI bots

Crawl purpose

Select...

Crawl purpose

Traffic volume by crawl purpose

Content type

Distribution of content types returned to AI bots & crawlers, grouped by MIME type category

Markdown for agents savings

Median response size reduction when serving markdown instead of HTML to AI bots & crawlers

Learn more...View in Data Explorer

Response status

Distribution of HTTP response status codes returned to AI bots & crawlers

Group by

Status code

HTML page requests by client type

Share of HTML page requests by client type

Crawl-to-refer ratio

Ratio of HTML page crawl requests to HTML page referrals by platform. Change reflects comparison with the previous period

Read the blog post

AI bot transparency

Tracking best practices from bots of leading AI models and service providers

Generative AI services popularity

Top services based on 1.1.1.1 DNS resolver traffic

AI user agents found in robots.txt

robots.txt files found across the top 10,000 domains

Domain category

Select...

Directive

Disallow

Enforce robots.txt with AI Crawl Control

Adoption of AI agent standards

Standards adoption across of the top 200,000 scanned domains

Domain category

Select...

Scan your website for agent readiness

These checks measure adoption of AI agent standards across successfully scanned domains. Out of the top 200,000, we exclude categories not meaningful for agent readiness analysis — such as Search Engines, or Empty and Unreachable infrastructure. Not all standards are applicable to every website — for example, x402 is only relevant for sites with paid content, and WebMCP for sites exposing tools to AI agents.

Workers AI model popularity

Distribution of AI model usage on Cloudflare Workers AI

Workers AI task popularity

Distribution of AI tasks performed on Cloudflare Workers AI

© 2026 Cloudflare, Inc.

Privacy Policy Terms of Use Disclosure Trust & Safety Trademark Support