AI bot & crawler traffic
AI bots scan public websites to collect data used in search engines, AI model training, and other data processing tasks
HTTP traffic by bot
HTTP request trends for top five most active AI bots
Learn more... (opens in new tab)Crawl purpose
Traffic volume by crawl purpose
Learn more... (opens in new tab)Content type
Distribution of content types returned to AI bots & crawlers, grouped by MIME type category
Learn more... (opens in new tab)Markdown for agents savings
Median response size reduction when serving markdown instead of HTML to AI bots & crawlers
Learn more... (opens in new tab)View in Data ExplorerResponse status
Distribution of HTTP response status codes returned to AI bots & crawlers
Learn more... (opens in new tab)HTML page requests by client type
Share of HTML page requests by client type
Learn more... (opens in new tab)Crawl-to-refer ratio
Ratio of HTML page crawl requests to HTML page referrals by platform. Change reflects comparison with the previous period
Learn more... (opens in new tab)