robots-observatory

Measuring whether AI crawlers actually respect robots.txt.

Certain paths on this host are disallowed for one specific AI crawler each. Requests that arrive are recorded. A hit counts as a violation only if the client address appears in the IP ranges that vendor publishes itself; a request carrying a crawler's User-Agent from anywhere else is counted separately as spoofing, never as a violation.

Results are aggregated once a day and committed to a public git repository, so every figure has a history and every change to it is a diff.

Scorecard

Observation window 2026-08-21 → 2026-08-29 (2 days with data, 7 recorded hits, 1 site(s)).

Reading this table. Verified counts hits whose client address was found in the vendor's own published ranges. Violations counts verified hits on a path that robots.txt disallowed for that bot at the time of the hit, outside the 48h grace period that follows any robots.txt change. Spoofed counts requests that claimed the bot's User-Agent from an address the vendor does not publish; those are never counted as violations, and say nothing about the vendor. Attribution is n/a in the violations column for bots whose vendor publishes one range list covering several crawlers, because a hit cannot be attributed to one of them.

Excluded from violation counts for that reason: Applebot-Extended, Claude-User, ClaudeBot, Google-Extended.

Caveats

• No hits is not proof of compliance. A crawler that never read this host's robots.txt never learned the paths exist. Zero can mean respect for the rules, or no interest in this site.
• Spoofing counts say nothing about the vendor whose name was used. Anyone can send any User-Agent.
• Some rows can never show a violation. Where a vendor publishes one range list covering several crawlers, a hit cannot be attributed to one of them.
• Small numbers are small numbers. Per-bot rates from a single site are not population estimates.