This host measures whether AI crawlers respect robots.txt. Paths disallowed in /robots.txt are recorded and published as an open dataset.
Methodology and data