Designing a test for the gap between a permission file and a server

How we designed a crawler access test: sampling frame, Googlebot control, scoring rules and limits, for sites whose robots.txt allows AI crawlers.