We read the robots.txt of 151 big websites. The syntax was fine. The problems were elsewhere.

On 29 October 2026 we fetched the robots.txt of 151 large websites (news, shops, SaaS, docs, social networks, education, SEO) and ran every file through a linter. 135 files could be read. The short ve