Can ChatGPT, Claude and Perplexity crawl your site? What we found in 2,000 robots.txt filesOriginally published on oarsmanai.com. Written with AI assistance and checked by the Oarsman AI team; data collected by us on 1 Nov 2026 (CC BY 4.0). Can ChatGPT, Claude and Perplexity crawl your site1d ago · 9 min readoarsmanai.hashnode.dev