GPTBot, ClaudeBot, PerplexityBot: which AI crawlers can read your site
If your robots.txt has no Disallow lines that apply to them, every AI crawler can already read your site. The ones that matter are GPTBot and OAI-SearchBot for ChatGPT, ClaudeBot for Claude, Perplexit
visibility100x.hashnode.dev10 min read