---
title: "Comment by Aldo on \"I checked 30 top websites for AI-crawler readiness. The ones ChatGPT cites most scored worst.\" | Hashnode"
description: "I would separate training permission from search eligibility in the conclusions. GPTBot and OAI-SearchBot have different jobs: blocking GPTBot is not a guarante"
url: "https://hashnode.com/posts/i-checked-30-top-websites-for-ai-crawler-readiness-the-ones-chatgpt-cites-most-scored-worst/6af52b98b9e8f367e65ec56a/comment/6af5b89c7928a346d67e9ed4"
---

[Discussion](https://hashnode.com/posts/i-checked-30-top-websites-for-ai-crawler-readiness-the-ones-chatgpt-cites-most-scored-worst/6af52b98b9e8f367e65ec56a)

Comment [I checked 30 top websites for AI-crawler readiness. The ones ChatGPT cites most scored worst.](https://hashnode.com/posts/i-checked-30-top-websites-for-ai-crawler-readiness-the-ones-chatgpt-cites-most-scored-worst/6af52b98b9e8f367e65ec56a)

[Aldo](https://hashnode.com/@aldoquantic)

[5d ago](https://hashnode.com/posts/i-checked-30-top-websites-for-ai-crawler-readiness-the-ones-chatgpt-cites-most-scored-worst/6af52b98b9e8f367e65ec56a/comment/6af5b89c7928a346d67e9ed4)

I would separate training permission from search eligibility in the conclusions. GPTBot and OAI-SearchBot have different jobs: blocking GPTBot is not a guarantee of zero ChatGPT search traffic. A browser-UA fetch of robots.txt also cannot establish what the CDN serves to the actual crawler.

For a follow-up, test representative cited URLs rather than only homepages, include OAI-SearchBot, and accept sitemap indexes as well as urlsets. Then measure citations on a fixed prompt set separately from those access checks. That would make the comparison much stronger without treating llms.txt as a ranking requirement.

I work on QuanticData, where we have an AI crawler checker. Keeping a rules check separate from evidence of a real bot visit or citation is an important distinction for this kind of audit.
