---
title: "Discussion on \"I checked 30 top websites for AI-crawler readiness. The ones ChatGPT cites most scored worst.\" | Hashnode"
description: "Discussion on \"I checked 30 top websites for AI-crawler readiness. The ones ChatGPT cites most scored worst.\". I pulled robots.txt, llms.txt and JSON-LD from 30 sites. Only 1 blocks GPTBot - and the most-cited sites are the least optimized."
url: "https://hashnode.com/posts/i-checked-30-top-websites-for-ai-crawler-readiness-the-ones-chatgpt-cites-most-scored-worst/6af52b98b9e8f367e65ec56a"
---

[Feed](https://hashnode.com/feed) /

Discussion

# [I checked 30 top websites for AI-crawler readiness. The ones ChatGPT cites most scored worst.](https://bidvix.hashnode.dev/i-checked-30-top-websites-for-ai-crawler-readiness-the-ones-chatgpt-cites-most-scored-worst?utm_source=hashnode&utm_medium=feed)

I pulled robots.txt, llms.txt and JSON-LD from 30 sites. Only 1 blocks GPTBot - and the most-cited sites are the least optimized.

5d ago · 5 min read · bidvix.hashnode.dev  
Read the article

-   [#ai](https://hashnode.com/tag/ai)
-   [#seo](https://hashnode.com/tag/seo)
-   [#marketing](https://hashnode.com/tag/marketing)

## Written by

[Ke Gao](https://hashnode.com/@dongdongdev)

@dongdongdev · Shanghai, China

  
[View profile](https://hashnode.com/@dongdongdev)

Building bidvix — AI visibility checks across ChatGPT, Gemini, Claude, Perplexity and DeepSeek.

Indie developer. I build bidvix, a tool that checks whether ChatGPT, Gemini, Claude, Perplexity and DeepSeek mention your brand — and which sources they cite when they do. Writing about AI visibility, retrieval and honest measurement.

## Responses 1

Markdown supported · ⌘↵ to post

[Aldo](https://hashnode.com/@aldoquantic)

[5d ago](https://hashnode.com/posts/i-checked-30-top-websites-for-ai-crawler-readiness-the-ones-chatgpt-cites-most-scored-worst/6af52b98b9e8f367e65ec56a/comment/6af5b89c7928a346d67e9ed4)

I would separate training permission from search eligibility in the conclusions. GPTBot and OAI-SearchBot have different jobs: blocking GPTBot is not a guarantee of zero ChatGPT search traffic. A browser-UA fetch of robots.txt also cannot establish what the CDN serves to the actual crawler.

For a follow-up, test representative cited URLs rather than only homepages, include OAI-SearchBot, and accept sitemap indexes as well as urlsets. Then measure citations on a fixed prompt set separately from those access checks. That would make the comparison much stronger without treating llms.txt as a ranking requirement.

I work on QuanticData, where we have an AI crawler checker. Keeping a rules check separate from evidence of a real bot visit or citation is an important distinction for this kind of audit.
