---
title: "Discussion on \"Who blocks AI crawlers? I checked the robots.txt of the top 1,000 sites\" | Hashnode"
description: "Discussion on \"Who blocks AI crawlers? I checked the robots.txt of the top 1,000 sites\". Which of the top 1,000 sites block AI crawlers like GPTBot and CCBot in robots.txt? News blocks them 4x more than everyone else."
url: "https://hashnode.com/posts/who-blocks-ai-crawlers-i-checked-the-robots-txt-of-the-top-1-000-sites/6abfceb6200ff04af34dad55"
---

[Feed](https://hashnode.com/feed) /

Discussion

# [Who blocks AI crawlers? I checked the robots.txt of the top 1,000 sites](https://datascrape.hashnode.dev/who-blocks-ai-crawlers-i-checked-the-robots-txt-of-the-top-1-000-sites?utm_source=hashnode&utm_medium=feed)

Which of the top 1,000 sites block AI crawlers like GPTBot and CCBot in robots.txt? News blocks them 4x more than everyone else.

Oct 2 · 5 min read · datascrape.hashnode.dev  
Read the article

-   [#web-scraping](https://hashnode.com/tag/web-scraping)
-   [#artificial-intelligence](https://hashnode.com/tag/artificial-intelligence)
-   [#seo](https://hashnode.com/tag/seo)
-   [#web-development](https://hashnode.com/tag/web-development)
-   [#data-science](https://hashnode.com/tag/data-science)

## Written by

[Mehul Jalondhara](https://hashnode.com/@mrhacker789)

@mrhacker789 · Bengaluru, India

  
[View profile](https://hashnode.com/@mrhacker789)

Founder of DataScrape Solutions. I build custom web scrapers and data pipelines for businesses. I write about web data, scraping, and what the public web really exposes.

## Responses 2

Markdown supported · ⌘↵ to post

[OnlineProxy](https://hashnode.com/@onlineproxyio)

[Oct 6](https://hashnode.com/posts/who-blocks-ai-crawlers-i-checked-the-robots-txt-of-the-top-1-000-sites/6abfceb6200ff04af34dad55/comment/6ac4e0b3cc43be88b145888c)

The 403s are probably just bot blocking and not a crawl policy, since a 4xx on robots.txt normally means no rules were found

I checked the WSJ file today and it looks pretty strict, it disallows everything by default and only allowlists a few agents, so counting only named blocks might undercount closed sites.

Small thing -- the text says 8 crawlers but the table has 9, and a sample size for the news numbers would be nice to see

[Mehul Jalondhara](https://hashnode.com/@mrhacker789)

[Oct 6](https://hashnode.com/posts/who-blocks-ai-crawlers-i-checked-the-robots-txt-of-the-top-1-000-sites/6abfceb6200ff04af34dad55/comment/6ac4e0b3cc43be88b145888c#reply-6ac4fd66d98e2a495a1d2063)

Yeah, fair on all of these.

The 403s, you're right, that's just the server blocking my request, not an actual robots rule. A 4xx there basically means "no rules, go ahead," so lumping them in with real blocks was sloppy of me. Will reword it.

WSJ is the one that actually bugs me. I only counted a block when a site named the bot and did Disallow: /. WSJ flips it: blocks everyone with User-agent: \* and then allowlists a few. So my script saw "no named blocks" and called it open, when it's really the opposite. Means I'm undercounting all the default-deny sites, not just WSJ. Gonna rerun it to catch that and fix the numbers.

8 vs 9, yeah that's just a mistake. Text says eight, table has nine because I threw Amazonbot in and forgot to line them up. Will fix.

News number is 20 out of 25 sites, so small sample, fair to want that called out. I'll add it.

Good catches. Reworking it.
