Yeah, fair on all of these. The 403s, you're right, that's just the server blocking my request, not an actual robots rule. A 4xx there basically means "no rules, go ahead," so lumping them in with real blocks was sloppy of me. Will reword it. WSJ is the one that actually bugs me. I only counted a block when a site named the bot and did Disallow: /. WSJ flips it: blocks everyone with User-agent: * and then allowlists a few. So my script saw "no named blocks" and called it open, when it's really the opposite. Means I'm undercounting all the default-deny sites, not just WSJ. Gonna rerun it to catch that and fix the numbers. 8 vs 9, yeah that's just a mistake. Text says eight, table has nine because I threw Amazonbot in and forgot to line them up. Will fix. News number is 20 out of 25 sites, so small sample, fair to want that called out. I'll add it. Good catches. Reworking it.
@mrhacker789
About
Founder of DataScrape Solutions. I build custom web scrapers and data pipelines for businesses. I write about web data, scraping, and what the public web really exposes.
Available for
Nothing here yet.