---
title: "Discussion on \"The state of web scraping in the AI era: still relevant or a relic?\" | Hashnode"
description: "Discussion on \"The state of web scraping in the AI era: still relevant or a relic?\". Why scraping didn't die when LLMs arrived, and how I decide between Beautiful Soup and Scrapy for a given job."
url: "https://hashnode.com/posts/the-state-of-web-scraping-in-the-ai-era/6a598262748724525af5b8cb"
---

[Feed](https://hashnode.com/feed) /

Discussion

# [The state of web scraping in the AI era: still relevant or a relic?](https://chandlerthompson.hashnode.dev/the-state-of-web-scraping-in-the-ai-era?utm_source=hashnode&utm_medium=feed)

Why scraping didn't die when LLMs arrived, and how I decide between Beautiful Soup and Scrapy for a given job.

Jul 17 · 4 min read · chandlerthompson.hashnode.dev  
Read the article

-   [#ai](https://hashnode.com/tag/ai)
-   [#web-scraping](https://hashnode.com/tag/web-scraping)
-   [#python](https://hashnode.com/tag/python)

## Written by

[Chandler Thompson](https://hashnode.com/@chandlerthompson)

@chandlerthompson · Austin, USA

  
[View profile](https://hashnode.com/@chandlerthompson)

Engineering Leader and Coach

I'm an engineering leader and coach based in Austin. I write about applied AI, developer tooling, and the parts of building software that don't fit in a pull request: judgment, team health, and deciding what deserves to exist. I care about shipping thoughtfully and helping engineers grow into the harder, less-technical half of the job. Read more at chandlerthompson.dev.

## Responses 2

Markdown supported · ⌘↵ to post

[Datablue](https://hashnode.com/@datablue)

[Aug 18](https://hashnode.com/posts/the-state-of-web-scraping-in-the-ai-era/6a598262748724525af5b8cb/comment/6a845b39050c5ae1bd970caa)

The distinction between the input and output sides of the pipeline is a great way to frame this. LLMs have made the transformation and structuring of retrieved content much easier, but reliably getting clean data from the web is still a separate engineering problem. I also like the Beautiful Soup vs. Scrapy rule of thumb prototype with the simpler tool, then move to a framework when scheduling, retries, and scale become part of the requirements.

[OnlineProxy](https://hashnode.com/@onlineproxyio)

[Jul 21](https://hashnode.com/posts/the-state-of-web-scraping-in-the-ai-era/6a598262748724525af5b8cb/comment/6a5ffec4411c410603b2acc5)

your breakdown is solid, but pay attention to how much boilerplate you're copy-pasting across projects. Most teams don't realize they've written retry logic like three different times until it blows up in their face, and by then the headache of maintaining beautiful soup soup usually costs more than just biting the bullet and learning scrapy upfront, especially once you've got multiple scrapers running in production. On the js rendering front, I'd pump the brakes a little, like before you go nuts with playwright, just crack open devtools and check the network tab for like thirty secons, tons of modern SPAs are literally leaking JSON through XHR calls that you can hit way faster and cheaper by just using regular requests. LLMs didn't kill scraping, they just made the downstream stuff easier, which honestly does justify spinning up more scraping pipelines than before. But most production scrapers are flying blind with basically zero instrumentation - no success rate tracking, no data quality checks, nothing on latency by domain - so your failures just silently trash your data pipeline, and that's where things get gnarly when you're juggling multiple crawlers running at the same time
