The state of web scraping in the AI era: still relevant or a relic?

your breakdown is solid, but pay attention to how much boilerplate you're copy-pasting across projects. Most teams don't realize they've written retry logic like three different times until it blows up in their face, and by then the headache of maintaining beautiful soup soup usually costs more than just biting the bullet and learning scrapy upfront, especially once you've got multiple scrapers running in production. On the js rendering front, I'd pump the brakes a little, like before you go nuts with playwright, just crack open devtools and check the network tab for like thirty secons, tons of modern SPAs are literally leaking JSON through XHR calls that you can hit way faster and cheaper by just using regular requests. LLMs didn't kill scraping, they just made the downstream stuff easier, which honestly does justify spinning up more scraping pipelines than before. But most production scrapers are flying blind with basically zero instrumentation - no success rate tracking, no data quality checks, nothing on latency by domain - so your failures just silently trash your data pipeline, and that's where things get gnarly when you're juggling multiple crawlers running at the same time