---
title: "Comment by Datablue on \"The state of web scraping in the AI era: still relevant or a relic?\" | Hashnode"
description: "The distinction between the input and output sides of the pipeline is a great way to frame this. LLMs have made the transformation and structuring of retrieved"
url: "https://hashnode.com/posts/the-state-of-web-scraping-in-the-ai-era/6a598262748724525af5b8cb/comment/6a845b39050c5ae1bd970caa"
---

[Discussion](https://hashnode.com/posts/the-state-of-web-scraping-in-the-ai-era/6a598262748724525af5b8cb)

Comment [The state of web scraping in the AI era: still relevant or a relic?](https://hashnode.com/posts/the-state-of-web-scraping-in-the-ai-era/6a598262748724525af5b8cb)

[Datablue](https://hashnode.com/@datablue)

[Aug 18](https://hashnode.com/posts/the-state-of-web-scraping-in-the-ai-era/6a598262748724525af5b8cb/comment/6a845b39050c5ae1bd970caa)

The distinction between the input and output sides of the pipeline is a great way to frame this. LLMs have made the transformation and structuring of retrieved content much easier, but reliably getting clean data from the web is still a separate engineering problem. I also like the Beautiful Soup vs. Scrapy rule of thumb prototype with the simpler tool, then move to a framework when scheduling, retries, and scale become part of the requirements.
