---
title: "Discussion on \"Web Scraping at Scale With Scrapy and MongoDB\" | Hashnode"
description: "Discussion on \"Web Scraping at Scale With Scrapy and MongoDB\". A Complete End-to-End ETL Project\nIntroduction\nWeb scraping is often the first step in building real-world data pipelines. While simple scripts using requests and BeautifulSoup work for small tasks, they quickly break down when projects grow in size,..."
url: "https://hashnode.com/posts/web-scraping-at-scale-with-scrapy-and-mongodb/696274398fe80dba76cd1097"
---

[Feed](https://hashnode.com/feed) /

Discussion

# [Web Scraping at Scale With Scrapy and MongoDB](https://harshcodes.dev/web-scraping-at-scale-with-scrapy-and-mongodb?utm_source=hashnode&utm_medium=feed)

A Complete End-to-End ETL Project Introduction Web scraping is often the first step in building real-world data pipelines. While simple scripts using requests and BeautifulSoup work for small tasks, they quickly break down when projects grow in size,...

Jan 10 · 5 min read · harshcodes.dev  
Read the article

-   [#python](https://hashnode.com/tag/python)
-   [#webscraping](https://hashnode.com/tag/webscraping)
-   [#mongodb](https://hashnode.com/tag/mongodb)
-   [#scrapy](https://hashnode.com/tag/scrapy)
-   [#dataengineering](https://hashnode.com/tag/dataengineering)

## Written by

[Harsh Singh](https://hashnode.com/@harsh1507)

@harsh1507

  
[View profile](https://hashnode.com/@harsh1507)

Just a curious learner

## Responses

Markdown supported · ⌘↵ to post

No responses yet.
