---
title: "Discussion on \"Design a web crawler\" | Hashnode"
description: "Discussion on \"Design a web crawler\". At the start we seed a few webpage which recursively add more pages from the links found on the seed pages.\nUses Cases\nUsed for search engine indexing, web archiving, web minining (scraping), web moni"
url: "https://hashnode.com/posts/design-a-web-crawler/6a8c7044409c822f9d092a95"
---

[Feed](https://hashnode.com/feed) /

Discussion

# [Design a web crawler](https://blog.danwald.me/design-a-web-crawler?utm_source=hashnode&utm_medium=feed)

At the start we seed a few webpage which recursively add more pages from the links found on the seed pages. Uses Cases Used for search engine indexing, web archiving, web minining (scraping), web moni

Aug 24 · 2 min read · blog.danwald.me  
Read the article

-   [#scalability](https://hashnode.com/tag/scalability)
-   [#backend-architecture](https://hashnode.com/tag/backend-architecture)
-   [#design](https://hashnode.com/tag/design)
-   [#distributed-systems](https://hashnode.com/tag/distributed-systems)

## Written by

[Danny Crasto](https://hashnode.com/@danwald)

@danwald · dubai, uae

  
[View profile](https://hashnode.com/@danwald)

backend/cloud engineer | mentor | pythonista

I am developer/code-reviewer/debugger/bug-fixer/architect/teacher/builder from dubai, uae

## Responses 4

Markdown supported · ⌘↵ to post

[Sanoary Istiaque](https://hashnode.com/@istyy)

[Aug 26](https://hashnode.com/posts/design-a-web-crawler/6a8c7044409c822f9d092a95/comment/6a8ebe08ee7679c5659c0908)

I really enjoyed this breakdown of designing a web crawler. I especially liked how it focuses on the practical engineering challenges rather than just explaining the basic concept. The discussion around crawling at scale, handling URLs, and managing resources makes the topic much easier to think about from a real-world perspective. Great read for anyone interested in web infrastructure and backend engineering.

[Debjyoti Roy](https://hashnode.com/@debjyoti-)

[Aug 26](https://hashnode.com/posts/design-a-web-crawler/6a8c7044409c822f9d092a95/comment/6a8eac9cbf7ea00fa1e734ae)

I was also working on this topic , this is helpful

[Nube Colectiva](https://hashnode.com/@nubecolectiva)

[Aug 25](https://hashnode.com/posts/design-a-web-crawler/6a8c7044409c822f9d092a95/comment/6a8ddddd37c5bc9bf447935e)

I'm researching this topic and your article is helpful, thank you 👍🏼

[OnlineProxy](https://hashnode.com/@onlineproxyio)

[Aug 25](https://hashnode.com/posts/design-a-web-crawler/6a8c7044409c822f9d092a95/comment/6a8db5e8b5c5c1ad37241350)

the 500tb a month number always bugs me in these writeups because nobody asks what happens after you dedup. if youre storing first copy only and hashing content youre not looking at raw page count times average size, youre looking at unique bodies times size times whatever replication factor you picked, and those two diverge fast once overlapping crawls hit the same articles or product pages from different urls. i've seen real pipelines where content hash dedup alone cuts raw ingest by 30 to 40 percent before compression even touches it. same deal with the qps number, 400 aggregate doesnt tell a worker how many requests it can throw at one host, you need a separate token bucket or delay per domain or you get banned by the first site with any rate limiting at all. treating those two numbers as clean inputs instead of things that need their own logic is where people get tripped up in interviews