---
title: "Discussion on \"Design a Web Crawler | System Design Question\" | Hashnode"
description: "Discussion on \"Design a Web Crawler | System Design Question\". A web crawler is the program behind every search engine: it starts from a handful of URLs, downloads those pages, pulls the links out of them, and follows those links to find more pages, over and over"
url: "https://hashnode.com/posts/design-a-web-crawler-system-design-question/6a5dfb4218fb6065d93c8787"
---

[Feed](https://hashnode.com/feed) /

Discussion

# [Design a Web Crawler | System Design Question](https://tech.latencot.com/design-a-web-crawler-system-design-question?utm_source=hashnode&utm_medium=feed)

A web crawler is the program behind every search engine: it starts from a handful of URLs, downloads those pages, pulls the links out of them, and follows those links to find more pages, over and over

Jul 20 · 21 min read · tech.latencot.com  
Read the article

## Written by

[Sankalp pol](https://hashnode.com/@latencot)

@latencot

  
[View profile](https://hashnode.com/@latencot)

## Responses

Markdown supported · ⌘↵ to post

No responses yet.
