Design a Web Crawler | System Design Question

A web crawler is the program behind every search engine: it starts from a handful of URLs, downloads those pages, pulls the links out of them, and follows those links to find more pages, over and over