Building a Web Crawler for Scraping Index Files from Common Crawl Server.

This is my first time hearing about common crawl, my mind couldn't wrap around everything written here, but it was an awesome read!