Web Scraping at Scale: Concurrency, Rate Limits, and Backpressure

A web scraper that performs well across a few thousand pages can behave very differently when the workload grows to hundreds of thousands or millions of URLs. At smaller volumes, performance improveme