Once you crawl more than one domain I'd split retries by transport versus content. A 200 that is just an empty shell or a challenge page should not follow the same path as a timeout. If you already log how each provider behaves use that to pick the route rather than adding workers. When errors pile up on one ASN and go away on another path that is a network problem, not a capacity problem. Tracking that extra stuff is only worth it once you have enough volume. On dedup, stripping query params depends on what you're crawling. Dropping UTM junk is fine for a product catalog but the same rule will collapse different listing variants. So do a per site allowlist, not a global normalize everything. Per domain circuit breakers start mattering as soon as one slow host can starve the others, this is not a hundred million URL thing. If the threshold is too tight you will trip the breaker on sites that are just slow for real