Comment by OnlineProxy on "Expensive Discovery, Cheap Enrichment: A Marketplace Scraper Architecture Built Around Rate Limits"

your caveat on run 4 is doing heavy lifting but that follow-up about monotonic degradation being unexplainable any other way still built on shaky ground because you've got too many variables mixed together. So overnight rest tanked things, but you genuinely can't tell if it's the IP getting reclassified, the datacenter being slammed at that hour, or some weird session state hanging around. one measurement where you bounce to a fresh datacenter IP mid-penalty would settle it once and for all. On the threshold thing your numbers tell the story -1700ms latency is still crushing it with 90% success, no rotation needed. But 7500ms is dead on arrival. Where rotation actually matters is gonna depend on the site, so it's worth testing each one instead of just saying do this everywhere. The two-tier model doesn't say discovery's always gonna be pricey, it's more like measure your specific host, divvy up the work based on what you find and tbh the avito thing backing that up means your method's holding up, not falling apart. You've already got room for this stuff. What'd make the writeup tighter, though: spell out a categorization of what kinds of targets you're actually dealing with. wildberries is totally sliceable coz you can measure and find the discovery bottleneck. Avito flips the script -- enrichment's the killer. Lazada just has a hard transport ceiling no matter how you architect it. You touch on Lazada not being a design thing, but if practitioners knew upfront which bucket a new target falls into, they could skip straight to the right measurement game plan

Fair hit on run 4 — the confound is real, and the clean experiment you describe (bounce to a fresh datacenter IP mid-penalty) is the one I skipped. Since publishing I got a partial answer from production instead: the actor now falls back to a different residential pool when search keeps 429-ing, and on the single night it actually triggered, the run finished on the fallback attempt after the original IP had been failing for four retries in a row. That leans toward IP-level state rather than datacenter load at that hour — but it's one night, not a controlled test.

On the categorization: agreed, that's the more useful framing than "discovery is expensive". What I'd write today, roughly — Wildberries: measurable bottleneck at discovery, sliceable; Avito: discovery is cheap (sitemaps do it), enrichment is the wall; Lazada: hard transport ceiling that architecture doesn't move. Deciding which bucket a new target lands in, before writing any code, deserves its own section. Thanks for the read.

Search Hashnode