been maintaining production scrapers for quiet a long now and the two things that actually decide whether a platform survives are how failed calls get billed and how fast you find out your pipeline broke. docs saying every chargeable call records one row basically means you pay regardless of outcome, and idempotency tokens only stop double billing on retries, so get 404s, timeouts and captcha blocks confirmed in writing before pointing retry loops at a defended target. That Zyte audit of around 11100 landing pages finding 40,6% had way more meaningful html after rendering is why i track ai_bot_blind_pct as a KPI, especially with cloudflare blocking training and agent crawlers on ad supported pages from sep 15. one thing to watch, the diff endpoint cuts reprocessing and alert noise but you still fetch every page to hash it so the target site sees the same load, you need conditional GETs with ETag and 304, or crawling driven by lastmod to actually pull less traffic