the dual dedup strategy is genuinely thoughtful, but a concrete example would make it click way harder. Like, imagine a paginated API spits back record IDs in one batch -- the second 6 gets caught by the seen-set and logged as duplicate within batch, while the completed-map catches 6 if it already succeeded last run. These two layers working together - that's the magic, but it deserves a walkthrough so people actually get why. Your exception handling is already there like timeouts and flaky stuff retry at the item level, process death triggers a full checkpoint, though teams would 100% benefit from explicit decision criteria they can actually apply to their own use case. The demo explains what it covers, but honestly numbering that checkpoint sequence explicitly like 1. read state, 2. fetch page N+1, etc. would let people see why this order mathematically guarantees zero duplicates instead of just crossing their fingers. One more thing tho, when your config fingerprint changes, say you're adding availability to an existing title+price extraction, the article correctly throws an error, but teams are left hanging. Do they spin up a new output directory, run some migration script on old records, or flip a force-resume flag?
the dual dedup strategy is genuinely thoughtful, but a concrete example would make it click way harder. Like, imagine a paginated API spits back record IDs in one batch -- the second 6 gets caught by the seen-set and logged as duplicate within batch, while the completed-map catches 6 if it already succeeded last run. These two layers working together - that's the magic, but it deserves a walkthrough so people actually get why. Your exception handling is already there like timeouts and flaky stuff retry at the item level, process death triggers a full checkpoint, though teams would 100% benefit from explicit decision criteria they can actually apply to their own use case. The demo explains what it covers, but honestly numbering that checkpoint sequence explicitly like 1. read state, 2. fetch page N+1, etc. would let people see why this order mathematically guarantees zero duplicates instead of just crossing their fingers. One more thing tho, when your config fingerprint changes, say you're adding availability to an existing title+price extraction, the article correctly throws an error, but teams are left hanging. Do they spin up a new output directory, run some migration script on old records, or flip a force-resume flag?