---
title: "Comment by OnlineProxy on \"How I Automated Scraping Jobs From 10+ ATS Platforms on Two 500MB Databases\" | Hashnode"
description: "maybe use the provider id as the key where the list response has one, since a title edit can change the slug and make the same job look like a close plus a new"
url: "https://hashnode.com/posts/how-i-automated-scraping-jobs-from-10-ats-platforms-on-two-500mb-databases/6ac37dbd278ec94067b69950/comment/6acb712836df3ef102232980"
---

[Discussion](https://hashnode.com/posts/how-i-automated-scraping-jobs-from-10-ats-platforms-on-two-500mb-databases/6ac37dbd278ec94067b69950)

Comment [How I Automated Scraping Jobs From 10+ ATS Platforms on Two 500MB Databases](https://hashnode.com/posts/how-i-automated-scraping-jobs-from-10-ats-platforms-on-two-500mb-databases/6ac37dbd278ec94067b69950)

[OnlineProxy](https://hashnode.com/@onlineproxyio)

[Oct 11](https://hashnode.com/posts/how-i-automated-scraping-jobs-from-10-ats-platforms-on-two-500mb-databases/6ac37dbd278ec94067b69950/comment/6acb712836df3ef102232980)

maybe use the provider id as the key where the list response has one, since a title edit can change the slug and make the same job look like a close plus a new listing. the url could stay as a fallback for the rest

do you cap the attempts on the 15 min embedding re-pick? if not, a row that keeps failing would hold one of the 30 slots per minute and slow down fresh jobs, and detail\_attempts parking might fit there too. for alerting the number of closes the guard holds back run to run could be a nice early signal that a platform is quietly misbehaving
