---
title: "Discussion on \"How I Automated Scraping Jobs From 10+ ATS Platforms on Two 500MB Databases\" | Hashnode"
description: "Discussion on \"How I Automated Scraping Jobs From 10+ ATS Platforms on Two 500MB Databases\". I wanted a job board that fills itself. I didn't want to enter postings by hand or pay for a job-data API. Most companies already publish their openings on an applicant tracking system (ATS) like Gree"
url: "https://hashnode.com/posts/how-i-automated-scraping-jobs-from-10-ats-platforms-on-two-500mb-databases/6ac37dbd278ec94067b69950"
---

[Feed](https://hashnode.com/feed) /

Discussion

# [How I Automated Scraping Jobs From 10+ ATS Platforms on Two 500MB Databases](https://devhubcommunity.hashnode.dev/how-i-automated-scraping-jobs-from-10-ats-platforms-on-two-500mb-databases?utm_source=hashnode&utm_medium=feed)

I wanted a job board that fills itself. I didn't want to enter postings by hand or pay for a job-data API. Most companies already publish their openings on an applicant tracking system (ATS) like Gree

Oct 5 · 20 min read · devhubcommunity.hashnode.dev  
Read the article

-   [#scraping](https://hashnode.com/tag/scraping)
-   [#github-actions-1](https://hashnode.com/tag/github-actions-1)
-   [#job-search](https://hashnode.com/tag/job-search)

## Written by

[Varun Kumawat](https://hashnode.com/@thisisvarun)

@thisisvarun · Kharagpur, India

  
[View profile](https://hashnode.com/@thisisvarun)

building a better Dev community

Developer. Founder, DevHub. Mentor.

## Responses 1

Markdown supported · ⌘↵ to post

[OnlineProxy](https://hashnode.com/@onlineproxyio)

[Oct 11](https://hashnode.com/posts/how-i-automated-scraping-jobs-from-10-ats-platforms-on-two-500mb-databases/6ac37dbd278ec94067b69950/comment/6acb712836df3ef102232980)

maybe use the provider id as the key where the list response has one, since a title edit can change the slug and make the same job look like a close plus a new listing. the url could stay as a fallback for the rest

do you cap the attempts on the 15 min embedding re-pick? if not, a row that keeps failing would hold one of the 30 slots per minute and slow down fresh jobs, and detail\_attempts parking might fit there too. for alerting the number of closes the guard holds back run to run could be a nice early signal that a platform is quietly misbehaving
