---
title: "Discussion on \"Deep Dive of the Distributed Job Scheduler that powers over 2 Billion daily jobs at PhonePe\" | Hashnode"
description: "Discussion on \"Deep Dive of the Distributed Job Scheduler that powers over 2 Billion daily jobs at PhonePe\". Architectural deep dive of Clockwork - the in-house distributed job scheduler at PhonePe"
url: "https://hashnode.com/posts/deep-dive-of-the-distributed-job-scheduler-that-powers-over-2-billion-daily-jobs-at-phonepe/65d1086c135bdc53dbd20e4f"
---

[Feed](https://hashnode.com/feed) /

Discussion

# [Deep Dive of the Distributed Job Scheduler that powers over 2 Billion daily jobs at PhonePe](https://snehasishroy.com/deep-dive-of-the-distributed-job-scheduler-that-powers-over-2-billion-daily-jobs-at-phonepe?utm_source=hashnode&utm_medium=feed)

Architectural deep dive of Clockwork - the in-house distributed job scheduler at PhonePe

Feb 17, 2024 · 10 min read · snehasishroy.com  
Updated Feb 19, 2024  
Read the article

-   [#system-architecture](https://hashnode.com/tag/system-architecture)
-   [#system-design](https://hashnode.com/tag/system-design)
-   [#distributed-system](https://hashnode.com/tag/distributed-system)
-   [#2articles1week](https://hashnode.com/tag/2articles1week)
-   [#databases](https://hashnode.com/tag/databases)
-   [#backend](https://hashnode.com/tag/backend)

## Written by

[Snehasish Roy](https://hashnode.com/@snehasishroy)

@snehasishroy · Bengaluru, India

  
[View profile](https://hashnode.com/@snehasishroy)

SDE at PhonePe | Ex {Google, Amazon, Goldman Sachs, Morgan Stanley} | ACM ICPC | IIIT Allahabad

Experienced Software developer with 10+ years of experience in Distributed Systems.

## Responses

Markdown supported · ⌘↵ to post

[Akshay Siddhpura](https://hashnode.com/@akshaysiddhpura)

[Mar 1, 2024](https://hashnode.com/posts/deep-dive-of-the-distributed-job-scheduler-that-powers-over-2-billion-daily-jobs-at-phonepe/65d1086c135bdc53dbd20e4f/comment/65e1b71ca651b604036b3e2f)

Any thoughts on Kafka instead of RabbitMQ.? if we use Kafka instead of RabbitMQ few task can be avoided. for example 1) Kafka by default handle leader election and making sure no consumers are consuming from the same partition. In case some worker node goes down partition re-balance will also happen automatically.

2)Back pressure will also be handled automatically as per consumer speed.

putting this comment to understand why RabbitMQ was selected over other solutions?

[Snehasish Roy](https://hashnode.com/@snehasishroy)

[Mar 1, 2024](https://hashnode.com/posts/deep-dive-of-the-distributed-job-scheduler-that-powers-over-2-billion-daily-jobs-at-phonepe/65d1086c135bdc53dbd20e4f/comment/65e1b71ca651b604036b3e2f#reply-65e230e596bbf62784846c99)

You are correct about backpressure being handled automatically because of pull model instead of the push model by RabbitMQ.

But you are incorrect regarding the leader election. We dont push directly to RMQ. We enqueue first to HBase and only the eligible messages are pushed to RMQ. We still would need leader election while consuming from HBase.

Any messaging queue could have been used when eligible messages are found from HBase. Kafka is ops heavy. It requires a lot of operational overhead. RMQ is very light on operations. If your organization already has a Kafka cluster then it can be used.

[anil verma](https://hashnode.com/@vermanil)

[Feb 18, 2024](https://hashnode.com/posts/deep-dive-of-the-distributed-job-scheduler-that-powers-over-2-billion-daily-jobs-at-phonepe/65d1086c135bdc53dbd20e4f/comment/65d1dfa50e54ec858f6b4c04)

[Snehasish Roy](https://hashnode.com/@snehasishroy) I appreciate the detailed explanations.

I have a question about the job extractor: what is the necessity of running it on each instance? Is it computationally intensive due to massive record? If we were to run it as a single job only, would we eliminate the need for the leader election component?

I'm curious about the challenges that drive the decision to go with this approach

[Snehasish Roy](https://hashnode.com/@snehasishroy)

[Feb 18, 2024](https://hashnode.com/posts/deep-dive-of-the-distributed-job-scheduler-that-powers-over-2-billion-daily-jobs-at-phonepe/65d1086c135bdc53dbd20e4f/comment/65d1dfa50e54ec858f6b4c04#reply-65d1e803d3d8be9e2568d74d)

> If we were to run it as a single job only, would we eliminate the need for the leader election component?

Yes, but we would limit our throughput (the number of records fetched by an HBase scan in a given period) - If we run multiple concurrent Hbase scans, each without any overlap, then we will be able to scale our scans horizontally and serve the ever increasing traffic :)
