Discussion on "Deep Dive of the Distributed Job Scheduler that powers over 2 Billion daily jobs at PhonePe"

Snehasish Roy · 2024-02-17T19:26:36.170Z

While working at PhonePe, I had the privilege of working on Clockwork - the system that powers job scheduling across PhonePe. During an internal knowledge transfer session, I presented an architectural overview of the system to the team. After receiv...

Akshay Siddhpura

SSE at Walmart

Mar 1, 2024

Any thoughts on Kafka instead of RabbitMQ.? if we use Kafka instead of RabbitMQ few task can be avoided. for example 1) Kafka by default handle leader election and making sure no consumers are consuming from the same partition. In case some worker node goes down partition re-balance will also happen automatically.

2)Back pressure will also be handled automatically as per consumer speed.

putting this comment to understand why RabbitMQ was selected over other solutions?

Snehasish Roy

SDE at PhonePe | Ex {Google, Amazon, Goldman Sachs, Morgan Stanley} | ACM ICPC | IIIT Allahabad

Mar 1, 2024

You are correct about backpressure being handled automatically because of pull model instead of the push model by RabbitMQ.

But you are incorrect regarding the leader election. We dont push directly to RMQ. We enqueue first to HBase and only the eligible messages are pushed to RMQ. We still would need leader election while consuming from HBase.

Any messaging queue could have been used when eligible messages are found from HBase. Kafka is ops heavy. It requires a lot of operational overhead. RMQ is very light on operations. If your organization already has a Kafka cluster then it can be used.

anil verma

Feb 18, 2024

Snehasish Roy I appreciate the detailed explanations.

I have a question about the job extractor: what is the necessity of running it on each instance? Is it computationally intensive due to massive record? If we were to run it as a single job only, would we eliminate the need for the leader election component?

I'm curious about the challenges that drive the decision to go with this approach

If we were to run it as a single job only, would we eliminate the need for the leader election component?

Yes, but we would limit our throughput (the number of records fetched by an HBase scan in a given period) - If we run multiple concurrent Hbase scans, each without any overlap, then we will be able to scale our scans horizontally and serve the ever increasing traffic :)

Search Hashnode

Deep Dive of the Distributed Job Scheduler that powers over 2 Billion daily jobs at PhonePe

Responses