AS
Any thoughts on Kafka instead of RabbitMQ.? if we use Kafka instead of RabbitMQ few task can be avoided. for example 1) Kafka by default handle leader election and making sure no consumers are consuming from the same partition. In case some worker node goes down partition re-balance will also happen automatically. 2)Back pressure will also be handled automatically as per consumer speed. putting this comment to understand why RabbitMQ was selected over other solutions?
CommentArticleMar 1, 2024Deep Dive of the Distributed Job Scheduler that powers over 2 Billion daily jobs at PhonePe