
Snehasish Roy I appreciate the detailed explanations.
I have a question about the job extractor: what is the necessity of running it on each instance? Is it computationally intensive due to massive record? If we were to run it as a single job only, would we eliminate the need for the leader election component?
I'm curious about the challenges that drive the decision to go with this approach
Akshay Siddhpura
SSE at Walmart
Any thoughts on Kafka instead of RabbitMQ.? if we use Kafka instead of RabbitMQ few task can be avoided. for example 1) Kafka by default handle leader election and making sure no consumers are consuming from the same partition. In case some worker node goes down partition re-balance will also happen automatically.
2)Back pressure will also be handled automatically as per consumer speed.
putting this comment to understand why RabbitMQ was selected over other solutions?