What's the benefit of using a messaging queue like RabitMQ or AWS SQS?
I read that individual components will be independently scalable if we use a message queue. I don't understand this statement. What's the downside of not using a Queue and doing operations inside a VM which handles HTTP requests?
In short, using HTTP does not guarantee message delivery, using AMQP (RabbitMQ) does give you message delivery guarantees and you can tweak how reliable you want it to be trading speed for reliability or vice versa.
HTTP is slow compared to AMQP (can you process a 1 million messages per minute via HTTP without effort? AMQP does that effortlessly on a small server).
RabbitMQ gives you clustering out of the box, so if you do hit its limit, you simply start up more instances of RabbitMQ and add them to the cluster.
Endpoint discovery is not needed, you only need to know where RabbitMQ is sitting and RabbitMQ will route your messages to the right application.
If your email application can no longer cope with the volume, simply start another email application and tell it to connect to the same queue, now you have double the email sending capacity, need more capacity? Start another application connected to the same queue.
Jan Vladimir Mostert
Idea Incubator
I'm not sure what Amazon queues are running under the bonnet, but I'm sure they designed it to have similar benefits to AMQP.