I am trying to create a job queue for sending emails , sms and image resizes . The API is built with Express and Node.js . I have looked at Kue , Bull , Step and RabbitMQ. Just wanted to know what are the defacto tool for creating a solution like this . What are the differences between worker queue and in app queue ?
depends on the scale. I last worked with kafka because the "at least once" delivery is important to that company. the downside you probably need go or java to have a continued support for this MQ.
others use something like zeroMQ for log processing or RabbitMQ for job scheduling.
I don't know what's an app queue, is but a worker queue is just an expression used for distributed systems.
your app pushes an task like "send email" into a queue and another "worker" machine picks up that entry and processes it asynchronous accordingly.
so you can have 12 workers using 1 queue to distribute the workload.
a queue is just a system and they have different channel systems or subscription systems.
maybe difference is the pipeline away from your application to the workers and the other way around ?
Sandeep Panda
co-founder, Hashnode
As @chilimatic said, depends on the scale. Redis based queues will work in most cases. However, I suggest RabbitMQ. This is what I use. As you are using Node.js, you may find amqplib interesting. It is an AMQP library and client.