I am not sure if that works because as I said, I have to process and write in another table too.

Actually I was worried about GVL as these threads would not really be running in parallel. Another issue would be that sidekiq might shoot up RAM usage for redis this way. Though sidekiq status looks promising but that means I would need to save the job-id. I was thinking of checking the expected record and try to use polling that how many such records were created via an API

That is close to what I was looking for. Thanks!

Are you sure message queues are reliable enough? I was using sidekiq for queuing but was told by my senior that it might not be reliable or underlying redis server can crash.

That is exactly what I am doing right now. I also tried to use Sidekiq (asynchronous processing library in Ruby) but problem with that is the underlying redis server could crash given the time frame. I was wondering if there is some event driven approach(or a library) for this.