My FeedDiscussionsHeadless CMS
New
Sign in
Log inSign up
Learn more about Hashnode Headless CMSHashnode Headless CMS
Collaborate seamlessly with Hashnode Headless CMS for Enterprise.
Upgrade ✨Learn more

Best way to loop over large number of records in Rails?

Rails Reboot's photo
Rails Reboot
·Oct 14, 2016

So, I need to loop over some 90-100k records in rails and create an entry in another other table that these records affect. Currently, I am processing these records serially, which does not seem to scale up. MySQL is repeatedly timing out. So I am thinking of following approach:

  1. Spawn X sidekiq workers to process Y records each
  2. Each of these Y records is executed in batches(say Y/20)

Does this approach seems good enough or is there a better way?

Also, I am thinking to build a dashboard for recording the progress(and failure) for the above processing. How do I go about it?