So, I need to loop over some 90-100k records in rails and create an entry in another other table that these records affect. Currently, I am processing these records serially, which does not seem to scale up. MySQL is repeatedly timing out. So I am thinking of following approach:
Does this approach seems good enough or is there a better way?
Also, I am thinking to build a dashboard for recording the progress(and failure) for the above processing. How do I go about it?
Instead of batching in each worker, you can simply trigger a sidekiq worker for each batch and adjust sidekiq concurrency as per your machine specs. To signify batch boundaries prefer something other than limit+offset combination - eg. id range or date range (if indexed). If your ids are random/uuid/binary you may need to be more creative because you risk losing items that are inserted while the batch operations is underway.
If you are using sidekiq pro you can take advantage of its batch monitoring functionality.
For progress Sidekiq status looks promising.
Also instead of composing a SQL string through string concatenation consider using Arel to build a bulk insertion query or a higher level solution like activerecord-import.
Lorefnon
Open Web Enthusiast
Marcus
Full Stack Developer
You could implement some kind of search algorithm with a sorting algorithm. I have been meaning to write something for this but essentially make use of logarithmic functions to query a database. I know this is a challenge since database queries get in the way of actually writing functional searches. Possibly what you could do are unique searches on indexed items in your table and create an join a subtable and then search that table so you wont have any time out issues. Just an idea it might not even be a good one.