When you have to run long tasks like this you don't want to have your main server unresponsive while processing all this data. A good solution could be running these tasks on a separate worker. Put the tasks on a queue and let a worker handle it by chunks. If your tasks (or jobs) keep growing, you can span another worker to keep up with the load while your main web server is responsive.