Sign in
Log inSign up

Give me your Concurrent Programming advice

Todd's photo
Todd
·Oct 17, 2018

I'm diving deeper into concurrent programming (using multithreading to split up workloads in my case) and although I am tackling a few specific problems in Rust, I'm sure that there are a handful of common problems which arise from concurrent programming and typical methods to solve these problems.

Unfortunately, I've not gotten my hands on a good high-level concurrent programming book or course but could use some general advice on problems that you've solved in addition to guidance with this specific but probably very common scenario:

One such recurring theme for me has been the need to read from a buffered reader some given number of bytes at a time and process it. The data needs to be pulled from the same buffered reader by all threads... This could also be done by mapping a large area of memory and loading the entire file up and then splitting the file into pieces and giving each thread a piece to work on. But this for me so far has been easier said than done when it comes to such things as keeping track of the line numbers between threads, and properly splitting the workload.

For example, say for some reason I wanted to load a 2GB text file up, and line-by-line, run some algorithm on each line... Having one thread do this is pretty slow, but if I could fire up 8 threads and have the first thread work on the first eighth of the file, the second thread work on the second eighth, and so on, I could accomplish this task quicker... However, I've run into a few issues with this method and was wondering if anyone else had any advice for doing this type of thing. I've heard of such things as worker pools but never really gotten proper treatment. Thank you.