It's a pity that threading isn't useful in Python (not in CPython anyway).
Although it's worth emphasizing, like you briefly mentioned, that for quite some use cases, using processes that communicate through Pickling is a viable alternative:
from multiprocessing import Pool
deff(x):return x*x
if __name__ == '__main__':
with Pool(5) as p:
print(p.map(f, [1, 2, 3]))
What will happen is that it will pickle (Python's way of encoding almost all objects) the 1, 2, 3, send them to processes in the pool who do the computation, then pickle the results and send them back (preserving order).
Since they use processes instead of threads, they are unaffected by GIL.
However note that starting up the pool takes time (as it does in any language), and encoding/decoding takes time too (which is only necessary because of Python's GIL). So it's mostly useful for big calculations using little data.
It's super simple though, basically just put pool. before your map.
Yes, thanks for your sharing. However, if you are not using built-in data types, you might face problem pickling Python objects (especially datetime object). Another solution is to jsonify your object before pickling it.
There are indeed some objects that can't be pickled, but in new versions of Python in my experience it's quite workable. Standard datetimes for example can be pickled in recent versions. Lambdas can't, but functools.partial is a good replacement most times.
It's a pity that threading isn't useful in Python (not in CPython anyway).
Although it's worth emphasizing, like you briefly mentioned, that for quite some use cases, using processes that communicate through Pickling is a viable alternative:
docs.python.org/3/library/multiprocessing.html
from multiprocessing import Pool def f(x): return x*x if __name__ == '__main__': with Pool(5) as p: print(p.map(f, [1, 2, 3]))What will happen is that it will pickle (Python's way of encoding almost all objects) the 1, 2, 3, send them to processes in the pool who do the computation, then pickle the results and send them back (preserving order).
Since they use processes instead of threads, they are unaffected by GIL.
However note that starting up the pool takes time (as it does in any language), and encoding/decoding takes time too (which is only necessary because of Python's GIL). So it's mostly useful for big calculations using little data.
It's super simple though, basically just put
pool.before your map.