Nowadays, the hardware is increasingly parallel:
Using several cores and/or several nodes can often:
On the software side, this requires however to share the computational work among processes or threads.
Processes and threads are independent execution of code.
Each process or thread can be performed by a separate core, and can thus run in parallel.
The difference between them is that:
Thus, programming with threads is more flexible (e.g. two threads can simultaneously work on one same array) but also more more dangerous (e.g. two threads can simultaneously try to modify one given number in memory - in this case the result is undetermined).
For this reason, Python forbids threads to simultaneously execute Python code (this is known as the Global Interpreter Lock, or GIL). There are however exceptions: I/O operations and numpy number crunching release the GIL, and can thus be executed simultaneously.
There are many Python packages that handle parallel execution on processes or threads, both for general purpose (e.g. celery, ipyparallel, multiprocessing, ...) and more specialized purpose (e.g. dask, tensorflow, some numpy functions).
Here we will introduce two very different, and complementary, general-purpose packages: mpi4py and concurrent.futures.
mpi4py |
concurrent.futures |
|---|---|
| Can run on several nodes | Runs only on one node |
| Only handles processes | Can handle processes and threads |
| No interactivity | Some interactivity (e.g. integrates in Jupyter notebook) |
| Allows elaborate communications between processes | No communication between processes or threads |