1. EN.540.635 - Software Carpentry
Python – An Introduction to Parallelization
Today I’ll be introducing a slightly advanced topic: Parallelization. It is an important concept to be familiar with in the field of computational science research. I’ll be going over some basic information that you guys might remember from the first few classes, just to refresh your memory

2. Central Processing Units (CPUs)
While you guys were design your own personal computers, almost all of you chose a powerful CPU and while doing so must have looked into the frequency the processor works at and of course the number of cores it had. It might seem obvious but the more cores the better. Since each core can technically work on 2 thigs at once, it leads to twice the number of threads.
AMD recently released its most powerful CPU: 24/48 and 32/64 processors
Intel i7 7700k
4 Cores
8 Threads
AMD Ryzen X
8 Cores
16 Threads

3. MultiCore Processors and HPC
Going on slight detour here, in my Windows device I can find out the details of my CPU by doing…
High performance Computing or HPC takes it to the next level by using considerable more than one CPU communicating information between each other to increase the speed of computation that would other wise take very long
Saini, Prerna & Bansal, Ankit & Sharma, Abhishek. (2015). Time Critical Multitasking for Multicore Microcontroller using XMOS Kit. International Journal of Embedded Systems and Applications /ijesa

4. Cores and Threads
All code is run serially in the Processing Unit of the CPU
Multithreading is when a single core tries parallelizing code.
Standard practice restricts multithreading to 2 threads per core
CPU
Core

5. Synchronous vs Asynchronous
Synchronous Returns to the caller after completing all its tasks/ finishes executing
Asynchronous
Returns to caller/main program immediately and keeps running in the background
Completed in same order that they are started. In this case the main program is locked until the individual processes finish. In asynchronous, there is no locking and therefore results in some mixed up results. In theses cases when the order is important we sometimes have to lock each process so that we get the results in the right order

6. Global Interpreter Lock
GIL prevents more than one thread to run the Python interpreter
This is to prevent writing to the same memory allocation
A workaround is to use multiple processes
Processes vs Threads
Process is an instance of a program and consists of multiple threads
Multithreading vs Multiprocessing
Apparently this a very heated topic in the programming community. People are really worked up about it on Stack Overflow
There is a lot of overhead in communicating especially in High Performance Computing

7. Message Passing Interface (MPI)
“Message-passing standard” that defines syntax and semantics to be used for writing message-passing programs
OpenMPI and MPICH are free implementations
Why do you need a message passing interface: So that if a particular process running in parallel requires data, it can communicate with another process and get the information it needs

8. Example: Molecular Dynamics

9. Mpi4py
It is a package that wraps MPI and allows message passing using an object-oriented approach
Think of it as an API for interaction with MPI
Run mpirun on just python script that has no imported mpi4py
Try broadcasting, ending and receiving data