Parallelization: Area Under a Curve

AUC: An important task in science Neuroscience – Endocrine levels in the body over time Economics – Discounting: the fee incurred for delaying a payment over time Pharmacokinetics and clinical pharmacology, machine learning, medicine, psychiatry and psychology, chemistry, environmental science, fisheries and aquatic sciences, and many others

Newton and Leibniz Calculus Integration: exact AUC for simple curves Problems: – Doesn’t work for many curves, even simple ones – Doesn’t work in a reasonable number of steps Solution: approximation

Bernhard Riemann’s methods Domain of the function is segmented into the widths of shapes (rectangles in this case) Height of the rectangle is the y-value of the function for some x-value in the segment Sum of the areas of the rectangles is the approximate total area

The left Riemann sum Rectangle height = y-value of the left-most x- value of the rectangle width

How do we approximate? Small number of rectangles: by hand or calculator Beyond a small number of rectangles: single computer Even more: parallel processing More and more: cluster computer

Parallelism Concurrency (doing things at the same time) Multiple flows of execution (virtual entities that perform computations) working on the same problem Distributed Memory Shared Memory Hybrid

Flows of execution Processes – Distributed memory – Must communicate (message passing) Threads – Shared memory Processes with threads – Hybrid

Parallel hardware Multiple cores on a single compute node – Shared memory Multiple compute nodes sharing a network – Distributed memory Multiple compute nodes with multiple cores sharing a network – Hybrid

What are some standards? Message Passing Interface (MPI) – distributed memory/message passing OpenMP – shared memory MPI/OpenMP - hybrid

How to approach the parallel algorithm Identify data structures – constants and variables Determine the process and thread locality of the data structures Analyze the interactions of the data structures Lay out the algorithm (a step by step set of instructions)

Considerations Assume hybrid parallelism – Distributed memory, shared memory, or serial can be refined from hybrid Hybrid on 1 thread per process is just distributed memory Hybrid on 1 process is just shared memory Hybrid on 1 thread and 1 process is just serial The entire code is executed by each process Threads only execute code for which they are spawned

How do we visually represent the data structures?

How do we represent the data structures in writing?

What are the scopes of the data structures?

How do the data structures interact?

What is the order of interactions?

How does this look in pseudocode?

MPI functions

MPI_Reduce in more detail (Fortran only) Store any error codes in this variable

OpenMP regions