1. This module created with support form NSF under grant # DUE 1141022 Module developed Fall 2014 by Apan Qasem Parallel Computing Fundamentals Course TBD Lecture TBD Term TBD

2. TXST TUES Module : # 16 MHz 25 MHz Full Speed Level 2 Cache instruction pipeline longer issue pipeline double speed arithmetic Why Study Parallel Computing? more responsibility on software Source : Scientific American 2005, “A Split at the Core” Extended by Apan Qasem

3. TXST TUES Module : # Why Study Parallel Computing? Parallelism is mainstream We are dedicating all of our future product development to multicore designs. … This is a sea change in computing Paul Otellini, President, Intel (2004) Requires fundamental change in almost all layers of abstraction Dirk Meyer CEO, AMD (2006) In future, all software will be parallel Andrew Chien, CTO, Intel (~2007)

4. TXST TUES Module : # Why Study Parallel Computing? Parallelism is Ubiquitous

5. TXST TUES Module : # Basic Computer Architecture image source: Gaddis, Starting our with C++

6. TXST TUES module # Program Execution Tuning Level int main() { int x, y, result; x = 17; y = 13; result = x + y; return result; } int main() { int x, y, result; x = 17; y = 13; result = x + y; return result; }.text.globl _main _main: LFB2: pushq%rbp LCFI0: movq%rsp, %rbp LCFI1: movl$17, -4(%rbp) movl$13, -8(%rbp) movl-8(%rbp), %eax addl-4(%rbp), %eax movl%eax, -12(%rbp) movl-12(%rbp), %eax leave ret.text.globl _main _main: LFB2: pushq%rbp LCFI0: movq%rsp, %rbp LCFI1: movl$17, -4(%rbp) movl$13, -8(%rbp) movl-8(%rbp), %eax addl-4(%rbp), %eax movl%eax, -12(%rbp) movl-12(%rbp), %eax leave ret compile code …11001010100… execute

7. TXST TUES module # Program Execution Tuning Level x = 17; y = 13; result = x + y; return result; Processor executes one instruction at a time* int main() { int x, y, result; x = 17; y = 13; result = x + y; return result; } int main() { int x, y, result; x = 17; y = 13; result = x + y; return result; } Instruction execution follows program order 10101010100 00001010100 11001110101 11001010100 code compile

8. TXST TUES module # Program Execution Tuning Level x = 17; y = 13; result = x + y; return result; int main() { int x, y, result; x = 17; y = 13; result = x + y; return result; } int main() { int x, y, result; x = 17; y = 13; result = x + y; return result; } code compile Parallel The two assignment statements x = 17; and y = 13; will execute in parallel

9. TXST TUES module # Parallel Program Execution x = 17;y = 13; result = x + y; return result; int main() { int x, y, result; x = 17; y = 13; result = x + y; return result; } int main() { int x, y, result; x = 17; y = 13; result = x + y; return result; } Cannot arbitrarily assign instructions to processors

10. TXST TUES Module : # Dependencies in Parallel Code If statement s i needs the value produced by statement s j then, s i is said to be dependent on s j All dependencies in the program must be preserved This means that if s i is dependent on s j then we need to ensure that s j completes execution before s i y = 17; … x = y + foo(); sisi sjsj responsibility lies with programmer (and software)

11. TXST TUES module # Running Quicksort in Parallel quickSort(values, pivotNew + 1, right); quickSort(values, left, pivotNew - 1); int quickSort(int values[], int left, int right) { if (left < right) { int pivot = (left + right)/2; int pivotNew = partition(values, left, right, pivot); quickSort(values, left, pivotNew - 1); quickSort(values, pivotNew + 1, right); } int quickSort(int values[], int left, int right) { if (left < right) { int pivot = (left + right)/2; int pivotNew = partition(values, left, right, pivot); quickSort(values, left, pivotNew - 1); quickSort(values, pivotNew + 1, right); }...partition()... To get benefit from parallelism want run “big chunks” of code in parallel Also need to balance the load

12. TXST TUES Module : # Parallel Programming Tools Most languages have extensions that support parallel programming A collection of APIs pthreads, OpenMP, MPI Java Threads Some APIs will perform some of the dependence checks for you Some languages specifically designed for parallel programming Cilk, Charm++, Chapel

13. TXST TUES Module : # Parallel Performance In the ideal case, the more processors we add to the system the higher the performance speedup If a sequential program runs in T seconds on one processor then the parallel program should run in T/N seconds on N processors In reality this almost never happens. Almost all parallel programs will have some parts that must run sequentially aka Amdahl’s Law Amount of speedup obtained is limited by the amount of parallelism available in the program Gene Amdahl image source : Wikipedia

14. TXST TUES module # max theoretical speedup max speedup in relation to number of processors Reality is often different!