Recognizing Potential Parallelism Intel Software College Introduction to Parallel Programming – Part 1

Copyright © 2006, Intel Corporation. All rights reserved. Intel and the Intel logo are trademarks or registered trademarks of Intel Corporation or its subsidiaries in the United States or other countries. *Other brands and names are the property of their respective owners. Intel ® Software College 2 Recognizing Potential Parallelism Objectives At the end of this module you should be able to: Define parallel computing Explain why parallel computing is becoming mainstream Explain why explicit parallel programming is necessary Identify opportunities for parallelism in code segments and applications

Copyright © 2006, Intel Corporation. All rights reserved. Intel and the Intel logo are trademarks or registered trademarks of Intel Corporation or its subsidiaries in the United States or other countries. *Other brands and names are the property of their respective owners. Intel ® Software College 3 Recognizing Potential Parallelism What Can We Do with Faster Computers? Solve problems faster Reduce turn-around time of big jobs Increase responsiveness of interactive apps Get better solutions in same amount of time Increase resolution of models Make model more sophisticated

Copyright © 2006, Intel Corporation. All rights reserved. Intel and the Intel logo are trademarks or registered trademarks of Intel Corporation or its subsidiaries in the United States or other countries. *Other brands and names are the property of their respective owners. Intel ® Software College 4 Recognizing Potential Parallelism What Is Parallel Computing? Attempt to speed solution of a particular task by 1. Dividing task into sub-tasks 2. Executing sub-tasks simultaneously on multiple processors Successful attempts require both 1. Understanding of where parallelism can be effective 2. Knowledge of how to design and implement good solutions

Copyright © 2006, Intel Corporation. All rights reserved. Intel and the Intel logo are trademarks or registered trademarks of Intel Corporation or its subsidiaries in the United States or other countries. *Other brands and names are the property of their respective owners. Intel ® Software College 5 Recognizing Potential Parallelism Why Parallel Computing? “The free lunch is over.” —Herb Sutter We want applications to execute faster Clock speeds no longer increasing exponentially 10 GHz 1 GHz 100 MHz 10 MHz 1 MHz ’79 ’87 ’95 ’03 ’11

Copyright © 2006, Intel Corporation. All rights reserved. Intel and the Intel logo are trademarks or registered trademarks of Intel Corporation or its subsidiaries in the United States or other countries. *Other brands and names are the property of their respective owners. Intel ® Software College 6 Recognizing Potential Parallelism Clock Speeds Have Flattened Out Problems caused by higher speeds Excessive power consumption Heat dissipation Current leakage Power consumption critical for mobile devices Mobile computing platforms increasingly important Retail laptop sales now exceed desktop sales Laptops may be 35% of PC market in 2007

Copyright © 2006, Intel Corporation. All rights reserved. Intel and the Intel logo are trademarks or registered trademarks of Intel Corporation or its subsidiaries in the United States or other countries. *Other brands and names are the property of their respective owners. Intel ® Software College 7 Recognizing Potential Parallelism Execution Optimization Popular optimizations to increase CPU speed Instruction prefetching Instruction reordering Pipelined functional units Branch prediction Functional unit allocation Register allocation Hyperthreading Added sophistication  more silicon devoted to control hardware

Copyright © 2006, Intel Corporation. All rights reserved. Intel and the Intel logo are trademarks or registered trademarks of Intel Corporation or its subsidiaries in the United States or other countries. *Other brands and names are the property of their respective owners. Intel ® Software College 8 Recognizing Potential Parallelism Multi-core Architectures Potential performance = CPU speed  # of CPUs Strategy: Limit CPU speed and sophistication Put multiple CPUs (“cores”) on a single chip Potential performance the same

Copyright © 2006, Intel Corporation. All rights reserved. Intel and the Intel logo are trademarks or registered trademarks of Intel Corporation or its subsidiaries in the United States or other countries. *Other brands and names are the property of their respective owners. Intel ® Software College 9 Recognizing Potential Parallelism History of Parallel Computing, Part I Multiple-processor systems supporting parallel computing 1960s: Experimental systems 1980s: Microprocessor-based commercial systems

Copyright © 2006, Intel Corporation. All rights reserved. Intel and the Intel logo are trademarks or registered trademarks of Intel Corporation or its subsidiaries in the United States or other countries. *Other brands and names are the property of their respective owners. Intel ® Software College 10 Recognizing Potential Parallelism Old Dynamic of Parallel Computing Parallel computers are expensive There are not many parallel computers Most people do not learn parallel programming Parallel computing not mainstream Parallel programming is difficult Parallel programming environments are inadequate

Copyright © 2006, Intel Corporation. All rights reserved. Intel and the Intel logo are trademarks or registered trademarks of Intel Corporation or its subsidiaries in the United States or other countries. *Other brands and names are the property of their respective owners. Intel ® Software College 11 Recognizing Potential Parallelism Sequential Language Approach Problem has inherent parallelism Programming language cannot express parallelism Programmer hides parallelism in sequential constructs Compiler and/or hardware must find hidden parallelism Sadly, doesn’t work

Copyright © 2006, Intel Corporation. All rights reserved. Intel and the Intel logo are trademarks or registered trademarks of Intel Corporation or its subsidiaries in the United States or other countries. *Other brands and names are the property of their respective owners. Intel ® Software College 12 Recognizing Potential Parallelism Alternative Approach: Programmer and Compiler Work Together Problem has inherent parallelism Programmer has way to express parallelism Compiler translates program for multiple processors

Copyright © 2006, Intel Corporation. All rights reserved. Intel and the Intel logo are trademarks or registered trademarks of Intel Corporation or its subsidiaries in the United States or other countries. *Other brands and names are the property of their respective owners. Intel ® Software College 13 Recognizing Potential Parallelism Nothing Radical about a Programmer/Compiler Team Programmers of modern CPUs must take architecture and compiler into account in order to get peak performance “…you can actively reorganize data and algorithms to take advantage of architectural capabilities…” Introduction to Microarchitectural Optimization for Itanium® 2 Processors, p. 3

Copyright © 2006, Intel Corporation. All rights reserved. Intel and the Intel logo are trademarks or registered trademarks of Intel Corporation or its subsidiaries in the United States or other countries. *Other brands and names are the property of their respective owners. Intel ® Software College 14 Recognizing Potential Parallelism History of Parallel Computing, Part II 2004: Intel demos Montecito dual-core CPU 2006: Intel demos Clovertown quad-core CPU Clovertown scalable to 32+ cores in a single package

Copyright © 2006, Intel Corporation. All rights reserved. Intel and the Intel logo are trademarks or registered trademarks of Intel Corporation or its subsidiaries in the United States or other countries. *Other brands and names are the property of their respective owners. Intel ® Software College 15 Recognizing Potential Parallelism New Dynamic of Parallel Computing PCs are parallel computers Everyone has a parallel computer More people learning parallel programming Parallel programming considered mainstream Parallel programming gets easier Parallel programming environments improve

Copyright © 2006, Intel Corporation. All rights reserved. Intel and the Intel logo are trademarks or registered trademarks of Intel Corporation or its subsidiaries in the United States or other countries. *Other brands and names are the property of their respective owners. Intel ® Software College 16 Recognizing Potential Parallelism Methodology Study problem, sequential program, or code segment Look for opportunities for parallelism Try to keep all processors busy doing useful work

Copyright © 2006, Intel Corporation. All rights reserved. Intel and the Intel logo are trademarks or registered trademarks of Intel Corporation or its subsidiaries in the United States or other countries. *Other brands and names are the property of their respective owners. Intel ® Software College 17 Recognizing Potential Parallelism Ways of Exploiting Parallelism Domain decomposition Task decomposition Pipelining

Copyright © 2006, Intel Corporation. All rights reserved. Intel and the Intel logo are trademarks or registered trademarks of Intel Corporation or its subsidiaries in the United States or other countries. *Other brands and names are the property of their respective owners. Intel ® Software College 18 Recognizing Potential Parallelism Domain Decomposition First, decide how data elements should be divided among processors Second, decide which tasks each processor should be doing Example: Vector addition

Copyright © 2006, Intel Corporation. All rights reserved. Intel and the Intel logo are trademarks or registered trademarks of Intel Corporation or its subsidiaries in the United States or other countries. *Other brands and names are the property of their respective owners. Intel ® Software College 19 Recognizing Potential Parallelism Domain Decomposition Find the largest element of an array

Copyright © 2006, Intel Corporation. All rights reserved. Intel and the Intel logo are trademarks or registered trademarks of Intel Corporation or its subsidiaries in the United States or other countries. *Other brands and names are the property of their respective owners. Intel ® Software College 20 Recognizing Potential Parallelism Domain Decomposition Find the largest element of an array CPU 0CPU 1CPU 2CPU 3

Copyright © 2006, Intel Corporation. All rights reserved. Intel and the Intel logo are trademarks or registered trademarks of Intel Corporation or its subsidiaries in the United States or other countries. *Other brands and names are the property of their respective owners. Intel ® Software College 21 Recognizing Potential Parallelism Domain Decomposition Find the largest element of an array CPU 0CPU 1CPU 2CPU 3

Copyright © 2006, Intel Corporation. All rights reserved. Intel and the Intel logo are trademarks or registered trademarks of Intel Corporation or its subsidiaries in the United States or other countries. *Other brands and names are the property of their respective owners. Intel ® Software College 22 Recognizing Potential Parallelism Domain Decomposition Find the largest element of an array CPU 0CPU 1CPU 2CPU 3

Copyright © 2006, Intel Corporation. All rights reserved. Intel and the Intel logo are trademarks or registered trademarks of Intel Corporation or its subsidiaries in the United States or other countries. *Other brands and names are the property of their respective owners. Intel ® Software College 23 Recognizing Potential Parallelism Domain Decomposition Find the largest element of an array CPU 0CPU 1CPU 2CPU 3

Copyright © 2006, Intel Corporation. All rights reserved. Intel and the Intel logo are trademarks or registered trademarks of Intel Corporation or its subsidiaries in the United States or other countries. *Other brands and names are the property of their respective owners. Intel ® Software College 24 Recognizing Potential Parallelism Domain Decomposition Find the largest element of an array CPU 0CPU 1CPU 2CPU 3

Copyright © 2006, Intel Corporation. All rights reserved. Intel and the Intel logo are trademarks or registered trademarks of Intel Corporation or its subsidiaries in the United States or other countries. *Other brands and names are the property of their respective owners. Intel ® Software College 25 Recognizing Potential Parallelism Domain Decomposition Find the largest element of an array CPU 0CPU 1CPU 2CPU 3

Copyright © 2006, Intel Corporation. All rights reserved. Intel and the Intel logo are trademarks or registered trademarks of Intel Corporation or its subsidiaries in the United States or other countries. *Other brands and names are the property of their respective owners. Intel ® Software College 26 Recognizing Potential Parallelism Domain Decomposition Find the largest element of an array CPU 0CPU 1CPU 2CPU 3

Copyright © 2006, Intel Corporation. All rights reserved. Intel and the Intel logo are trademarks or registered trademarks of Intel Corporation or its subsidiaries in the United States or other countries. *Other brands and names are the property of their respective owners. Intel ® Software College 27 Recognizing Potential Parallelism Domain Decomposition Find the largest element of an array CPU 0CPU 1CPU 2CPU 3

Copyright © 2006, Intel Corporation. All rights reserved. Intel and the Intel logo are trademarks or registered trademarks of Intel Corporation or its subsidiaries in the United States or other countries. *Other brands and names are the property of their respective owners. Intel ® Software College 28 Recognizing Potential Parallelism Domain Decomposition Find the largest element of an array CPU 0CPU 1CPU 2CPU 3

Copyright © 2006, Intel Corporation. All rights reserved. Intel and the Intel logo are trademarks or registered trademarks of Intel Corporation or its subsidiaries in the United States or other countries. *Other brands and names are the property of their respective owners. Intel ® Software College 29 Recognizing Potential Parallelism Domain Decomposition Find the largest element of an array CPU 0CPU 1CPU 2CPU 3

Copyright © 2006, Intel Corporation. All rights reserved. Intel and the Intel logo are trademarks or registered trademarks of Intel Corporation or its subsidiaries in the United States or other countries. *Other brands and names are the property of their respective owners. Intel ® Software College 30 Recognizing Potential Parallelism Domain Decomposition Find the largest element of an array CPU 0CPU 1CPU 2CPU 3

Copyright © 2006, Intel Corporation. All rights reserved. Intel and the Intel logo are trademarks or registered trademarks of Intel Corporation or its subsidiaries in the United States or other countries. *Other brands and names are the property of their respective owners. Intel ® Software College 31 Recognizing Potential Parallelism Task (Functional) Decomposition First, divide tasks among processors Second, decide which data elements are going to be accessed (read and/or written) by which processors Example: Event-handler for GUI

Copyright © 2006, Intel Corporation. All rights reserved. Intel and the Intel logo are trademarks or registered trademarks of Intel Corporation or its subsidiaries in the United States or other countries. *Other brands and names are the property of their respective owners. Intel ® Software College 32 Recognizing Potential Parallelism Task Decomposition f() s() r() q() h() g()

Copyright © 2006, Intel Corporation. All rights reserved. Intel and the Intel logo are trademarks or registered trademarks of Intel Corporation or its subsidiaries in the United States or other countries. *Other brands and names are the property of their respective owners. Intel ® Software College 33 Recognizing Potential Parallelism Task Decomposition f() s() r() q() h() g() CPU 0 CPU 2 CPU 1

Copyright © 2006, Intel Corporation. All rights reserved. Intel and the Intel logo are trademarks or registered trademarks of Intel Corporation or its subsidiaries in the United States or other countries. *Other brands and names are the property of their respective owners. Intel ® Software College 34 Recognizing Potential Parallelism Task Decomposition f() s() r() q() h() g() CPU 0 CPU 2 CPU 1

Copyright © 2006, Intel Corporation. All rights reserved. Intel and the Intel logo are trademarks or registered trademarks of Intel Corporation or its subsidiaries in the United States or other countries. *Other brands and names are the property of their respective owners. Intel ® Software College 35 Recognizing Potential Parallelism Task Decomposition f() s() r() q() h() g() CPU 0 CPU 2 CPU 1

Copyright © 2006, Intel Corporation. All rights reserved. Intel and the Intel logo are trademarks or registered trademarks of Intel Corporation or its subsidiaries in the United States or other countries. *Other brands and names are the property of their respective owners. Intel ® Software College 36 Recognizing Potential Parallelism Task Decomposition f() s() r() q() h() g() CPU 0 CPU 2 CPU 1

Copyright © 2006, Intel Corporation. All rights reserved. Intel and the Intel logo are trademarks or registered trademarks of Intel Corporation or its subsidiaries in the United States or other countries. *Other brands and names are the property of their respective owners. Intel ® Software College 37 Recognizing Potential Parallelism Task Decomposition f() s() r() q() h() g() CPU 0 CPU 2 CPU 1

Copyright © 2006, Intel Corporation. All rights reserved. Intel and the Intel logo are trademarks or registered trademarks of Intel Corporation or its subsidiaries in the United States or other countries. *Other brands and names are the property of their respective owners. Intel ® Software College 38 Recognizing Potential Parallelism Pipelining Special kind of task decomposition “Assembly line” parallelism Example: 3D rendering in computer graphics RasterizeClipProjectModel Input Output

Copyright © 2006, Intel Corporation. All rights reserved. Intel and the Intel logo are trademarks or registered trademarks of Intel Corporation or its subsidiaries in the United States or other countries. *Other brands and names are the property of their respective owners. Intel ® Software College 39 Recognizing Potential Parallelism Processing One Data Set (Step 1) RasterizeClipProjectModel

Copyright © 2006, Intel Corporation. All rights reserved. Intel and the Intel logo are trademarks or registered trademarks of Intel Corporation or its subsidiaries in the United States or other countries. *Other brands and names are the property of their respective owners. Intel ® Software College 40 Recognizing Potential Parallelism Processing One Data Set (Step 2) RasterizeClipProjectModel

Copyright © 2006, Intel Corporation. All rights reserved. Intel and the Intel logo are trademarks or registered trademarks of Intel Corporation or its subsidiaries in the United States or other countries. *Other brands and names are the property of their respective owners. Intel ® Software College 41 Recognizing Potential Parallelism Processing One Data Set (Step 3) RasterizeClipProjectModel

Copyright © 2006, Intel Corporation. All rights reserved. Intel and the Intel logo are trademarks or registered trademarks of Intel Corporation or its subsidiaries in the United States or other countries. *Other brands and names are the property of their respective owners. Intel ® Software College 42 Recognizing Potential Parallelism Processing One Data Set (Step 4) RasterizeClipProjectModel The pipeline processes 1 data set in 4 steps

Copyright © 2006, Intel Corporation. All rights reserved. Intel and the Intel logo are trademarks or registered trademarks of Intel Corporation or its subsidiaries in the United States or other countries. *Other brands and names are the property of their respective owners. Intel ® Software College 43 Recognizing Potential Parallelism Processing Two Data Sets (Step 1) RasterizeClipProjectModel

Copyright © 2006, Intel Corporation. All rights reserved. Intel and the Intel logo are trademarks or registered trademarks of Intel Corporation or its subsidiaries in the United States or other countries. *Other brands and names are the property of their respective owners. Intel ® Software College 44 Recognizing Potential Parallelism Processing Two Data Sets (Time 2) RasterizeClipProjectModel

Copyright © 2006, Intel Corporation. All rights reserved. Intel and the Intel logo are trademarks or registered trademarks of Intel Corporation or its subsidiaries in the United States or other countries. *Other brands and names are the property of their respective owners. Intel ® Software College 45 Recognizing Potential Parallelism Processing Two Data Sets (Step 3) RasterizeClipProjectModel

Copyright © 2006, Intel Corporation. All rights reserved. Intel and the Intel logo are trademarks or registered trademarks of Intel Corporation or its subsidiaries in the United States or other countries. *Other brands and names are the property of their respective owners. Intel ® Software College 46 Recognizing Potential Parallelism Processing Two Data Sets (Step 4) RasterizeClipProjectModel

Copyright © 2006, Intel Corporation. All rights reserved. Intel and the Intel logo are trademarks or registered trademarks of Intel Corporation or its subsidiaries in the United States or other countries. *Other brands and names are the property of their respective owners. Intel ® Software College 47 Recognizing Potential Parallelism Processing Two Data Sets (Step 5) RasterizeClipProjectModel The pipeline processes 2 data sets in 5 steps

Copyright © 2006, Intel Corporation. All rights reserved. Intel and the Intel logo are trademarks or registered trademarks of Intel Corporation or its subsidiaries in the United States or other countries. *Other brands and names are the property of their respective owners. Intel ® Software College 48 Recognizing Potential Parallelism Pipelining Five Data Sets (Step 1) Data set 0 Data set 1 Data set 2 Data set 3 Data set 4 CPU 0CPU 1CPU 2CPU 3

Copyright © 2006, Intel Corporation. All rights reserved. Intel and the Intel logo are trademarks or registered trademarks of Intel Corporation or its subsidiaries in the United States or other countries. *Other brands and names are the property of their respective owners. Intel ® Software College 49 Recognizing Potential Parallelism Pipelining Five Data Sets (Step 2) Data set 0 Data set 1 Data set 2 Data set 3 Data set 4 CPU 0CPU 1CPU 2CPU 3

Copyright © 2006, Intel Corporation. All rights reserved. Intel and the Intel logo are trademarks or registered trademarks of Intel Corporation or its subsidiaries in the United States or other countries. *Other brands and names are the property of their respective owners. Intel ® Software College 50 Recognizing Potential Parallelism Pipelining Five Data Sets (Step 3) Data set 0 Data set 1 Data set 2 Data set 3 Data set 4 CPU 0CPU 1CPU 2CPU 3

Copyright © 2006, Intel Corporation. All rights reserved. Intel and the Intel logo are trademarks or registered trademarks of Intel Corporation or its subsidiaries in the United States or other countries. *Other brands and names are the property of their respective owners. Intel ® Software College 51 Recognizing Potential Parallelism Pipelining Five Data Sets (Step 4) Data set 0 Data set 1 Data set 2 Data set 3 Data set 4 CPU 0CPU 1CPU 2CPU 3

Copyright © 2006, Intel Corporation. All rights reserved. Intel and the Intel logo are trademarks or registered trademarks of Intel Corporation or its subsidiaries in the United States or other countries. *Other brands and names are the property of their respective owners. Intel ® Software College 52 Recognizing Potential Parallelism Pipelining Five Data Sets (Step 5) Data set 0 Data set 1 Data set 2 Data set 3 Data set 4 CPU 0CPU 1CPU 2CPU 3

Copyright © 2006, Intel Corporation. All rights reserved. Intel and the Intel logo are trademarks or registered trademarks of Intel Corporation or its subsidiaries in the United States or other countries. *Other brands and names are the property of their respective owners. Intel ® Software College 53 Recognizing Potential Parallelism Pipelining Five Data Sets (Step 6) Data set 0 Data set 1 Data set 2 Data set 3 Data set 4 CPU 0CPU 1CPU 2CPU 3

Copyright © 2006, Intel Corporation. All rights reserved. Intel and the Intel logo are trademarks or registered trademarks of Intel Corporation or its subsidiaries in the United States or other countries. *Other brands and names are the property of their respective owners. Intel ® Software College 54 Recognizing Potential Parallelism Pipelining Five Data Sets (Step 7) Data set 0 Data set 1 Data set 2 Data set 3 Data set 4 CPU 0CPU 1CPU 2CPU 3

Copyright © 2006, Intel Corporation. All rights reserved. Intel and the Intel logo are trademarks or registered trademarks of Intel Corporation or its subsidiaries in the United States or other countries. *Other brands and names are the property of their respective owners. Intel ® Software College 55 Recognizing Potential Parallelism Pipelining Five Data Sets (Step 8) Data set 0 Data set 1 Data set 2 Data set 3 Data set 4 CPU 0CPU 1CPU 2CPU 3

Copyright © 2006, Intel Corporation. All rights reserved. Intel and the Intel logo are trademarks or registered trademarks of Intel Corporation or its subsidiaries in the United States or other countries. *Other brands and names are the property of their respective owners. Intel ® Software College 56 Recognizing Potential Parallelism Dependence Graph Graph = (nodes, arrows) Node for each Variable assignment (except index variables) Constant Operator or function call Arrows indicate use of variables and constants Data flow Control flow

Copyright © 2006, Intel Corporation. All rights reserved. Intel and the Intel logo are trademarks or registered trademarks of Intel Corporation or its subsidiaries in the United States or other countries. *Other brands and names are the property of their respective owners. Intel ® Software College 57 Recognizing Potential Parallelism Dependence Graph Example #1 for (i = 0; i < 3; i++) a[i] = b[i] / 2.0; b[0]b[1]b[2] a[0]a[1]a[2] /// 222

Copyright © 2006, Intel Corporation. All rights reserved. Intel and the Intel logo are trademarks or registered trademarks of Intel Corporation or its subsidiaries in the United States or other countries. *Other brands and names are the property of their respective owners. Intel ® Software College 58 Recognizing Potential Parallelism Dependence Graph Example #1 for (i = 0; i < 3; i++) a[i] = b[i] / 2.0; b[0]b[1]b[2] a[0]a[1]a[2] /// 222 Domain decomposition possible

Copyright © 2006, Intel Corporation. All rights reserved. Intel and the Intel logo are trademarks or registered trademarks of Intel Corporation or its subsidiaries in the United States or other countries. *Other brands and names are the property of their respective owners. Intel ® Software College 59 Recognizing Potential Parallelism Dependence Graph Example #2 for (i = 1; i < 4; i++) a[i] = a[i-1] * b[i]; b[1]b[2]b[3] a[1]a[2]a[3] *** a[0]

Copyright © 2006, Intel Corporation. All rights reserved. Intel and the Intel logo are trademarks or registered trademarks of Intel Corporation or its subsidiaries in the United States or other countries. *Other brands and names are the property of their respective owners. Intel ® Software College 60 Recognizing Potential Parallelism Dependence Graph Example #2 for (i = 1; i < 4; i++) a[i] = a[i-1] * b[i]; b[1]b[2]b[3] a[1]a[2]a[3] *** a[0] No domain decomposition

Copyright © 2006, Intel Corporation. All rights reserved. Intel and the Intel logo are trademarks or registered trademarks of Intel Corporation or its subsidiaries in the United States or other countries. *Other brands and names are the property of their respective owners. Intel ® Software College 61 Recognizing Potential Parallelism Dependence Graph Example #3 a = f(x, y, z); b = g(w, x); t = a + b; c = h(z); s = t / c; x f wyz ab g t c s / h

Copyright © 2006, Intel Corporation. All rights reserved. Intel and the Intel logo are trademarks or registered trademarks of Intel Corporation or its subsidiaries in the United States or other countries. *Other brands and names are the property of their respective owners. Intel ® Software College 62 Recognizing Potential Parallelism Dependence Graph Example #3 a = f(x, y, z); b = g(w, x); t = a + b; c = h(z); s = t / c; x f wyz ab g t c s / h Task decomposition with 3 CPUs.

Copyright © 2006, Intel Corporation. All rights reserved. Intel and the Intel logo are trademarks or registered trademarks of Intel Corporation or its subsidiaries in the United States or other countries. *Other brands and names are the property of their respective owners. Intel ® Software College 63 Recognizing Potential Parallelism Dependence Graph Example #4 for (i = 0; i < 3; i++) a[i] = a[i] / 2.0; a[0]a[1]a[2] a[0]a[1]a[2] /// 222

Copyright © 2006, Intel Corporation. All rights reserved. Intel and the Intel logo are trademarks or registered trademarks of Intel Corporation or its subsidiaries in the United States or other countries. *Other brands and names are the property of their respective owners. Intel ® Software College 64 Recognizing Potential Parallelism Dependence Graph Example #4 for (i = 0; i < 3; i++) a[i] = a[i] / 2.0; a[0]a[1]a[2] a[0]a[1]a[2] /// 222 Domain decomposition

Copyright © 2006, Intel Corporation. All rights reserved. Intel and the Intel logo are trademarks or registered trademarks of Intel Corporation or its subsidiaries in the United States or other countries. *Other brands and names are the property of their respective owners. Intel ® Software College 65 Recognizing Potential Parallelism Dependence Graph Example #5 for (i = 0; i < 3; i++) { a[i] = a[i] / 2.0; if (a[i] < 1.0) break; } a[0]a[1]a[2] a[0]a[1]a[2] /// < 1 <

Copyright © 2006, Intel Corporation. All rights reserved. Intel and the Intel logo are trademarks or registered trademarks of Intel Corporation or its subsidiaries in the United States or other countries. *Other brands and names are the property of their respective owners. Intel ® Software College 66 Recognizing Potential Parallelism Can You Find the Parallelism? Resizing a photo Searching a document for all instances of a word Updating a spreadsheet Compiling a program Prefetching pages in a Web browser Using a word processor to type a report

Copyright © 2006, Intel Corporation. All rights reserved. Intel and the Intel logo are trademarks or registered trademarks of Intel Corporation or its subsidiaries in the United States or other countries. *Other brands and names are the property of their respective owners. Intel ® Software College 67 Recognizing Potential Parallelism Good/Bad Opportunities for a Parallel Solution Parallel Solution EasierParallel Solution More Difficult or Even Impossible Larger data setsSmaller data sets Dense matricesSparse matrices Dividing space among processors Dividing time among processors

Copyright © 2006, Intel Corporation. All rights reserved. Intel and the Intel logo are trademarks or registered trademarks of Intel Corporation or its subsidiaries in the United States or other countries. *Other brands and names are the property of their respective owners. Intel ® Software College 68 Recognizing Potential Parallelism Speculative Computation in a Turn- Based Strategy Game Make moves for distant AI-controlled countries in parallel

Copyright © 2006, Intel Corporation. All rights reserved. Intel and the Intel logo are trademarks or registered trademarks of Intel Corporation or its subsidiaries in the United States or other countries. *Other brands and names are the property of their respective owners. Intel ® Software College 69 Recognizing Potential Parallelism Risk: Unexpected Interaction

Copyright © 2006, Intel Corporation. All rights reserved. Intel and the Intel logo are trademarks or registered trademarks of Intel Corporation or its subsidiaries in the United States or other countries. *Other brands and names are the property of their respective owners. Intel ® Software College 70 Recognizing Potential Parallelism

Copyright © 2006, Intel Corporation. All rights reserved. Intel and the Intel logo are trademarks or registered trademarks of Intel Corporation or its subsidiaries in the United States or other countries. *Other brands and names are the property of their respective owners. Intel ® Software College 71 Recognizing Potential Parallelism Orange Cannot Move a Ship that Has Already Been Sunk by Green

Copyright © 2006, Intel Corporation. All rights reserved. Intel and the Intel logo are trademarks or registered trademarks of Intel Corporation or its subsidiaries in the United States or other countries. *Other brands and names are the property of their respective owners. Intel ® Software College 72 Recognizing Potential Parallelism Solution: Reverse Time Must be able to “undo” an erroneous, speculative computation Analogous to what is done in hardware after incorrect branch prediction Speculative computations typically do not have a big payoff in parallel computing

Copyright © 2006, Intel Corporation. All rights reserved. Intel and the Intel logo are trademarks or registered trademarks of Intel Corporation or its subsidiaries in the United States or other countries. *Other brands and names are the property of their respective owners. Intel ® Software College 73 Recognizing Potential Parallelism References Richard H. Carver and Kuo-Chung Tai, Modern Multithreading: Implementing, Testing, and Debugging Java and C++/Pthreads/ Win32 Programs, Wiley-Interscience (2006). Robert L. Mitchell, “Decline of the Desktop,” Computerworld (September 26, 2005). Michael J. Quinn, Parallel Programming in C with MPI and OpenMP, McGraw-Hill (2004). Herb Sutter, “The Free Lunch is Over: A Fundamental Turn Toward Concurrency in Software,” Dr. Dobb’s Journal 30, 3 (March 2005).

Copyright © 2006, Intel Corporation. All rights reserved. Intel and the Intel logo are trademarks or registered trademarks of Intel Corporation or its subsidiaries in the United States or other countries. *Other brands and names are the property of their respective owners. Intel ® Software College 74 Recognizing Potential Parallelism