1. INTEL CONFIDENTIAL Finding Parallelism Introduction to Parallel Programming – Part 3

2. Copyright © 2009, Intel Corporation. All rights reserved. Intel and the Intel logo are trademarks or registered trademarks of Intel Corporation or its subsidiaries in the United States or other countries. * Other brands and names are the property of their respective owners. Review & Objectives Previously: Described three methods of dividing independent work At the end of this part you should be able to: Identify opportunities for parallelism in code segments and applications 2

3. Copyright © 2009, Intel Corporation. All rights reserved. Intel and the Intel logo are trademarks or registered trademarks of Intel Corporation or its subsidiaries in the United States or other countries. * Other brands and names are the property of their respective owners. Where Can You Find Independent Tasks? If you are familiar with the application, you should have an idea about where independent computations can be found Is there a large data set involved? Are there portions of code that can execute in a different order? Is the computation a set of stages that don’t interact except for using the output of one as the input for the next? A more formal method of discovering parallelism uses dependence graphs 3

4. Copyright © 2009, Intel Corporation. All rights reserved. Intel and the Intel logo are trademarks or registered trademarks of Intel Corporation or its subsidiaries in the United States or other countries. * Other brands and names are the property of their respective owners. Dependence Graph Graph = (nodes, arrows) Node for each Variable assignment (except index variables) Constant Operator or function call Arrows indicate use of variables and constants Data flow Control flow 4

5. Copyright © 2009, Intel Corporation. All rights reserved. Intel and the Intel logo are trademarks or registered trademarks of Intel Corporation or its subsidiaries in the United States or other countries. * Other brands and names are the property of their respective owners. Dependence Graph Example #1 5 for (i = 0; i < 3; i++) a[i] = b[i] / 2.0; b[0]b[1]b[2] a[0]a[1]a[2] 222 ///

6. Copyright © 2009, Intel Corporation. All rights reserved. Intel and the Intel logo are trademarks or registered trademarks of Intel Corporation or its subsidiaries in the United States or other countries. * Other brands and names are the property of their respective owners. Dependence Graph Example #1 6 for (i = 0; i < 3; i++) a[i] = b[i] / 2.0; b[0]b[1]b[2] a[0]a[1]a[2] 222 /// Domain decomposition possible

7. Copyright © 2009, Intel Corporation. All rights reserved. Intel and the Intel logo are trademarks or registered trademarks of Intel Corporation or its subsidiaries in the United States or other countries. * Other brands and names are the property of their respective owners. Dependence Graph Example #2 7 for (i = 1; i < 4; i++) a[i] = a[i-1] * b[i]; b[1]b[2]b[3] a[1]a[2]a[3] a[0] ***

8. Copyright © 2009, Intel Corporation. All rights reserved. Intel and the Intel logo are trademarks or registered trademarks of Intel Corporation or its subsidiaries in the United States or other countries. * Other brands and names are the property of their respective owners. Dependence Graph Example #2 8 for (i = 1; i < 4; i++) a[i] = a[i-1] * b[i]; b[1]b[2]b[3] a[1]a[2]a[3] a[0] *** No domain decomposition

9. Copyright © 2009, Intel Corporation. All rights reserved. Intel and the Intel logo are trademarks or registered trademarks of Intel Corporation or its subsidiaries in the United States or other countries. * Other brands and names are the property of their respective owners. Dependence Graph Example #3 9 a = f(x, y, z); b = g(w, x); t = a + b; c = h(z); s = t / c; xwyz abc s / h()f()g() + t

10. Copyright © 2009, Intel Corporation. All rights reserved. Intel and the Intel logo are trademarks or registered trademarks of Intel Corporation or its subsidiaries in the United States or other countries. * Other brands and names are the property of their respective owners. Dependence Graph Example #3 10 a = f(x, y, z); b = g(w, x); t = a + b; c = h(z); s = t / c; xwyz abc s Task decomposition with 3 cores. / hfg + t

11. Copyright © 2009, Intel Corporation. All rights reserved. Intel and the Intel logo are trademarks or registered trademarks of Intel Corporation or its subsidiaries in the United States or other countries. * Other brands and names are the property of their respective owners. Dependence Graph Example #4 11 for (i = 0; i < 3; i++) a[i] = a[i] / 2.0; a[0]a[1]a[2] a[0]a[1]a[2] 222 ///

12. Copyright © 2009, Intel Corporation. All rights reserved. Intel and the Intel logo are trademarks or registered trademarks of Intel Corporation or its subsidiaries in the United States or other countries. * Other brands and names are the property of their respective owners. Dependence Graph Example #4 12 for (i = 0; i < 3; i++) a[i] = a[i] / 2.0; a[0]a[1]a[2] a[0]a[1]a[2] 222 /// Domain decomposition

13. Copyright © 2009, Intel Corporation. All rights reserved. Intel and the Intel logo are trademarks or registered trademarks of Intel Corporation or its subsidiaries in the United States or other countries. * Other brands and names are the property of their respective owners. Dependence Graph Example #5 13 for (i = 0; i < 3; i++) { a[i] = a[i] / 2.0; if (a[i] < 1.0) break; } a[0]a[1]a[2] a[0]a[2] 222 /// << 11 a[1]

14. Copyright © 2009, Intel Corporation. All rights reserved. Intel and the Intel logo are trademarks or registered trademarks of Intel Corporation or its subsidiaries in the United States or other countries. * Other brands and names are the property of their respective owners. Can You Find the Parallelism? Resizing a photo Searching a document for all instances of a word Updating a spreadsheet Compiling a program Prefetching pages in a Web browser Using a word processor to type a report 14

15. Copyright © 2009, Intel Corporation. All rights reserved. Intel and the Intel logo are trademarks or registered trademarks of Intel Corporation or its subsidiaries in the United States or other countries. * Other brands and names are the property of their respective owners. Opportunities for a Parallel Solution? Parallel Solution EasierParallel Solution More Difficult or Even Impossible Larger data setsSmaller data sets Dense matricesSparse matrices Dividing space among cores Dividing time among cores 15

16. Copyright © 2009, Intel Corporation. All rights reserved. Intel and the Intel logo are trademarks or registered trademarks of Intel Corporation or its subsidiaries in the United States or other countries. * Other brands and names are the property of their respective owners. References Richard H. Carver and Kuo-Chung Tai, Modern Multithreading: Implementing, Testing, and Debugging Java and C++/Pthreads/ Win32 Programs, Wiley-Interscience (2006). Robert L. Mitchell, “Decline of the Desktop,” Computerworld (September 26, 2005). Michael J. Quinn, Parallel Programming in C with MPI and OpenMP, McGraw-Hill (2004). Herb Sutter, “The Free Lunch is Over: A Fundamental Turn Toward Concurrency in Software,” Dr. Dobb’s Journal 30, 3 (March 2005). 16