1. Parallel Algorithms on Networks of Processors Roy (Hutch) Pargas, PhD Computer Science (UNC Chapel Hill) School of Computing, Clemson University pargas@clemson.edu

2. Outline What are parallel algorithms? Why use them? Challenges for parallel algorithm designers Choosing a network Partitioning the data Designing the algorithm Example Recurrences (binary tree) Analysis (Speedup and Efficiency) Summary and Conclusions Roy Pargas, Clemson University pargas@clemson.edu July 30, 2011 50 Golden Years Ateneo Mathematics Program Quezon City, Philippines 3

3. Why Parallel Computation? Roy Pargas, Clemson University pargas@clemson.edu July 30, 2011 50 Golden Years Ateneo Mathematics Program Quezon City, Philippines 4

4. Why Parallel Computation? Roy Pargas, Clemson University pargas@clemson.edu July 30, 2011 50 Golden Years Ateneo Mathematics Program Quezon City, Philippines 5

5. Why Parallel Computation? Roy Pargas, Clemson University pargas@clemson.edu July 30, 2011 50 Golden Years Ateneo Mathematics Program Quezon City, Philippines 6

6. Why Parallel Computation? Roy Pargas, Clemson University pargas@clemson.edu July 30, 2011 50 Golden Years Ateneo Mathematics Program Quezon City, Philippines 7

7. Why Parallel Computation? Roy Pargas, Clemson University pargas@clemson.edu July 30, 2011 50 Golden Years Ateneo Mathematics Program Quezon City, Philippines 8

8. Why Parallel Computation? Roy Pargas, Clemson University pargas@clemson.edu July 30, 2011 50 Golden Years Ateneo Mathematics Program Quezon City, Philippines 9

9. New Processors Faster and Cheaper Roy Pargas, Clemson University pargas@clemson.edu July 30, 2011 50 Golden Years Ateneo Mathematics Program Quezon City, Philippines 10 January 2011

10. Partition the Data Roy Pargas, Clemson University pargas@clemson.edu July 30, 2011 50 Golden Years Ateneo Mathematics Program Quezon City, Philippines 11

11. Organize the Processors Roy Pargas, Clemson University pargas@clemson.edu July 30, 2011 50 Golden Years Ateneo Mathematics Program Quezon City, Philippines 12

12. Build a Network Roy Pargas, Clemson University pargas@clemson.edu July 30, 2011 50 Golden Years Ateneo Mathematics Program Quezon City, Philippines 13

13. Build a Network Roy Pargas, Clemson University pargas@clemson.edu July 30, 2011 50 Golden Years Ateneo Mathematics Program Quezon City, Philippines 14

14. Choosing a Network Roy Pargas, Clemson University pargas@clemson.edu July 30, 2011 50 Golden Years Ateneo Mathematics Program Quezon City, Philippines 15

15. Choosing a Network Roy Pargas, Clemson University pargas@clemson.edu July 30, 2011 50 Golden Years Ateneo Mathematics Program Quezon City, Philippines 16

16. Choosing a Network Roy Pargas, Clemson University pargas@clemson.edu July 30, 2011 50 Golden Years Ateneo Mathematics Program Quezon City, Philippines 17

17. Choosing a Network Roy Pargas, Clemson University pargas@clemson.edu July 30, 2011 50 Golden Years Ateneo Mathematics Program Quezon City, Philippines 18

18. Choosing a Network Roy Pargas, Clemson University pargas@clemson.edu July 30, 2011 50 Golden Years Ateneo Mathematics Program Quezon City, Philippines 19

19. Are There Really Any Multiprocessing Systems in Use Today? Roy Pargas, Clemson University pargas@clemson.edu July 30, 2011 50 Golden Years Ateneo Mathematics Program Quezon City, Philippines 20

20. Are There Really Any Multiprocessing Systems in Use Today? Roy Pargas, Clemson University pargas@clemson.edu July 30, 2011 50 Golden Years Ateneo Mathematics Program Quezon City, Philippines 21 Hamburg June 2011 Top 500

21. Supercomputers NEC/HP Tsubame (Japan) Roy Pargas, Clemson University pargas@clemson.edu July 30, 2011 50 Golden Years Ateneo Mathematics Program Quezon City, Philippines 22 1.192 petaflops 1.28 quadrillion floating point operations per sec 73,278 Xeon cores Infiniband grid network

22. Roy Pargas, Clemson University pargas@clemson.edu July 30, 2011 50 Golden Years Ateneo Mathematics Program Quezon City, Philippines 23 Supercomputers Dawning Nebulae (China) 1.27 petaflops 1.36 quadrillion floating point operations per sec 9280 Intel 6-core Xeon processors = 55,680 cores Infiniband grid network

23. Roy Pargas, Clemson University pargas@clemson.edu July 30, 2011 50 Golden Years Ateneo Mathematics Program Quezon City, Philippines 24 Supercomputers Cray Jaguar (USA) 1.75 petaflops 1.876 quadrillion floating point operations per sec 224,256 AMD cores 3D torus network

24. Roy Pargas, Clemson University pargas@clemson.edu July 30, 2011 50 Golden Years Ateneo Mathematics Program Quezon City, Philippines 25 Supercomputers NUDT Tianhe-1A (China) 2.566 petaflops 2.75 quadrillion floating point operations per sec 14336 CPUs Undisclosed proprietary network

25. Roy Pargas, Clemson University pargas@clemson.edu July 30, 2011 50 Golden Years Ateneo Mathematics Program Quezon City, Philippines 26 Supercomputers Fujitsu K (Japan) K = kei = Japanese for 10 quadrillion 8.162 petaflops 9 quadrillion floating point operations per sec 68,544 8-core SPARC64 processors = 548,352 cores 3-dimensional torus network called Tofu

26. Roy Pargas, Clemson University pargas@clemson.edu July 30, 2011 50 Golden Years Ateneo Mathematics Program Quezon City, Philippines 27 TOP500 Top 500 Computers in the World

27. Where Does that Leave Us? Roy Pargas, Clemson University pargas@clemson.edu July 30, 2011 50 Golden Years Ateneo Mathematics Program Quezon City, Philippines 28

28. Where Does that Leave Us? Roy Pargas, Clemson University pargas@clemson.edu July 30, 2011 50 Golden Years Ateneo Mathematics Program Quezon City, Philippines 29 In a wonderful playground of mathematical algorithmic design where imagination and creativity are key!

29. Where Does that Leave Us? In a wonderful playground of mathematical algorithmic design where imagination and creativity are key! Roy Pargas, Clemson University pargas@clemson.edu July 30, 2011 50 Golden Years Ateneo Mathematics Program Quezon City, Philippines 30

30. Where Does that Leave Us? In a wonderful playground of mathematical algorithmic design where imagination and creativity are key! Roy Pargas, Clemson University pargas@clemson.edu July 30, 2011 50 Golden Years Ateneo Mathematics Program Quezon City, Philippines 31

31. Where Does that Leave Us? In a wonderful playground of mathematical algorithmic design where imagination and creativity are key! Roy Pargas, Clemson University pargas@clemson.edu July 30, 2011 50 Golden Years Ateneo Mathematics Program Quezon City, Philippines 32

32. Challenges for Parallel Algorithm Designers Choosing a network Partitioning the problem Designing the parallel algorithm Roy Pargas, Clemson University pargas@clemson.edu July 30, 2011 50 Golden Years Ateneo Mathematics Program Quezon City, Philippines 33

33. So Lets Try It: Choosing a network Partitioning the problem Designing the parallel algorithm Roy Pargas, Clemson University pargas@clemson.edu July 30, 2011 50 Golden Years Ateneo Mathematics Program Quezon City, Philippines 34

34. So Lets Try It: Elliptic Partial Diff Eqns Choosing a network Partitioning the problem Designing the parallel algorithm Roy Pargas, Clemson University pargas@clemson.edu July 30, 2011 50 Golden Years Ateneo Mathematics Program Quezon City, Philippines 35

35. Elliptic PDEs Roy Pargas, Clemson University pargas@clemson.edu July 30, 2011 50 Golden Years Ateneo Mathematics Program Quezon City, Philippines 36 Problems involving second-order elliptic partial differential equations are equilibrium problems. Given a region R bounded by a curve C and that the unknown function z satisfies Laplaces or Poissons equation in R, the objective is to approximate the value of z at any point in R. The method of finite differences is an often used numerical method for solving this problem. The basic strategy is to approximate the differential equation by a difference equation and to solve the difference equation.

36. Designing the Algorithm Why Solve Linear Recurrences? Roy Pargas, Clemson University pargas@clemson.edu July 30, 2011 50 Golden Years Ateneo Mathematics Program Quezon City, Philippines 37

37. Designing the Algorithm Why Solve Linear Recurrences? The problem: Solving PDEs using the Method of Finite Differences leads to Roy Pargas, Clemson University pargas@clemson.edu July 30, 2011 50 Golden Years Ateneo Mathematics Program Quezon City, Philippines 38

38. Designing the Algorithm Why Solve Linear Recurrences? The problem: Solving PDEs using the Method of Finite Differences leads to Solving Block Tridiagonal Systems which leads to Roy Pargas, Clemson University pargas@clemson.edu July 30, 2011 50 Golden Years Ateneo Mathematics Program Quezon City, Philippines 39

39. Designing the Algorithm Why Solve Linear Recurrences? The problem: Solving PDEs using the Method of Finite Differences leads to Solving Block Tridiagonal Systems which leads to Solving Tridiagonal Systems which leads to Roy Pargas, Clemson University pargas@clemson.edu July 30, 2011 50 Golden Years Ateneo Mathematics Program Quezon City, Philippines 40

40. Designing the Algorithm Why Solve Linear Recurrences? The problem: Solving PDEs using the Method of Finite Differences leads to Solving Block Tridiagonal Systems which leads to Solving Tridiagonal Systems which leads to Solving Linear Recurrences Roy Pargas, Clemson University pargas@clemson.edu July 30, 2011 50 Golden Years Ateneo Mathematics Program Quezon City, Philippines 41

41. Designing the Algorithm Why Solve Linear Recurrences? The problem: Solving PDEs using the Method of Finite Differences leads to Solving Block Tridiagonal Systems which leads to Solving Tridiagonal Systems which leads to Solving Linear Recurrences many many many times Roy Pargas, Clemson University pargas@clemson.edu July 30, 2011 50 Golden Years Ateneo Mathematics Program Quezon City, Philippines 42

42. Designing the Algorithm Why Solve Linear Recurrences? The problem: Solving PDEs using the Method of Finite Differences leads to Solving Block Tridiagonal Systems which leads to Solving Tridiagonal Systems which leads to Solving Linear Recurrences many many many times Why Use a Binary Tree? Roy Pargas, Clemson University pargas@clemson.edu July 30, 2011 50 Golden Years Ateneo Mathematics Program Quezon City, Philippines 43

43. Linear Recurrences Roy Pargas, Clemson University pargas@clemson.edu July 30, 2011 50 Golden Years Ateneo Mathematics Program Quezon City, Philippines 44 Key Idea: Successfully solving the original pde problem depends upon solving recurrences quickly and efficiently.

44. Linear Recurrences Roy Pargas, Clemson University pargas@clemson.edu July 30, 2011 50 Golden Years Ateneo Mathematics Program Quezon City, Philippines 45 Consider the following set of n equations: x 0 = a 0 x 1 = a 1 + b 1 x 0 x 2 = a 2 + b 2 x 1... x n-1 = a n-1 + b n-1 x n-2

45. Linear Recurrences Roy Pargas, Clemson University pargas@clemson.edu July 30, 2011 50 Golden Years Ateneo Mathematics Program Quezon City, Philippines 46 Consider the following set of n equations: x 0 = a 0 x 1 = a 1 + b 1 x 0 x 2 = a 2 + b 2 x 1... x n-1 = a n-1 + b n-1 x n-2 Can we solve for x i in parallel?

46. Linear Recurrences Roy Pargas, Clemson University pargas@clemson.edu July 30, 2011 50 Golden Years Ateneo Mathematics Program Quezon City, Philippines 47 For uniformity: x 0 = a 0 + b 0 x -1 b 0 =0, x -1 =dummy variable x 1 = a 1 + b 1 x 0 x 2 = a 2 + b 2 x 1... x n-1 = a n-1 + b n-1 x n-2

47. Linear Recurrences Roy Pargas, Clemson University pargas@clemson.edu July 30, 2011 50 Golden Years Ateneo Mathematics Program Quezon City, Philippines 48 For uniformity: x 0 = a 0 + b 0 x -1 x 1 = a 1 + b 1 x 0 x 2 = a 2 + b 2 x 1... x n-1 = a n-1 + b n-1 x n-2 Observe, if x i = a + b x j x j = a + b x k Then Then x i = (a + ba) +bb x k = a + b x k = a + b x k

48. Linear Recurrences Roy Pargas, Clemson University pargas@clemson.edu July 30, 2011 50 Golden Years Ateneo Mathematics Program Quezon City, Philippines 49 Notation change x 0 = a 0 + b 0 x -1 C 0,-1 = (a 0,b 0 ) x 1 = a 1 + b 1 x 0 C 1,0 = (a 1,b 1 ) x 2 = a 2 + b 2 x 1 C 2,1 = (a 2,b 2 )... x n-1 = a n-1 + b n-1 x n-2 C n-1,n-2 = (a n-1,b n-1 ) Observe, if x i = a + b x j x j = a + b x k Then Then x i = (a + ba) +bb x k = a + b x k = a + b x k

49. Linear Recurrences Roy Pargas, Clemson University pargas@clemson.edu July 30, 2011 50 Golden Years Ateneo Mathematics Program Quezon City, Philippines 50 Notation change x 0 = a 0 + b 0 x -1 C 0,-1 = (a 0,b 0 ) x 1 = a 1 + b 1 x 0 C 1,0 = (a 1,b 1 ) x 2 = a 2 + b 2 x 1 C 2,1 = (a 2,b 2 )... x n-1 = a n-1 + b n-1 x n-2 C n-1,n-2 = (a n-1,b n-1 ) Observe, if x i = a + b x j C i,j = (a,b) x j = a + b x k C j,k = (a,b) Then Then x i = (a + ba) +bb x k C i,j C j,k = = a + b x k C i,k = (a+ba,bb) = a + b x k C i,k = (a+ba,bb)

50. Linear Recurrences Roy Pargas, Clemson University pargas@clemson.edu July 30, 2011 50 Golden Years Ateneo Mathematics Program Quezon City, Philippines 51 To summarize: x i = a + b x j C i,j = (a,b) x j = a + b x k C j,k = (a,b) Then Then x i = (a + ba) +bb x k C i,j C j,k = = a + b x k C i,k = (a+ba,bb) = a + b x k C i,k = (a+ba,bb)

51. Linear Recurrences Roy Pargas, Clemson University pargas@clemson.edu July 30, 2011 50 Golden Years Ateneo Mathematics Program Quezon City, Philippines 52 To summarize: x i = a + b x j C i,j = (a,b) x j = a + b x k C j,k = (a,b) Then Then x i = (a + ba) +bb x k C i,j C j,k = = a + b x k C i,k = (a+ba,bb) = a + b x k C i,k = (a+ba,bb) One last observation:

52. Linear Recurrences Roy Pargas, Clemson University pargas@clemson.edu July 30, 2011 50 Golden Years Ateneo Mathematics Program Quezon City, Philippines 53 To summarize: x i = a + b x j C i,j = (a,b) x j = a + b x k C j,k = (a,b) Then Then x i = (a + ba) +bb x k C i,j C j,k = = a + b x k C i,k = (a+ba,bb) = a + b x k C i,k = (a+ba,bb) One last observation: If any variable is expressed in terms of the dummy variable x -1 (e.g., x 0 = a 0 + b 0 x -1 ) then that variable is solved.

53. Linear Recurrences Roy Pargas, Clemson University pargas@clemson.edu July 30, 2011 50 Golden Years Ateneo Mathematics Program Quezon City, Philippines 54 To summarize: x i = a + b x j C i,j = (a,b) x j = a + b x k C j,k = (a,b) Then Then x i = (a + ba) +bb x k C i,j C j,k = = a + b x k C i,k = (a+ba,bb) = a + b x k C i,k = (a+ba,bb) One last observation: If any variable is expressed in terms of the dummy variable x -1 (e.g., x 0 = a 0 + b 0 x -1 ) then that variable is solved. So C i,-1 = (a,b)

54. Linear Recurrences Roy Pargas, Clemson University pargas@clemson.edu July 30, 2011 50 Golden Years Ateneo Mathematics Program Quezon City, Philippines 55 To summarize: x i = a + b x j C i,j = (a,b) x j = a + b x k C j,k = (a,b) Then Then x i = (a + ba) +bb x k C i,j C j,k = = a + b x k C i,k = (a+ba,bb) = a + b x k C i,k = (a+ba,bb) One last observation: If any variable is expressed in terms of the dummy variable x -1 (e.g., x 0 = a 0 + b 0 x -1 ) then that variable is solved. So C i,-1 = (a,b) means that b=0

55. Linear Recurrences Roy Pargas, Clemson University pargas@clemson.edu July 30, 2011 50 Golden Years Ateneo Mathematics Program Quezon City, Philippines 56 To summarize: x i = a + b x j C i,j = (a,b) x j = a + b x k C j,k = (a,b) Then Then x i = (a + ba) +bb x k C i,j C j,k = = a + b x k C i,k = (a+ba,bb) = a + b x k C i,k = (a+ba,bb) One last observation: If any variable is expressed in terms of the dummy variable x -1 (e.g., x 0 = a 0 + b 0 x -1 ) then that variable is solved. So C i,-1 = (a,b) means that b=0 and that x i = a + b x -1 = a + 0 x -1 = a

56. C 0,-1 C 1,0 C 2,1 C 3,2 C 4,3 C 5,4 C 6,5 C 7,6 Roy Pargas, Clemson University pargas@clemson.edu July 30, 2011 50 Golden Years Ateneo Mathematics Program Quezon City, Philippines 57 Linear Recurrences

57. Roy Pargas, Clemson University pargas@clemson.edu July 30, 2011 50 Golden Years Ateneo Mathematics Program Quezon City, Philippines 58 C 0,-1 C 1,0 C 2,1 C 3,2 C 4,3 C 5,4 C 6,5 C 7,6 C 0,-1 C 1,0 C 3,2 C 4,3 C 5,4 C 6,5 C 7,6 C 2,1

58. Linear Recurrences Roy Pargas, Clemson University pargas@clemson.edu July 30, 2011 50 Golden Years Ateneo Mathematics Program Quezon City, Philippines 59 C 0,-1 C 1,0 C 2,1 C 3,2 C 4,3 C 5,4 C 6,5 C 7,6 C 0,-1 C 1,0 C 3,2 C 4,3 C 5,4 C 6,5 C 7,6 C 2,1 C 7,5 C 5,3 C 3,1 C 1,-1

59. Linear Recurrences Roy Pargas, Clemson University pargas@clemson.edu July 30, 2011 50 Golden Years Ateneo Mathematics Program Quezon City, Philippines 60 C 0,-1 C 1,0 C 2,1 C 3,2 C 4,3 C 5,4 C 6,5 C 7,6 C 0,-1 C 1,0 C 3,2 C 4,3 C 5,4 C 6,5 C 7,6 C 2,1 C 7,5 C 5,3 C 3,1 C 1,-1 C 7,5 C 5,3 C 3,1 C 1,-1

60. Linear Recurrences Roy Pargas, Clemson University pargas@clemson.edu July 30, 2011 50 Golden Years Ateneo Mathematics Program Quezon City, Philippines 61 C 0,-1 C 1,0 C 2,1 C 3,2 C 4,3 C 5,4 C 6,5 C 7,6 C 0,-1 C 1,0 C 3,2 C 4,3 C 5,4 C 6,5 C 7,6 C 2,1 C 7,5 C 5,3 C 3,1 C 1,-1 C 7,5 C 5,3 C 3,1 C 1,-1 C 7,3 C 3,-1

61. Linear Recurrences Roy Pargas, Clemson University pargas@clemson.edu July 30, 2011 50 Golden Years Ateneo Mathematics Program Quezon City, Philippines 62 C 0,-1 C 1,0 C 2,1 C 3,2 C 4,3 C 5,4 C 6,5 C 7,6 C 0,-1 C 1,0 C 3,2 C 4,3 C 5,4 C 6,5 C 7,6 C 2,1 C 7,5 C 5,3 C 3,1 C 1,-1 C 7,5 C 5,3 C 3,1 C 1,-1 C 7,3 C 3,-1 C 7,3 C 3,-1

62. Linear Recurrences Roy Pargas, Clemson University pargas@clemson.edu July 30, 2011 50 Golden Years Ateneo Mathematics Program Quezon City, Philippines 63 C 0,-1 C 1,0 C 2,1 C 3,2 C 4,3 C 5,4 C 6,5 C 7,6 C 0,-1 C 1,0 C 3,2 C 4,3 C 5,4 C 6,5 C 7,6 C 2,1 C 7,5 C 5,3 C 3,1 C 1,-1 C 7,5 C 5,3 C 3,1 C 1,-1 C 7,3 C 3,-1 C 7,3 C 3,-1 C 7,-1

63. Linear Recurrences Roy Pargas, Clemson University pargas@clemson.edu July 30, 2011 50 Golden Years Ateneo Mathematics Program Quezon City, Philippines 64 C 0,-1 C 1,0 C 2,1 C 3,2 C 4,3 C 5,4 C 6,5 C 7,6 C 0,-1 C 1,0 C 3,2 C 4,3 C 5,4 C 6,5 C 7,6 C 2,1 C 7,5 C 5,3 C 3,1 C 1,-1 C 7,5 C 5,3 C 3,1 C 1,-1 C 7,3 C 3,-1 C 7,3 C 3,-1 C 7,-1

64. Linear Recurrences Roy Pargas, Clemson University pargas@clemson.edu July 30, 2011 50 Golden Years Ateneo Mathematics Program Quezon City, Philippines 65 C 0,-1 C 1,0 C 2,1 C 3,2 C 4,3 C 5,4 C 6,5 C 7,6 C 0,-1 C 1,0 C 3,2 C 4,3 C 5,4 C 6,5 C 7,6 C 2,1 C 7,5 C 5,3 C 3,1 C 1,-1 C 7,5 C 5,3 C 3,1 C 1,-1 C 7,3 C 3,-1 C 7,3 C 3,-1 C 7,-1 Solved variable

65. Linear Recurrences Roy Pargas, Clemson University pargas@clemson.edu July 30, 2011 50 Golden Years Ateneo Mathematics Program Quezon City, Philippines 66 C 0,-1 C 1,0 C 2,1 C 3,2 C 4,3 C 5,4 C 6,5 C 7,6 C 0,-1 C 1,0 C 3,2 C 4,3 C 5,4 C 6,5 C 7,6 C 2,1 C 7,5 C 5,3 C 3,1 C 1,-1 C 7,5 C 5,3 C 3,1 C 1,-1 C 7,3 C 3,-1 C 7,3 C 3,-1 C 7,-1 Solved variables

66. Linear Recurrences Roy Pargas, Clemson University pargas@clemson.edu July 30, 2011 50 Golden Years Ateneo Mathematics Program Quezon City, Philippines 67 C 0,-1 C 1,0 C 2,1 C 3,2 C 4,3 C 5,4 C 6,5 C 7,6 C 0,-1 C 1,0 C 3,2 C 4,3 C 5,4 C 6,5 C 7,6 C 2,1 C 7,5 C 5,3 C 3,1 C 1,-1 C 7,5 C 5,3 C 3,1 C 1,-1 C 7,3 C 3,-1 C 7,3 C 3,-1 C 7,-1 How do we solve for the other variables?

67. C 6,5 C 7,-1 Linear Recurrences Roy Pargas, Clemson University pargas@clemson.edu July 30, 2011 50 Golden Years Ateneo Mathematics Program Quezon City, Philippines 68 C 0,-1 C 1,0 C 2,1 C 3,2 C 4,3 C 5,4 C 7,6 C 0,-1 C 1,0 C 3,2 C 4,3 C 5,4 C 6,5 C 7,6 C 2,1 C 7,5 C 5,3 C 3,1 C 1,-1 C 7,5 C 5,3 C 3,1 C 1,-1 C 7,3 C 3,-1 C 7,3 C 3,-1 C 7,-1 In the downsweep!

68. C 6,5 C 7,-1 Linear Recurrences Roy Pargas, Clemson University pargas@clemson.edu July 30, 2011 50 Golden Years Ateneo Mathematics Program Quezon City, Philippines 69 C 0,-1 C 1,0 C 2,1 C 3,2 C 4,3 C 5,4 C 7,6 C 0,-1 C 1,0 C 3,2 C 4,3 C 5,4 C 6,5 C 7,6 C 2,1 C 7,5 C 5,3 C 3,1 C 1,-1 C 7,5 C 5,3 C 3,1 C 1,-1 C 7,3 C 3,-1 C 7,3 C 3,-1 C 7,-1 (x 7,x -1 )

69. C 6,5 (x 7,x -1 ) Linear Recurrences Roy Pargas, Clemson University pargas@clemson.edu July 30, 2011 50 Golden Years Ateneo Mathematics Program Quezon City, Philippines 70 C 0,-1 C 1,0 C 2,1 C 3,2 C 4,3 C 5,4 C 7,6 C 0,-1 C 1,0 C 3,2 C 4,3 C 5,4 C 6,5 C 7,6 C 2,1 C 7,5 C 5,3 C 3,1 C 1,-1 C 7,5 C 5,3 C 3,1 C 1,-1 C 7,3 C 3,-1 C 7,3 C 3,-1 C 7,-1 (x 7,x -1 )

70. C 6,5 (x 7,x -1 ) Linear Recurrences Roy Pargas, Clemson University pargas@clemson.edu July 30, 2011 50 Golden Years Ateneo Mathematics Program Quezon City, Philippines 71 C 0,-1 C 1,0 C 2,1 C 3,2 C 4,3 C 5,4 C 7,6 C 0,-1 C 1,0 C 3,2 C 4,3 C 5,4 C 6,5 C 7,6 C 2,1 C 7,5 C 5,3 C 3,1 C 1,-1 C 7,5 C 5,3 C 3,1 C 1,-1 C 7,3 C 3,-1 C 7,3 C 3,-1 x3x3 (x 7,x -1 )

71. C 6,5 (x 7,x -1 ) Linear Recurrences Roy Pargas, Clemson University pargas@clemson.edu July 30, 2011 50 Golden Years Ateneo Mathematics Program Quezon City, Philippines 72 C 0,-1 C 1,0 C 2,1 C 3,2 C 4,3 C 5,4 C 7,6 C 0,-1 C 1,0 C 3,2 C 4,3 C 5,4 C 6,5 C 7,6 C 2,1 C 7,5 C 5,3 C 3,1 C 1,-1 C 7,5 C 5,3 C 3,1 C 1,-1 C 7,3 C 3,-1 x3x3 (x 7,x -1 ) (x 7,x 3 ) (x 3,x -1 )

72. C 6,5 (x 7,x -1 ) Linear Recurrences Roy Pargas, Clemson University pargas@clemson.edu July 30, 2011 50 Golden Years Ateneo Mathematics Program Quezon City, Philippines 73 C 0,-1 C 1,0 C 2,1 C 3,2 C 4,3 C 5,4 C 7,6 C 0,-1 C 1,0 C 3,2 C 4,3 C 5,4 C 6,5 C 7,6 C 2,1 C 7,5 C 5,3 C 3,1 C 1,-1 C 3,1 C 1,-1 x3x3 (x 7,x -1 ) (x 7,x 3 ) (x 3,x -1 ) x5x5 C 7,5 C 5,3 x1x1

73. C 6,5 (x 7,x -1 ) Linear Recurrences Roy Pargas, Clemson University pargas@clemson.edu July 30, 2011 50 Golden Years Ateneo Mathematics Program Quezon City, Philippines 74 C 0,-1 C 1,0 C 2,1 C 3,2 C 4,3 C 5,4 C 7,6 C 0,-1 C 1,0 C 3,2 C 4,3 C 5,4 C 6,5 C 7,6 C 2,1 C 7,5 C 5,3 C 3,1 C 1,-1 x3x3 (x 7,x -1 ) (x 7,x 3 ) (x 3,x -1 ) x5x5 x1x1 (x 7,x 5 ) (x 5,x 3 ) (x 3,x 1 )(x 1,x -1 )

74. (x 7,x 5 ) C 6,5 (x 7,x -1 ) Linear Recurrences Roy Pargas, Clemson University pargas@clemson.edu July 30, 2011 50 Golden Years Ateneo Mathematics Program Quezon City, Philippines 75 C 0,-1 C 1,0 C 2,1 C 3,2 C 4,3 C 5,4 C 7,6 C 0,-1 C 1,0 C 3,2 C 4,3 C 5,4 C 6,5 C 7,6 C 2,1 x3x3 (x 7,x -1 ) (x 7,x 3 ) (x 3,x -1 ) x5x5 x1x1 (x 5,x 3 ) (x 3,x 1 )(x 1,x -1 ) x6x6 x4x4 x3x3 x1x1

75. (x 7,x 5 ) (x 7,x -1 ) Linear Recurrences Roy Pargas, Clemson University pargas@clemson.edu July 30, 2011 50 Golden Years Ateneo Mathematics Program Quezon City, Philippines 76 C 0,-1 C 1,0 C 3,2 C 4,3 C 5,4 C 6,5 C 7,6 C 2,1 x3x3 (x 7,x -1 ) (x 7,x 3 ) (x 3,x -1 ) x5x5 x1x1 (x 5,x 3 ) (x 3,x 1 )(x 1,x -1 ) x6x6 x4x4 x3x3 x1x1 (x 7,x 6 ) (x 6,x 5 ) (x 5,x 4 ) (x 0,x -1 ) (x 1,x 0 ) (x 4,x 3 ) (x 3,x 2 ) (x 2,x 1 )

76. (x 7,x 5 ) (x 7,x -1 ) Linear Recurrences Roy Pargas, Clemson University pargas@clemson.edu July 30, 2011 50 Golden Years Ateneo Mathematics Program Quezon City, Philippines 77 x3x3 (x 7,x -1 ) (x 7,x 3 ) (x 3,x -1 ) x5x5 x1x1 (x 5,x 3 ) (x 3,x 1 )(x 1,x -1 ) x6x6 x4x4 x3x3 x1x1 (x 7,x 6 ) (x 6,x 5 ) (x 5,x 4 ) (x 0,x -1 ) (x 1,x 0 ) (x 4,x 3 ) (x 3,x 2 ) (x 2,x 1 ) (x 7,x 6 ) (x 0,x -1 ) (x 6,x 5 ) (x 5,x 4 ) (x 4,x 3 ) (x 3,x 2 ) (x 2,x 1 ) (x 1,x 0 )

77. (x 7,x 5 ) (x 7,x -1 ) Linear Recurrences Roy Pargas, Clemson University pargas@clemson.edu July 30, 2011 50 Golden Years Ateneo Mathematics Program Quezon City, Philippines 78 x3x3 (x 7,x -1 ) (x 7,x 3 ) (x 3,x -1 ) x5x5 x1x1 (x 5,x 3 ) (x 3,x 1 )(x 1,x -1 ) x6x6 x4x4 x3x3 x1x1 (x 7,x 6 ) (x 6,x 5 ) (x 5,x 4 ) (x 0,x -1 ) (x 1,x 0 ) (x 4,x 3 ) (x 3,x 2 ) (x 2,x 1 ) (x 7,x 6 ) (x 0,x -1 ) (x 6,x 5 ) (x 5,x 4 ) (x 4,x 3 ) (x 3,x 2 ) (x 2,x 1 ) (x 1,x 0 ) Leaves contain solutions!

78. (x 7,x 5 ) (x 7,x -1 ) Linear Recurrences Roy Pargas, Clemson University pargas@clemson.edu July 30, 2011 50 Golden Years Ateneo Mathematics Program Quezon City, Philippines 79 x3x3 (x 7,x -1 ) (x 7,x 3 ) (x 3,x -1 ) x5x5 x1x1 (x 5,x 3 ) (x 3,x 1 )(x 1,x -1 ) x6x6 x4x4 x3x3 x1x1 (x 7,x 6 ) (x 6,x 5 ) (x 5,x 4 ) (x 0,x -1 ) (x 1,x 0 ) (x 4,x 3 ) (x 3,x 2 ) (x 2,x 1 ) (x 7,x 6 ) (x 0,x -1 ) (x 6,x 5 ) (x 5,x 4 ) (x 4,x 3 ) (x 3,x 2 ) (x 2,x 1 ) (x 1,x 0 ) But we can do better!

79. (x 7,x 5 ) (x 7,x -1 ) Linear Recurrences Roy Pargas, Clemson University pargas@clemson.edu July 30, 2011 50 Golden Years Ateneo Mathematics Program Quezon City, Philippines 80 x3x3 (x 7,x -1 ) (x 7,x 3 ) (x 3,x -1 ) x5x5 x1x1 (x 5,x 3 ) (x 3,x 1 )(x 1,x -1 ) x6x6 x4x4 x3x3 x1x1 (x 7,x 6 ) (x 6,x 5 ) (x 5,x 4 ) (x 0,x -1 ) (x 1,x 0 ) (x 4,x 3 ) (x 3,x 2 ) (x 2,x 1 ) (x 7,x 6 ) (x 0,x -1 ) (x 6,x 5 ) (x 5,x 4 ) (x 4,x 3 ) (x 3,x 2 ) (x 2,x 1 ) (x 1,x 0 ) Pipelining the solution

80. Roy Pargas, Clemson University pargas@clemson.edu July 30, 2011 50 Golden Years Ateneo Mathematics Program Quezon City, Philippines 81 Linear Recurrences Pipelining the solution

81. Roy Pargas, Clemson University pargas@clemson.edu July 30, 2011 50 Golden Years Ateneo Mathematics Program Quezon City, Philippines 82 Linear Recurrences Pipelining the solution

82. Roy Pargas, Clemson University pargas@clemson.edu July 30, 2011 50 Golden Years Ateneo Mathematics Program Quezon City, Philippines 83 Linear Recurrences Pipelining the solution

83. Roy Pargas, Clemson University pargas@clemson.edu July 30, 2011 50 Golden Years Ateneo Mathematics Program Quezon City, Philippines 84 Linear Recurrences Pipelining the solution

84. Roy Pargas, Clemson University pargas@clemson.edu July 30, 2011 50 Golden Years Ateneo Mathematics Program Quezon City, Philippines 85 Linear Recurrences Pipelining the solution

85. Roy Pargas, Clemson University pargas@clemson.edu July 30, 2011 50 Golden Years Ateneo Mathematics Program Quezon City, Philippines 86 Linear Recurrences Pipelining the solution

86. Roy Pargas, Clemson University pargas@clemson.edu July 30, 2011 50 Golden Years Ateneo Mathematics Program Quezon City, Philippines 87 Linear Recurrences Pipelining the solution

87. Roy Pargas, Clemson University pargas@clemson.edu July 30, 2011 50 Golden Years Ateneo Mathematics Program Quezon City, Philippines 88 Linear Recurrences Pipelining the solution

88. Roy Pargas, Clemson University pargas@clemson.edu July 30, 2011 50 Golden Years Ateneo Mathematics Program Quezon City, Philippines 89 Linear Recurrences Pipelining the solution

89. Roy Pargas, Clemson University pargas@clemson.edu July 30, 2011 50 Golden Years Ateneo Mathematics Program Quezon City, Philippines 90 Linear Recurrences Pipelining the solution

90. Roy Pargas, Clemson University pargas@clemson.edu July 30, 2011 50 Golden Years Ateneo Mathematics Program Quezon City, Philippines 91 Linear Recurrences Pipelining the solution

91. Analysis T1 = Time on one processor Tn = Time on n processors S = Speedup = T1/Tn (ideal: S = n) E = Efficiency = S/n (ideal: E = 1) Roy Pargas, Clemson University pargas@clemson.edu July 30, 2011 50 Golden Years Ateneo Mathematics Program Quezon City, Philippines 92

92. Speedup and Efficiency Single Pass Assume 1 floating point operation requires 1 time unit T1 = (n1) (1 mult + 1 add) = 2n2 time units n leaves 2n processors T2n = (log 2 n)( 2 mults + 1 add) // upsweep + (log 2 n)(1 mult + 1 add) // downsweep T2n = (log 2 n)( 2 mults + 1 add) // upsweep + (log 2 n)(1 mult + 1 add) // downsweep = 5 log 2 n time units S = Speedup = T1/T2n = (2n2)/(5 log 2 n) E = Efficiency = S/2n = (2n2)/[ (5 log 2 n) (2n) ] Roy Pargas, Clemson University pargas@clemson.edu July 30, 2011 50 Golden Years Ateneo Mathematics Program Quezon City, Philippines 93

93. Speedup Roy Pargas, Clemson University pargas@clemson.edu July 30, 2011 50 Golden Years Ateneo Mathematics Program Quezon City, Philippines 94 Speedup and Efficiency Single Pass

94. Speedup Efficiency Roy Pargas, Clemson University pargas@clemson.edu July 30, 2011 50 Golden Years Ateneo Mathematics Program Quezon City, Philippines 95 Speedup and Efficiency Single Pass

95. With Pipelining. Assume kn equations for large k T1 = (kn1) (1 mult + 1 add) = 2kn2 time units n leaves 2n processors T2n = (log 2 n)( 2 mults + 1 add) // pipefill up + (k – 2 log 2 n) (5) // pipeline on k2log 2 n waves T2n = (log 2 n)( 2 mults + 1 add) // pipefill up + (k – 2 log 2 n) (5) // pipeline on k2log 2 n waves + (log 2 n)( 1 mult + 1 add) // pipedrain down + (log 2 n)( 1 mult + 1 add) // pipedrain down = 5(k – log 2 n) time units S = Speedup = T1/T2n = (2kn2)/[5(k – log 2 n )] E = Efficiency = S/2n = (2kn2)/[5(k – log 2 n ) (2n)] Roy Pargas, Clemson University pargas@clemson.edu July 30, 2011 50 Golden Years Ateneo Mathematics Program Quezon City, Philippines 96 Speedup and Efficiency With Pipelining

96. Roy Pargas, Clemson University pargas@clemson.edu July 30, 2011 50 Golden Years Ateneo Mathematics Program Quezon City, Philippines 97 Speedup and Efficiency With Pipelining Speedup

97. Roy Pargas, Clemson University pargas@clemson.edu July 30, 2011 50 Golden Years Ateneo Mathematics Program Quezon City, Philippines 98 Speedup and Efficiency With Pipelining Speedup Efficiency

98. Technique Can Work For Roy Pargas, Clemson University pargas@clemson.edu July 30, 2011 50 Golden Years Ateneo Mathematics Program Quezon City, Philippines 99 Second-order linear recurrences x 0 = a 0 x 1 = a 1 + b 1 x 0 x 2 = a 2 + b 2 x 1 + c 2 x 0... x n-1 = a n-1 + b n-1 x n-2 + c n-1 x n-3 Higher order linear recurrences

99. Technique Can Work For Roy Pargas, Clemson University pargas@clemson.edu July 30, 2011 50 Golden Years Ateneo Mathematics Program Quezon City, Philippines 100 Quotients of linear recurrences x 0 = a 0 x i = (a i + b i x i-1 )/(c i + d i x i-1 ) i=1,2,…, n-1 Other recurrences

100. Summary and Conclusions Roy Pargas, Clemson University pargas@clemson.edu July 30, 2011 50 Golden Years Ateneo Mathematics Program Quezon City, Philippines 101

101. Summary and Conclusions Chip technology is going to get even better/faster/cheaper for the foreseeable future. Roy Pargas, Clemson University pargas@clemson.edu July 30, 2011 50 Golden Years Ateneo Mathematics Program Quezon City, Philippines 102

102. Summary and Conclusions Roy Pargas, Clemson University pargas@clemson.edu July 30, 2011 50 Golden Years Ateneo Mathematics Program Quezon City, Philippines 103 Chip technology is going to get even better/faster/cheaper for the foreseeable future. (Ignore the naysaying pundits!)

103. Summary and Conclusions Chip technology is going to get even better/faster/cheaper for the foreseeable future. (Ignore the naysaying pundits!) More massively parallel processing systems are going to be built and will become even better/faster/cheaper. Roy Pargas, Clemson University pargas@clemson.edu July 30, 2011 50 Golden Years Ateneo Mathematics Program Quezon City, Philippines 104

104. Summary and Conclusions Roy Pargas, Clemson University pargas@clemson.edu July 30, 2011 50 Golden Years Ateneo Mathematics Program Quezon City, Philippines 105 Chip technology is going to get even better/faster/cheaper for the foreseeable future. (Ignore the naysaying pundits!) More massively parallel processing systems are going to be built and will become even better/faster/cheaper. The challenging world of parallel algorithmic design beckons and awaits creative minds.

105. Summary and Conclusions Chip technology is going to get even better/faster/cheaper for the foreseeable future. (Ignore the naysaying pundits!) More massively parallel processing systems are going to be built and will become even better/faster/cheaper. The challenging world of parallel algorithmic design beckons and awaits creative minds. Yes, this means you! Roy Pargas, Clemson University pargas@clemson.edu July 30, 2011 50 Golden Years Ateneo Mathematics Program Quezon City, Philippines 106

106. Where in the World is Clemson University? 50 Golden Years Ateneo Mathematics Program Quezon City, Philippines 107 We are here! Roy Pargas, Clemson University pargas@clemson.edu July 30, 2011

107. Thank you for your kind attention! Roy Pargas, Clemson University pargas@clemson.edu July 30, 2011 50 Golden Years Ateneo Mathematics Program Quezon City, Philippines 108 Questions?

108. Extra Slides Roy Pargas, Clemson University pargas@clemson.edu July 30, 2011 50 Golden Years Ateneo Mathematics Program Quezon City, Philippines 109

109. Roy Pargas, Clemson University pargas@clemson.edu July 30, 2011 50 Golden Years Ateneo Mathematics Program Quezon City, Philippines 110 Links 1.Fujitsu K Computer (K = kei = Japanese word for 10 quadrillion) http://www.fujitsu.com/global/about/tech/k/ http://en.wikipedia.org/wiki/K_computer 2. NUDT Tianhe-1A Computer http://blog.zorinaq.com/?e=36 http://en.wikipedia.org/wiki/Tianhe-I 3. Cray Jaguar http://en.wikipedia.org/wiki/Jaguar_(computerhttp://en.wikipedia.org/wiki/Jaguar_(computer) http://www.nccs.gov/jaguar/ 4. Dawning Nebulae http://en.wikipedia.org/wiki/Dawning_Information_Industry http://www.theregister.co.uk/2010/05/31/top_500_supers_jun2010/ 5. NEC/HP Tsubame 2.0 http://en.wikipedia.org/wiki/TOP500

110. Hypercube Roy Pargas, Clemson University pargas@clemson.edu July 30, 2011 50 Golden Years Ateneo Mathematics Program Quezon City, Philippines 111 0 -Degree

111. Hypercube Roy Pargas, Clemson University pargas@clemson.edu July 30, 2011 50 Golden Years Ateneo Mathematics Program Quezon City, Philippines 112 0 -Degree

112. Hypercube Roy Pargas, Clemson University pargas@clemson.edu July 30, 2011 50 Golden Years Ateneo Mathematics Program Quezon City, Philippines 113 0 -Degree

113. Hypercube Roy Pargas, Clemson University pargas@clemson.edu July 30, 2011 50 Golden Years Ateneo Mathematics Program Quezon City, Philippines 114 0 -Degree

114. Hypercube Roy Pargas, Clemson University pargas@clemson.edu July 30, 2011 50 Golden Years Ateneo Mathematics Program Quezon City, Philippines 115 0 -Degree 1 -Degree 0 1

115. Hypercube Roy Pargas, Clemson University pargas@clemson.edu July 30, 2011 50 Golden Years Ateneo Mathematics Program Quezon City, Philippines 116 0 -Degree 1 -Degree 0 1 01

116. Hypercube Roy Pargas, Clemson University pargas@clemson.edu July 30, 2011 50 Golden Years Ateneo Mathematics Program Quezon City, Philippines 117 0 -Degree 1 -Degree 0 1 01 01

117. Hypercube Roy Pargas, Clemson University pargas@clemson.edu July 30, 2011 50 Golden Years Ateneo Mathematics Program Quezon City, Philippines 118 0 -Degree 1 -Degree 0 1 01 01

118. Hypercube Roy Pargas, Clemson University pargas@clemson.edu July 30, 2011 50 Golden Years Ateneo Mathematics Program Quezon City, Philippines 119 0 -Degree 1 -Degree 0 1 0001 1011 2 -Degree

119. Hypercube Roy Pargas, Clemson University pargas@clemson.edu July 30, 2011 50 Golden Years Ateneo Mathematics Program Quezon City, Philippines 120 3 -Degree 0 -Degree 1 -Degree 0 1 0001 1011 2 -Degree

120. Hypercube Roy Pargas, Clemson University pargas@clemson.edu July 30, 2011 50 Golden Years Ateneo Mathematics Program Quezon City, Philippines 121 3 -Degree 4 -Degree 0 -Degree 1 -Degree 0 1 0001 1011 2 -Degree

121. Hypercube Roy Pargas, Clemson University pargas@clemson.edu July 30, 2011 50 Golden Years Ateneo Mathematics Program Quezon City, Philippines 122 3 -Degree 4 -Degree 5 -Degree

122. All-to-All Communication (Hypercube) Roy Pargas, Clemson University pargas@clemson.edu July 30, 2011 50 Golden Years Ateneo Mathematics Program Quezon City, Philippines 123 ProblemMotivation