1. All-to-All Pattern A pattern where all (slave) processes can communicate with each other Somewhat the worst case scenario! 1 ITCS 4/5145 Parallel Computing, UNC-Charlotte, B. Wilkinson. SyncAll-AllPattern.ppt Jan 19, 2015

2. When a pattern is repeated until some termination condition occurs. Synchronization at each iteration, to establish termination condition, often a global condition. Note this is actually two patterns joined together sequentially if we call iteration a pattern. Iterative synchronous patterns Pattern Repeat Check termination condition Stop Note these pattern names are our names.

3. Iterative synchronous all-to-all pattern 3 Repeat Stop Check termination condition Implemented in Seeds as the “CompleteSynchGraph” pattern.

4. N-body problem –Finding positions and movements of bodies subject to forces from other bodies. –Could be bodies such as bodies in space (gravitational N- body problem), molecular bodies fundamental particles,... –Each iteration models one time interval. Solving system of linear equations by iteration –Each unknown given in terms of the other unknowns –Initial guess is made for the unknowns –Iterate to converge on the solution 4 Application examples

5. Equations: Gravitational force between two bodies of masses m a and m b is: G, gravitational constant. r distance between bodies. Subject to forces, body accelerates according to Newton’s 2nd law: m is the mass of body, F force it experiences, a is the resultant acceleration. F = ma Gravitational N-Body Problem Finding positions and movements of bodies in space subject to gravitational forces from other bodies. Use Newtonian laws of physics: 5

6. Velocity (from F=ma) Let time interval be  t. Position -- Over interval  t, position changes by Once bodies move to new positions, forces change. Computation has to be repeated to find movement over time. Details Force – First compute the force: 6 where mass of body is m, v t+1 is velocity at time t + 1 and v t is velocity at time t. where x t is its position at time t. New velocity is:

7. This then gives the velocity and positions in three directions.

8. This then gives the velocity and positions in two directions. two

9. Two-dimensional space -- a little easier to visualize. y x Force on body r Another body Add the force cause by each body in x and x directions Moves Movement

10. Code for 2-D Gravitational N-body problem Suppose a table is used to hold initial and computed data over time steps: BodyMassPosition in x direction Position in y direction Velocity in x direction Velocity in y direction 1 2.... N 10 On each iteration, position and velocities are updated. Table can be used to display movement of bodies

11. Sequential Code. The overall gravitational N-body computation can be described by the following steps: for (t = 0; t < tmax; t++) { //for each time period for (i = 0; i < N; i++) { //for body i, calculate force on body due to other bodies for (j = 0; j < N; j++) { if (i != j) { // for different bodies x_diff =... ; // compute distance between body i and body j in x direction y_diff =... ; // compute distance between body i and body j in y direction r =... ; // compute distance r F =... ; // compute force on bodies Fx[i] +=... ; // resolve and accumulate force in x direction Fy[i] += … ; // resolve and accumulate force in y direction } for (i = 0; i < N; i++) { // for each body, update positions and velocity A[i][x_velocity]=... ; // new velocity in x direction A[i][y_velocity]=... ; // new velocity in y direction A[i][x_position] =... ; // new position in x direction A[i][y_position] =... ; // new position in y direction } } // end time period

12. Time complexity Brute-force sequential algorithm is an O(N 2 ) algorithm for one iteration as each of the N bodies is influenced by each of the other N - 1 bodies. For t iterations, O(N 2 t) Not feasible to use this direct algorithm for most interesting N-body problems where N is very large. 12

13. Time complexity can be reduced approximating a cluster of distant bodies as a single distant body with mass sited at the center of mass of the cluster: 13 Reducing time complexity

14. Barnes-Hut Algorithm 3-D space: Start with whole space in which one cube contains the bodies. First, this cube is divided into eight subcubes. If a subcube contains no bodies, subcube deleted from further consideration. If a subcube contains one body, subcube retained. If a subcube contains more than one body, it is recursively divided until every subcube contains one body. 14

15. Creates an octtree - a tree with up to eight edges from each vertex (node). Leaves represent cells each containing one body. After tree constructed, total mass and center of mass of subcube stored at each vertex (node). 15

16. Force on each body obtained by traversing tree starting at root, stopping at a node when the clustering approximation can be used, e.g. when r is greater than some specified distance D. Constructing tree requires a time of O(NlogN), and so does computing all the forces, so that overall time complexity of method is O(NlogN). 16

17. Example for 2-dimensional space 17 At each vertex, store coordinates of center of mass and total mass of bodies in space below One body (bodies) Construction of tree:

18. 18 Computing force on each body -- traverse tree starting at root, stopping at a node when clustering approximation can be used, i.e. when r is greater than some set distance D. Mass and coordinates of center of mass of bodies in sub space For each body

19. An alternative way of dividing space. (For 2-dimensional area) First, a vertical line found that divides area into two areas each with equal number of bodies. For each area, a horizontal line found that divides it into two areas each with equal number of bodies. Repeated as required. Orthogonal Recursive Bisection 19

20. Solving General System of Linear Equations by iteration Suppose equations are of a general form with n equations and n unknowns: where the unknowns are x 0, x 1, x 2, … x n-1 (0 <= i < n). 20

21. By rearranging the ith equation: This equation gives x i in terms of the other unknowns. Can be used as an iteration formula for each of the unknowns to obtain better approximations. Process i computes x i 21

22. P0P0 P n-1 PiPi Suppose each process computes one unknown. P i computes x i Process P i needs unknowns from all other processes on each iteration (Excluding P i ) Needs iterative synchronous all-to-all pattern Computes: 22

23. Jacobi Iteration Name given to a computation that uses the previous iteration value to compute the next values. All values of x are updated together. 23 Other (non-parallel) methods use some of the present iteration values to compute the present values, see later.

24. Convergence: Can be proven that the Jacobi method will converge if diagonal values of a have an absolute value greater than sum of absolute values of other a’s on row, i.e. if This condition is a sufficient but not a necessary condition. Diagonal a’s

25. Simple, common approach is compare values computed in one iteration to values obtained from the previous iteration. Terminate computation when all values are within given tolerance; i.e., when However, this does not guarantee the solution to that accuracy. Why? Termination 25

26. Convergence Rate 26

27. Questions 27

28. More Information

29. Seeds “CompleteSynchGraph” Pattern All-to-all pattern that includes synchronous iteration feature to pass results of one iteration to all nodes before next iteration. Workers gets replicas of the initial data set. At each iteration, workers synchronize and update their replicas and proceed to new computations. Master node and framework will not get control of the data flow until all iterations done. 29 More information: “Seeds Framework – The CompleteSynchGraph Template Tutorial,” Jeremy Villalobos and Yawo K. Adibolo, June 18, 2012. at http://coitweb.uncc.edu/~abw/PatternProgGroup/index.html (to be moved)http://coitweb.uncc.edu/~abw/PatternProgGroup/index.html Gives details with code for the Jacobi iteration method of solving system of linear equations

30. System of linear equations can be described in matrix-vector form: AX = B where: Possible to create an iteration formula: X k = CX k-1 + D where X k is solution vector at iteration k X k-1 the solution vector at iteration k-1 C a matrix derived from input matrix A D a vector derived from input vector B. 30