Presentation is loading. Please wait.

Presentation is loading. Please wait.

Parallel Graph Algorithms Sathish Vadhiyar. Graph Traversal  Graph search plays an important role in analyzing large data sets  Relationship between.

Similar presentations

Presentation on theme: "Parallel Graph Algorithms Sathish Vadhiyar. Graph Traversal  Graph search plays an important role in analyzing large data sets  Relationship between."— Presentation transcript:

1 Parallel Graph Algorithms Sathish Vadhiyar

2 Graph Traversal  Graph search plays an important role in analyzing large data sets  Relationship between data objects represented in the form of graphs  Breadth first search used in finding shortest path or sets of paths

3 Level-synchronized algorithm  Proceeds level-by-level starting with the source vertex  Level of a vertex – its graph distance from the source  How to decompose the graph (vertices, edges and adjacency matrix) among processors?

4 Distributed BFS with 1D Partitioning  Each vertex and edges emanating from it are owned by one processor  1-D partitioning of the adjacency matrix  Edges emanating from vertex v is its edge list = list of vertex indices in row v of adjacency matrix A

5 1-D Partitioning  At each level, each processor owns a set F – set of frontier vertices owned by the processor  Edge lists of vertices in F are merged to form a set of neighboring vertices, N  Some vertices of N owned by the same processor, while others owned by other processors  Messages are sent to those processors to add these vertices to their frontier set for the next level

6 L vs (v) – level of v, i.e, graph distance from source vs

7 2D Partitioning  P=RXC processor mesh  Adjacency matric divided into R.C block rows and C block columns  A (i,j) (*) denotes a block owned by (i,j) processor; each processor owns C blocks

8 2D Partitioning  Processor (i,j) owns vertices belonging to block row (j-1).R+i  Thus a process stores some edges incident on its vertices, and some edges that are not

9 2D Paritioning  Assume that the edge list for a given vertex is the column of the adjacency matrix  Each block in the 2D partitioning contains partial edge lists  Each processor has a frontier set of vertices, F, owned by the processor

10 2D Paritioning Expand Operation  Consider v in F  The owner of v sends messages to other processors in frontier column telling that v is in the frontier; since any of these processors may have partial edge list of v

11 2D Partitioning Fold Operation  Partial edge lists on each processor merged to form N – potential vertices in the next frontier  Vertices in N sent to their owners to form new frontier set F on those processors  These owner processors are in the same processor row  This communication step referred as fold operation


13 Analysis  Advantage of 2D over 1D – processor- column and processor-row communications involve only R and C processors

14 BFS on GPUs

15  One GPU thread for a vertex  In each iteration, each vertex looks at its entry in the frontier array  If true, it forms the neighbors and frontiers  Severe load imbalance among the treads  Scope for improvement

16 Parallel Depth First Search  Easy to parallelize  Left subtree can be searched in parallel with the right subtree  Statically assign a node to a processor – the whole subtree rooted at that node can be searched independently.  Can lead to load imbalance; Load imbalance increases with the number of processors

17 Dynamic Load Balancing (DLB)  Difficult to estimate the size of the search space beforehand  Need to balance the search space among processors dynamically  In DLB, when a processor runs out of work, it gets work from another processor

18 Maintaining Search Space  Each processor searches the space depth-first  Unexplored states saved as stack; each processor maintains its own local stack  Initially, the entire search space assigned to one processor

19 Work Splitting  When a processor receives work request, it splits its search space  Half-split: Stack space divided into two equal pieces – may result in load imbalance  Giving stack space near the bottom of the stack can lead to giving bigger trees  Stack space near the top of the stack tend to have small trees  To avoid sending very small amounts of work – nodes beyond a specified stack depth are not given away – cutoff depth

20 Strategies  1. Send nodes near the bottom of the stack  2. Send nodes near the cutoff depth  3. Send half the nodes between the bottom of the stack and the cutoff depth  Example: Figures 11.5(a) and 11.9

21 Load Balancing Strategies  Asynchronous round-robin: Each processor has a target processor to get work from; the value of the target is incremented with modulo  Global round-robin: One single target processor variable is maintained for all processors  Random polling: randomly select a donor

22 Termination Detection  Dijikstra’s Token Termination Detection Algorithm Based on passing of a token in a logical ring; P0 initiates a token when idle; A processor holds a token until it has completed its work, and then passes to the next processor; when P0 receives again, then all processors have completed However, a processor may get more work after becoming idle

23 Algorithm Continued…. Taken care of by using white and black tokens Initially, the token is white; a processor j becomes black if it sends work to i

24 Tree Based Termination Detection  Uses weights  Initially processor 0 has weight 1  When a processor transfers work to another processor, the weights are halved in both the processors  When a processor finishes, weights are returned  Termination is when processor 0 gets back 1  Goes with the DFS algorithm; No separate communication steps  Figure 11.10

25  Minimal Spanning Tree, Single-Source and All-pairs Shortest Paths

26 Minimal Spanning Tree – Prim’s Algorithm  Spanning tree of a graph, G (V,E) – tree containing all vertices of G  MST – spanning tree with minimum sum of weights  Vertices are added to a set Vt that holds vertices of MST; Initially contains an arbitrary vertex,r, as root vertex

27 Minimal Spanning Tree – Prim’s Algorithm  An array d such that d[v in (V-Vt)] holds weight of the edge with least weight between v and any vertex in Vt; Initially d[v] = w[r,v]  Find the vertex in d with minimum weight and add to Vt  Update d  Time complexity – O(n 2 )

28 Parallelization  Vertex V and d array partitioned across P processors  Each processor finds local minimum in d  Then global minimum across all d performed by reduction on a processor  The processor finds the next vertex u, and broadcasts to all processors

29 Parallelization  All processors update d; The owning processor of u marks u as belonging to Vt  Process responsible for v must know w[u,v] to update v; 1-D block mapping of adjacency matrix  Complexity – O(n 2 /P) + (OnlogP) for communication

30 Single Source Shortest Path – Dijikistra’s Algorithm  Finds shortest path from the source vertex to all vertices  Follows a similar structure as Prim’s  Instead of d array, an array l that maintains the shortest lengths are maintained  Follow similar parallelization scheme

31 Single Source Shortest Path on GPUs

32 SSSP on GPUs  A single kernel is not enough since Ca cannot be updated while it is accessed.  Hence costs updated in a temporary array Ua

33 All-Pairs Shortest Paths  To find shortest paths between all pairs of vertices  Dijikstra’s algorithm for single-source shortest path can be used for all vertices  Two approaches

34 All-Pairs Shortest Paths  Source-partitioned formulation: Partition the vertices across processors Works well if p<=n; No communication Can at best use only n processors Time complexity?  Source-parallel formulation: Parallelize SSSP for a vertex across a subset of processors  Do for all vertices with different subsets of processors  Hierarchical formulation  Exploits more parallelism  Time complexity?

35 All-Pairs Shortest Paths Floyd’s Algorithm  Consider a subset S = {v1,v2,…,vk} of vertices for some k <= n  Consider finding shortest path between vi and vj  Consider all paths from vi to vj whose intermediate vertices belong to the set S; Let p i,j (k) be the minimum-weight path among them with weight d i,j (k)

36 All-Pairs Shortest Paths Floyd’s Algorithm  If vk is not in the shortest path, then p i,j (k) = p i,j (k-1)  If vk is in the shortest path, then the path is broken into two parts – from vi to vk, and from vk to vj  So d i,j (k) = min{d i,j (k-1), d i,k (k-1) + d k,j (k-1) }  The length of the shortest path from vi to vj is given by d i,j (n).  In general, solution is a matrix D (n)

37 Parallel Formulation 2-D Block Mapping  Processors laid in a 2D mesh  During kth iteration, each process Pi,j needs certain segments of the kth row and kth column of the D(k-1) matrix  For d l,r (k) : following are needed d l,k (k-1) (from a process along the same process row) d k,r (k-1) (from a process along the same process column) Figure 10.8

38 Parallel Formulation 2D Block Mapping  During kth iteration, each of the root(p) processes containing part of the kth row sends it to root(p)-1 in same column;  Similarly for the same row  Figure 10.8  Time complexity?

39 APSP on GPUs  Space complexity of Floyd’s algorithm is O(V 2 ) – Impossible to go beyond a few vertices on GPUs  Uses V 2 threads  A single O(V) operation looping over O(V 2) threads - can exhibit slowdown due to high context switching overhead between threads  Use Dijikistra’s – run SSSP algorithm from every vertex in graph  Will require only the final output size to be O(V 2 )  Intermediate outputs on GPU can be O(V) and can be copied to CPU memory

40 APSP on GPUs

41 Sources/References  Paper: A Scalable Distributed Parallel Breadth-First Search Algorithm on BlueGene/L. Yoo et al. SC  Paper:Accelerating large graph algorithms on the GPU usingCUDA. Harish and Narayanan. HiPC 2007.

42 Speedup Anomalies in DFS  The overall work (space searched) in parallel DFS can be smaller or larger than in sequential DFS  Can cause superlinear or sublinear speedups  Figures 11.18, 11.19

43 Parallel Formulation Pipelining  In the 2D formulation, the kth iteration in all processes start only after k-1(th) iteration completes in all the processes  A process can start working on the kth iteration as soon as it has computed (k- 1)th iteration and has relevant parts of the D(k-1) matrix  Example: Figure 10.9  Time complexity

Download ppt "Parallel Graph Algorithms Sathish Vadhiyar. Graph Traversal  Graph search plays an important role in analyzing large data sets  Relationship between."

Similar presentations

Ads by Google