1. Betweenness Centrality: Algorithms and Implementations Dimitrios Prountzos Keshav Pingali The University of Texas at Austin

2. Focus of this Talk A novel formulation of Betweenness Centrality – Based on Operator Formulation (Pingali et al. 2011) – Can express existing parallel solutions – Basis for new class of asynchronous parallel solutions Systematic derivation of parallel implementations from operator formulation – Ideas applicable to other irregular algorithms 2

3. Basic ingredient in Betweenness Centrality Problem Formulation – Compute shortest distance from source node S to every other node Many algorithms – Bellman-Ford (1957) – Dijkstra (1959) – Chaotic relaxation (Miranker 1969) – Delta-stepping (Meyer et al. 1998) Common structure – Each node has label d with known shortest distance from S Key operation – relax-edge(u,v) Warm-up: Single-Source Shortest-Path 3 2 5 1 7 A A B B C C D D E E F F G G S S 3 4 2 2 1 9 12 2 A A C C 3 If d(A) + W AC < d(C) d(C) = d(A) + W AC

4. Operator Formulation Concepts 4 u u v v dudu dvdv w d u + w < d v d u +w dudu w u u v v Operator: Conditional rewrite rule on graph Parallel Graph Algorithm Operators Schedule Order activity processing Identify new activities Identify new activities What should be done How it should be done : activity “TAO of parallelism” PLDI 2011

5. Identifies important nodes in a network Brandes’ Algorithm (2001) – Forward Pass: Compute shortest-path DAG for a given source S Compute shortest-path count σ(u) for each node u – Backward Pass: Traverse DAG and compute BC(u) Parallel Implementations – Bader et al. (2006) – Madduri et al. (2009) – Edmonds et al. (2010) – … Betweenness Centrality 5 # shortest paths from s to t # shortest paths from s to t through v B B A A D D C C E E 0,1 1,1 2,2 dC,σCdC,σC dC,σCdC,σC

6. B B BC Operator Formulation u u v v d u, σ u d v,σ v d uv,w,σ uv d v > d u + w d u +w, 0 d u, σ u d uv,w,σ uv P v = ∅, S v = ∅ u u v v Shortest Path (SP) A A D D C C E E 0,1 1,1 ∞,0 1,1 2,0 2,1 6 ∞,0 C C D D 1,1 predecessor successor u u v v d u, σ u d u +w,σ v d uv,w,σ uv d uv ≠ d u d u +w,σ v +σ u d u, σ u d uv,w, σ u Pv=uPv=u u u v v First Update (FU) SuvSuv u u v v d u, σ u d u +w,σ v d u,w,σ uv σ u ≠ σ uv d u +w, σ v +σ u - σ uv d u, σ u d uv,w,σ u u u v v Update Sigma (US) u u v v d u, σ u d v,σ v d u,w,σ uv d u ≠ ∞ ⋀ d u +w >d v d v,σ v d u, σ u, S u −=v ∞, w, σ uv u u v v Correct Node (CN) dA,σAdA,σA dA,σAdA,σA

7. B B BC Operator Formulation u u v v d u, σ u d v,σ v d uv,w,σ uv d v > d u + w d u +w, 0 d u, σ u d uv,w,σ uv P v = ∅, S v = ∅ u u v v Shortest Path (SP) A A D D C C E E 0,1 1,1 7 u u v v d u, σ u d u +w,σ v d uv,w,σ uv d uv ≠ d u d u +w,σ v +σ u d u, σ u d uv,w, σ u Pv=uPv=u u u v v First Update (FU) SuvSuv u u v v d u, σ u d u +w,σ v d u,w,σ uv σ u ≠ σ uv d u +w, σ v +σ u - σ uv d u, σ u d uv,w,σ u u u v v Update Sigma (US) u u v v d u, σ u d v,σ v d u,w,σ uv d u ≠ ∞ ⋀ d u +w >d v d v,σ v d u, σ u, S u −=v ∞, w, σ uv u u v v Correct Node (CN) 2,1

8. Operator Scheduling for Parallel Algorithm Derivation op ∈ {SP,FU,US,CN} while ∃ op(u,v) enabled apply op(u,v) Parallel Graph Algorithm Operators Schedule Order activity processing Identify new activities Identify new activities 8 op ∈ {SP,FU,US,CN} Wl = { (u,v) : op(u,v) enabled } while ¬Wl.empty { apply operator(s) Wl ∪ = … } Static Ordering Static Ordering Dynamic Ordering

9. Dynamic Operator Scheduling Worklist … … B B A A E E C C G G … D D F F J J H H I I … … … T1T1 TkTk … Our operators are general enough to enable this scheme 9 Variation of Delta-Stepping (Meyer et al. 1998)

10. Static Operator Scheduling Operator Grouping ⊕ Exploit locality, reduce worklist pressure ⊖ Load balancing Operator Merging – E.g. SP(u,v); FU(u,v)  Combine into SPFU ⊕ Optimize computation + locking Context-based Operator Inlining ⊕ Reduce worklist pressure ⊖ Load balancing 10 Bind scheduling decisions at compile-time by committing to particular code structure B1B1 B1B1 BnBn BnBn A A …

11. Algorithm Encodings Async1 : (SP|FU|US|CN)* – SPFU Async2 : (SP|FU|US|CN)* – Group (u,v*) – SPFU – SPFU inline CN Leveled : level-by-level (SP|FU)* – Group (u,v*) – SPFU 11

12. Experimental Evaluation 12

13. Experiments on Unweighted Graphs 24 core Intel Xeon @ 2 GHz 13 Scale-Free RMAT Graph 33 M nodes 268 M edges Random Graph 67 M nodes 268 M edges Leveled1 (Bader et al. 2006) Leveled2 (Madduri et al. 2009)

14. Experiments on Weighted Graphs 14 USA Road Network 24 M nodes 58 M edges USA Central Road Network 14 M nodes 34 M edges 24 core Intel Xeon @ 2 GHz Sun T5440 UltraSPARC T2+ @ 1.4 GHz 9.5x 38x

15. Conclusion New BC formulation – Expresses existing solutions – Basis for new asynchronous solutions Systematic derivation of parallel implementations – Dynamic + Static schedule transformations Enables automatic synthesis of parallel programs – Elixir [OOPSLA2012] 15 Thank You Parallel Graph Algorithm Operators Schedule

16. Backup 16

17. Identifies important nodes in a network Brandes’ Algorithm (2001) Update δ(v),BC(v) Betweenness Centrality A A B B C C E E F F H H S S Compute shortest path DAG (0,1) (1,1) (2,2) (3,2) (4,4) 17 L(u),σ(u)

18. B B BC Operator Formulation u u v v L u, σ u L v,σ v L uv,w,σ uv L v > L u + w L u +w, 0 L u, σ u L uv,w,σ uv P v = ∅, S v = ∅ u u v v Shortest Path (SP) A A D D C C E E 0,1 1,1 ∞,0 1,1 2,0 2,1 18 ∞,0 C C D D 1,1 predecessor successor u u v v L u, σ u L u +w,σ v L uv,w,σ uv L uv ≠ L u L u +w,σ v +σ u L u, σ u L uv,w, σ u Pv=uPv=u u u v v First Update (FU) SuvSuv u u v v L u, σ u L u +w,σ v L u,w,σ uv σ u ≠ σ uv L u +w, σ v +σ u - σ uv L u, σ u L uv,w,σ u u u v v Update Sigma (US) u u v v L u, σ u L v,σ v L u,w,σ uv L u ≠ ∞ ⋀ L u +w >L v L v,σ v L u, σ u, S u −=v ∞, w, σ uv u u v v Correct Node (CN) L(u),σ(u)

19. BC Operator Formulation B B A A D D C C E E 0,1 1,1 2,1 1,1 3,1 19 2,1 2,2 3,2 D D 1,0 1,1 ∞,1 u u v v L u, σ u L v,σ v L uv,w,σ uv L v > L u + w L u +w, 0 L u, σ u L uv,w,σ uv P v = ∅, S v = ∅ u u v v Shortest Path (SP) u u v v L u, σ u L u +w,σ v L uv,w,σ uv L uv ≠ L u L u +w,σ v +σ u L u, σ u L uv,w, σ u Pv=uPv=u u u v v First Update (FU) SuvSuv u u v v L u, σ u L u +w,σ v L u,w,σ uv σ u ≠ σ uv L u +w, σ v +σ u - σ uv L u, σ u L uv,w,σ u u u v v Update Sigma (US) u u v v L u, σ u L v,σ v L u,w,σ uv L u ≠ ∞ ⋀ L u +w >L v L v,σ v L u, σ u, S u −=v ∞, w, σ uv u u v v Correct Node (CN)

20. Deriving Algorithm Variants 20 Worklist Wl = { src } foreach u ∈ Wl { forall v ∈ outNbrs(u) { lock(u,v) if grd[SPFU,u,v] { … Wl ∪ = { v } if vHasPreds { forall w ∈ inNbrs(v) { lock(v,w) if grd[CN,w,v] {…} unlock(v,w) } } else-if … } Worklist Wl = { (src,w) : (src,w) ∈ G(V,E) } foreach (u,v) ∈ Wl { lock(u,v) if grd[SPFU,u,v] { apply SPFU ; unlock(u,v) Wl ∪ = { (v,w) : w ∈ outNbrs(v) } if vHasPreds Wl ∪ = { (w,u) : w ∈ inNbrs(v) } } else-if grd[CN,u,v] { apply CN ; unlock(u,v) } else-if grd[FU,u,v] { … } else-if grd[US,u,v] { … } Async1Async2

21. Insights Behind Elixir 21 What should be done How it should be done Unordered/Ordered algorithms Operator Delta : activity Parallel Graph Algorithm Operators Schedule Order activity processing Identify new activities Identify new activities Static Schedule Static Schedule Dynamic Schedule “TAO of parallelism” PLDI 2011