1 A High-Performance Interactive Tool for Exploring Large Graphs John R. Gilbert University of California, Santa Barbara Aydin Buluc & Viral Shah (UCSB) Brad McRae (NCEAS) Steve Reinhardt (Interactive Supercomputing) with thanks to Alan Edelman (MIT & ISC) and Jeremy Kepner (MIT-LL) Support: DOE Office of Science, NSF, DARPA, SGI, ISC

2 3D Spectral Coordinates

3 2D Histogram: RMAT Graph

4 Strongly Connected Components

5 Social Network Analysis in Matlab: 1993 Co-author graph from 1993 Householder symposium

6 Social Network Analysis in Matlab: 1993 Which author has the most collaborators? >>[count,author] = max(sum(A)) count = 32 author = 1 >>name(author,:) ans = Golub Sparse Adjacency Matrix

7 Social Network Analysis in Matlab: 1993 Have Gene Golub and Cleve Moler ever been coauthors? >> A(Golub,Moler) ans = 0 No. But how many coauthors do they have in common? >> AA = A^2; >> AA(Golub,Moler) ans = 2 And who are those common coauthors? >> name( find ( A(:,Golub).* A(:,Moler) ), :) ans = Wilkinson VanLoan

8 Outline Infrastructure: Array-based sparse graph computation An application: Computational ecology Some nuts and bolts: Sparse matrix multiplication

9 Combinatorial Scientific Computing Emerging large scale, high-performance applications: Web search and information retrieval Knowledge discovery Computational biology Dynamical systems Machine learning Bioinformatics Sparse matrix methods Geometric modeling... How will combinatorial methods be used by nonexperts?

10 Analogy: Matrix Division in Matlab x = A \ b; Works for either full or sparse A Is A square? no => use QR to solve least squares problem Is A triangular or permuted triangular? yes => sparse triangular solve Is A symmetric with positive diagonal elements? yes => attempt Cholesky after symmetric minimum degree Otherwise => use LU on A(:, colamd(A))

11 Matlab*P A = rand(4000*p, 4000*p); x = randn(4000*p, 1); y = zeros(size(x)); while norm(x-y) / norm(x) > 1e-11 y = x; x = A*x; x = x / norm(x); end;

12 MATLAB ® Star-P Architecture Ordinary Matlab variables Star-P client manager server manager package manager processor #0 processor #n-1 processor #1 processor #2 processor #3... ScaLAPACK FFTW FPGA interface matrix manager Distributed matrices sort dense/sparse UPC user code MPI user code

13 P0P0 P1P1 P2P2 PnPn Each processor stores local vertices & edges in a compressed row structure. Has been scaled to >10 8 vertices, >10 9 edges in interactive session. Distributed Sparse Array Structure

14 The sparse( ) Constructor A = sparse (I, J, V, nr, nc); Input: ddense vectors I, J, V, dimensions nr, nc Output: A ( I (k), J (k)) = V (k) Sum values with duplicate indices Sorts triples by Inverse: [I, J, V] = find(A);

15 Sparse Array and Matrix Operations dsparse layout, same semantics as ordinary full & sparse Matrix arithmetic: +, max, sum, etc. matrix * matrix and matrix * vector Matrix indexing and concatenation A (1:3, [4 5 2]) = [ B(:, J) C ] ; Linear solvers: x = A \ b; using SuperLU (MPI) Eigensolvers: [V, D] = eigs(A); using PARPACK (MPI)

16 Large-Scale Graph Algorithms Graph theory, algorithms, and data structures are ubiquitous in sparse matrix computation. Time to turn the relationship around! Represent a graph as a sparse adjacency matrix. A sparse matrix language is a good start on primitives for computing with graphs. Leverage the mature techniques and tools of high- performance numerical computation.

17 Sparse Adjacency Matrix and Graph Adjacency matrix: sparse array w/ nonzeros for graph edges Storage-efficient implementation from sparse data structures xATxATx ATAT 

18 Breadth-First Search: Sparse mat * vec xATxATx ATAT  Multiply by adjacency matrix  step to neighbor vertices Work-efficient implementation from sparse data structures

19 Breadth-First Search: Sparse mat * vec xATxATx ATAT  Multiply by adjacency matrix  step to neighbor vertices Work-efficient implementation from sparse data structures

20 Breadth-First Search: Sparse mat * vec ATAT (A T ) 2 x   xATxATx Multiply by adjacency matrix  step to neighbor vertices Work-efficient implementation from sparse data structures

21 Many tight clusters, loosely interconnected Input data is edge triples Vertices and edges permuted randomly SSCA#2: “Graph Analysis” Benchmark (spec version 1) Fine-grained, irregular data access Searching and clustering

22 Clustering by Breadth-First Search % Grow each seed to vertices % reached by at least k % paths of length 1 or 2 C = sparse(seeds, 1:ns, 1, n, ns); C = A * C; C = C + A * C; C = C >= k; Grow local clusters from many seeds in parallel Breadth-first search by sparse matrix * matrix Cluster vertices connected by many short paths

23 Toolbox for Graph Analysis and Pattern Discovery Layer 1: Graph Theoretic Tools Graph operations Global structure of graphs Graph partitioning and clustering Graph generators Visualization and graphics Scan and combining operations Utilities

24 Typical Application Stack Distributed Sparse Matrices Arithmetic, matrix multiplication, indexing, solvers (\, eigs) Graph Analysis & PD Toolbox Graph querying & manipulation, connectivity, spanning trees, geometric partitioning, nested dissection, NNMF,... Preconditioned Iterative Methods CG, BiCGStab, etc. + combinatorial preconditioners (AMG, Vaidya) Applications Computational ecology, CFD, data exploration

25 Landscape Connnectivity Modeling Landscape type and features facilitate or impede movement of members of a species Different species have different criteria, scales, etc. Habitat quality, gene flow, population stability Corridor identification, conservation planning

26 Pumas in Southern California Joshua Tree N.P. L.A. Palm Springs Habitat quality model

27 Predicting Gene Flow with Resistive Networks Circuit model predictions: Genetic vs. geographic distance:

28 Early Experience with Real Genetic Data Good results with wolverines, mahogany, pumas Matlab implementation Needed: –Finer resolution –Larger landscapes –Faster interaction 5km resolution(too coarse)

29 Combinatorics in Circuitscape Initial grid models connections to 4 or 8 neighbors. Partition landscape into connected components with GAPDT Graph contraction from GAPDT contracts habitats into single nodes in resistive network. (Need current flow between entire habitats.) Data-parallel computation on large graphs - graph construction, querying and manipulation. Ideally, model landscape at 100m resolution (for pumas). Tradeoff between resolution and time.

30 Numerics in Circuitscape Resistance computations for pairs of habitats in the landscape Direct methods are too slow for largest problems Use iterative solvers via Star-P: –Hypre (PCG+AMG) –Experimenting with support graph preconditioners

31 Parallel Circuitscape Results Pumas in southern California: –12 million nodes –Under 1 hour (16 processors) –Original code took 3 days at coarser resolution Targeting much larger problems: –Yellowstone-to-Yukon corridor Figures courtesy of Brad McRae, NCEAS

32 Sparse Matrix times Sparse Matrix A primitive in many array-based graph algorithms: –Parallel breadth-first search –Shortest paths –Graph contraction –Subgraph / submatrix indexing –Etc. Graphs are often not mesh-like, i.e. geometric locality and good separators. Often do not want to optimize for one repeated operation, as in matvec for iterative methods

33 Sparse Matrix times Sparse Matrix Current work: –Parallel algorithms with 2D data layout –Sequential hypersparse algorithms –Matrices over semirings

34 * = I J A(I,K) K K B(K,J) C(I,J) ParSpGEMM C(I,J) += A(I,K)*B(K,J) Based on SUMMA Simple for non-square matrices, etc.

35 How Sparse? HyperSparse ! blocks nnz(j) = c nnz(j) =  Any local data structure that depends on local submatrix dimension n (such as CSR or CSC) is too wasteful.

36 SparseDComp Data Structure “Doubly compressed” data structure Maintains both DCSC and DCSR C = A*B needs only A.DCSC and B.DCSR 4*nnz values communicated for A*B in the worst case (though we usually get away with much less)

37 Sequential Operation Counts Matlab: O(n+nnz(B)+f) SpGEMM: O(nzc(A)+nzr(B)+f*logk) Break-even point Required non- zero operations (flops) Number of columns of A containing at least one non-zero

38 Parallel Timings 16-processor Opteron, hypertransport, 64 GB memory R-MAT * R-MAT n = 2 20 nnz = {8, 4, 2, 1,.5} * 2 20 time vs n/nnz, log-log plot

39 Matrices over Semirings Matrix multiplication C = AB (or matrix/vector): C i,j = A i,1  B 1,j + A i,2  B 2,j + · · · + A i,n  B n,j Replace scalar operations  and + by  : associative, distributes over , identity 1  : associative, commutative, identity 0 annihilates under  Then C i,j = A i,1  B 1,j  A i,2  B 2,j  · · ·  A i,n  B n,j Examples: ( ,+) ; (and,or) ; (+,min) ;... Same data reference pattern and control flow

40 Remarks Tools for combinatorial methods built on parallel sparse matrix infrastructure Easy-to-use interactive programming environment –Rapid prototyping tool for algorithm development –Interactive exploration and visualization of data Sparse matrix * sparse matrix is a key primitive Matrices over semirings like (min,+) as well as (+,*)