Extreme Computing’05 Parallel Graph Algorithms: Architectural Demands of Pathological Applications Bruce Hendrickson Jonathan Berry Keith Underwood Sandia National Labs Richard Murphy Notre Dame University

Extreme Computing’05 Confessions of a Message-Passing Snob Bruce Hendrickson Sandia National Labs

Discrete Algorithms & Math Department Extreme Computing’05 Why Graphs? l Exemplar of memory-intensive application l Widely applicable and can be very large scale »Scientific computing –sparse direct solvers –preconditioning –radiation transport –mesh generation –computational biology, etc. »Informatics –data-centric computing –encode entities & relationships –look for patterns or subgraphs

Discrete Algorithms & Math Department Extreme Computing’05 Characteristics l Data patterns »Moderately structured for scientific applications –Even unstructured grids make “nice” graphs –Good partitions, lots of locality on multiple scales »Highly unstructured for informatics –Similar to random, power-law networks –Can’t be effectively partitioned l Algorithm characteristics »Typically, follow links of edges –Maybe many at once – high level of concurrency –Highly memory intensive l Random accesses to global memory – small fetches l Next access depends on current one l Minimal computation

Discrete Algorithms & Math Department Extreme Computing’05 Shortest Path Illustration

Discrete Algorithms & Math Department Extreme Computing’05 Architectural Challenges l Runtime is dominated by latency l Essential no computation to hide memory costs l Access pattern is data dependent »Prefetching unlikely to help »Often only want small part of cache line l Potentially abysmal locality at all levels of memory hierarchy

Discrete Algorithms & Math Department Extreme Computing’05 Caching Futility

Discrete Algorithms & Math Department Extreme Computing’05 Larger Blocks are Expensive

Discrete Algorithms & Math Department Extreme Computing’05 Properties Needed for Good Graph Performance l Low latency / high bandwidth »For small messages! l Latency tolerant l Light-weight synchronization mechanisms l Global address space »No graph partitioning required »Avoid memory-consuming profusion of ghost-nodes l These describe Burton Smith’s MTA!

Discrete Algorithms & Math Department Extreme Computing’05 MTA Introduction l Latency tolerance via massive multi-threading »Each processor has hardware support for 128 threads »Context switch in a single tick »Global address space, hashed to reduce hot-spots »No cache. Context switch on memory request. »Multiple outstanding loads l Good match for applications which: »Exhibit complex memory access patterns »Aren’t computationally intensive (slow clock) »Have lots of fine-grained parallelism l Programming model »Serial code with parallelization directives »Code is cleaner than MPI, but quite subtle »Support for “future” based parallelism

Discrete Algorithms & Math Department Extreme Computing’05 Case Study – Shortest Path l Compare codes optimized for different architectures l Option 1: Distributed Memory CompNets »Run on Linux cluster: 3GHz Xenons, Myrinet network »LLNL/SNL collaboration – just for short path finding »Finalist for Gordon Bell Prize on BlueGene/L »About 1100 lines of C code l Option 2: MTA parallelization »Part of general-purpose graph infrastructure »About 400 lines of C++ code

Discrete Algorithms & Math Department Extreme Computing’05 Short Paths on Erdos-Renyi Random Graphs (V=32M, E=128M)

Discrete Algorithms & Math Department Extreme Computing’05 Connected Components on MTA-2 Power-Law Graph V=34M, E=235M procstime

Discrete Algorithms & Math Department Extreme Computing’05 Remarks l Single processor MTA competitive with current micros, despite 10x clock difference l Excellent parallel scalability for MTA on range of graph problems »Identical to single processor code l Eldorado is coming next year »Hybrid of MTA & Red Storm »Less well balanced, but affordable

Discrete Algorithms & Math Department Extreme Computing’05 Broader Lessons l Space of important apps is broader than PDE solvers »Data-centric applications may be quite different from traditional scientific simulations l Architectural diversity is important »No single architecture can do everything well l As memory wall gets steeper, latency tolerance will be essential for more and more applications l High level of concurrency requires »Latency tolerance »Fine-grained synchronization