1. Martin Schulz Center for Applied Scientific Computing Lawrence Livermore National Laboratory ASC STAT Team: Greg Lee, Dong Ahn (LLNL), Dane Gardner (LANL) Developed at LLNL, University of Wisconsin & University of New Mexico Lawrence Livermore National Laboratory, P. O. Box 808, Livermore, CA 94551 This work performed under the auspices of the U.S. Department of Energy by Lawrence Livermore National Laboratory under Contract DE-AC52-07NA27344 STAT: A Debugging Tool For Extreme Scale LLNL-PRES-426152

2. Lawrence Livermore National Laboratory STAT: Debugging Support at Scale  The debugging challenge at scale Traditional debuggers break down at scale Data and control for too many tasks Sequential paradigm  How can STAT help? Identify equivalence classes Pre-analysis for subset debugging  Typical use case Application hang (life or dead-lock) Answer the question: What is my code doing now?

3. Lawrence Livermore National Laboratory Stacktraces: The Basis for STAT

4. Lawrence Livermore National Laboratory Gathering Stack Traces  STAT gathers stack traces from Multiple processes Multiple samples per process MPI 2D Trace/Space Call Graph Prefix Tree 3D Trace/Space/Time Call Graph Prefix Tree

5. Lawrence Livermore National Laboratory Interpreting Stacktrace Trees Your Favorite Debugger Task 0 Task 1 Task 2

6. Lawrence Livermore National Laboratory STAT GUI

7. Lawrence Livermore National Laboratory Availability PlatformVer.UsageDocumentationPOC LLNL/TLCC OCF 0.9.4STATGUI STAT https://computing.llnl.gov/code/STAT/Greg Lee lee218@llnl.gov LLNL/TLCC SCF 0.9.4STATGUI STAT https://computing.llnl.gov/code/STAT/Greg Lee lee218@llnl.gov LLNL/uBGL0.9.0 beta STAThttps://computing.llnl.gov/code/STAT/Greg Lee lee218@llnl.gov LLNL/Dawn0.9.4 beta STATGUI STAT https://computing.llnl.gov/code/STAT/Greg Lee lee218@llnl.gov SNL/Glory0.9.2see belowhttps://computing.llnl.gov/code/STAT/Mahesh Rajan mrajan@sandia.gov LANL/Yellow Turing 0.9.1b Mod: hpc-tools Mod: stat man statconsult@lanl.gov LANL/Turquoise Lobo 0.9.2 Mod: hpc-tools Mod: stat man statconsult@lanl.gov Usage for SNL/Glory: module switch mpi mpi/mvapich-1.1_intel-11.1-f064-c064 module load /home/jgalaro/privatemodules/openss-mvapich Note: Red Storm has a poor-man STAT-like utility called fast_where. Try "man fast_where” for usage instructions.

8. Lawrence Livermore National Laboratory Usage Instructions  Option 1: Graphical User Interface Launch GUI: STATGUI Attach, create stacktraces & views through GUI  Option 2: Command line STAT  -t : number of traces  -T : time between traces Reports output file to stdout STATview  Additional information man STAT / STAT –h acroread /usr/local/tools/stat/doc/*.pdf

9. Lawrence Livermore National Laboratory Advanced Topics  Scalable Implementation Tree-based overlay networks  Data aggregation on the fly  Tree depth configurable Parameters to STAT Useful for 10,000+ tasks  Temporal Analysis Finer grain analysis of process location Disambiguation of iteration instances Employs static analysis to determine loop variables FE CP BE CP BE …

10. Lawrence Livermore National Laboratory Reference & Demo Session  Usage documentation https://computing.llnl.gov/code/STAT/  Man page man STAT or man STATGUI STAT -h  Background information http://www.paradyn.org/STAT/STAT.html  Demo Session / Track 3