1. TAU Parallel Performance System
Allen D. Malony, Sameer S. Shende, Robert Bell Kai Li, Li Li, Kevin Huck, Nick Trebon
Department of Computer and Information Science
Performance Research Laboratory
University of Oregon

2. Outline Motivation TAU architecture and toolkit Example applications
Instrumentation
Measurement
Analysis
Example applications
Users of TAU
Conclusion

3. Problem Domain ASCI defines leading edge parallel systems and software
Large-scale systems and heterogeneous platforms
Multi-model simulation
Complex, multi-layered software integration
Multi-language programming
Mixed-model parallelism
Complexity challenges performance analysis tools
System diversity demands tool portability
Need for cross- and multi-language support
Coverage of alternative parallel computation models
Operate at scale

4. Tools for Performance Problem Solving
Empirical-based performance optimization process
Understand performance technology concerns
characterization
Performance Tuning
Performance
Diagnosis
Experimentation
Observation
hypotheses
properties
Experiment management
Performance database
Performance Technology
Instrumentation
Measurement
Analysis
Visualization
Performance Technology

5. TAU Performance System
Tuning and Analysis Utilities (11+ year project effort)
Performance system framework for scalable parallel and distributed high-performance computing
Targets a general complex system computation model
Entities: nodes / contexts / threads
Multi-level: system / software / parallelism
Measurement and analysis abstraction
Integrated toolkit for performance instrumentation, measurement, analysis, and visualization
Portable performance profiling and tracing facility
Open software approach with technology integration
University of Oregon , Forschungszentrum Jülich, LANL

6. TAU Performance Systems Goals
Multi-level performance instrumentation
Multi-language automatic source instrumentation
Flexible and configurable performance measurement
Widely-ported parallel performance profiling system
Computer system architectures and operating systems
Different programming languages and compilers
Support for multiple parallel programming paradigms
Multi-threading, message passing, mixed-mode, hybrid
Support for performance mapping
Support for object-oriented and generic programming
Integration in complex software systems and applications

7. General Complex System Computation Model
Node: physically distinct shared memory machine
Message passing node interconnection network
Context: distinct virtual memory space within node
Thread: execution threads (user/system) in context
Interconnection Network *
Inter-node message communication *
Node
Node
Node
Should we describe different parallel programming / computation models?
physical view
memory
node memory
SMP
memory VM
space …
model view …
Context
Threads

8. TAU Performance System Architecture
EPILOG
Paraver
ParaProf

9. TAU Instrumentation Approach
Support for standard program events
Routines
Classes and templates
Statement-level blocks
Support for user-defined events
Begin/End events (“user-defined timers”)
Atomic events
Selection of event statistics
Support definition of “semantic” entities for mapping
Support for event groups
Instrumentation optimization

10. TAU Instrumentation
Flexible instrumentation mechanisms at multiple levels
Source code
manual
automatic
C, C++, F77/90/95 (Program Database Toolkit (PDT))
OpenMP (directive rewriting (Opari), POMP spec)
Object code
pre-instrumented libraries (e.g., MPI using PMPI)
statically-linked and dynamically-loaded (e.g., Python)
Executable code
dynamic instrumentation (pre-execution) (DynInstAPI)
virtual machine instrumentation (e.g., Java using JVMPI)

11. TAU Source Instrumentation
Automatic source instrumentation (TAUinstr)
Routine entry/exit and class method entry/exit
Block entry/exit and statement level (to be added)
Uses an instrumentation specification file
Include/exclude list for events and files
Uses command line options for group selection
Instrumentation event selection (TAUselect)
Automatic generation of instrumentation specification file
Instrumentation language to describe event constraints
Event identity and location
Event performance properties (e.g., overhead analysis)
Create TAUselect scripts for performance experiments

12. Program Database Toolkit (PDT)
Program code analysis framework
develop source-based tools
High-level interface to source code information
Integrated toolkit for source code parsing, database creation, and database query
Commercial grade front-end parsers
Portable IL analyzer, database format, and access API
Open software approach for tool development
Multiple source languages
Implement automatic performance instrumentation tools
tau_instrumentor

13. Program Database Toolkit (PDT)
Application
/ Library
C / C++
parser
Fortran parser
F77/90/95
Program
documentation
PDBhtml
Application
component glue IL IL
SILOON
C / C++
IL analyzer
Fortran
IL analyzer
C++ / F90/95
interoperability
CHASM
Program
Database
Files
Automatic source
instrumentation
DUCTAPE
TAU_instr

14. PDT 3.0 Functionality C++ statement-level information implementation
for, while loops, declarations, initialization, assignment…
PDB records defined for most constructs
DUCTAPE
Processes PDB 1.x, 2.x, 3.x uniformly
PDT applications
XMLgen
PDB to XML converter (Sottile)
Used for CHASM and CCA tools
PDBstmt
Statement callgraph display tool

15. PDT 3.0 Functionality (continued)
Cleanscape Flint parser fully integrated for F90/95
Flint parser is very robust
Produces PDB records for TAU instrumentation (stage 1)
Linux x86, HP Tru64, IBM AIX
Tested on SAGE, POP, ESMF, PET benchmarking codes
Full PDB 2.0 specification (stage 2) [Q1 ‘04]
Statement level support (stage 3) [Q3 ‘04]
PDT 3.0 release at SC2003

16. TAU Performance Measurement
TAU supports profiling and tracing measurement
Robust timing and hardware performance support
Support for online performance monitoring
Profile and trace performance data export to file system
Selective exporting
Extension of TAU measurement for multiple counters
Creation of user-defined TAU counters
Access to system-level metrics
Support for callpath measurement
Integration with system-level performance data
Linux MAGNET/MUSE (Wu Feng, LANL)

17. TAU Measurement with Multiple Counters
Extend event measurement to capture multiple metrics
Begin/end (interval) events
User-defined (atomic) events
Multiple performance data sources can be queried
Associate counter function list to event
Defined statically or dynamically
Different counter sources
Timers and hardware counters
User-defined counters (application specified)
System-level counters
Monotonically increasing required for begin/end events
Extend user-defined counters to system-level counter

18. Performance Analysis and Visualization
Analysis of parallel profile and trace measurement
Parallel profile analysis
ParaProf
ParaVis
Profile generation from trace data
Performance database framework (PerfDBF)
Parallel trace analysis
Translation to VTF 3.0 and EPILOG
Integration with VNG (Technical University of Dresden)
Online parallel analysis and visualization

19. ParaProf Framework Architecture
Portable, extensible, and scalable tool for profile analysis
Try to offer “best of breed” capabilities to analysts
Build as profile analysis framework for extensibility

20. Profile Manager Window

21. Pprof Output (NAS Parallel Benchmark – LU)
Intel Quad PIII Xeon
F90 + MPICH
Profile - Node - Context - Thread
Events - code - MPI

22. ParaProf (NAS Parallel Benchmark – LU)
Routine profile across all nodes
node,context, thread
Global profiles
Event legend
Individual profile

23. TAU + Vampir (NAS Parallel Benchmark – LU)
Callgraph display
Timeline display
Parallelism display
Communications display

24. Case Study: SAMRAI (LLNL)
Structured Adaptive Mesh Refinement Application Infrastructure (SAMRAI)
Programming
C++ and MPI
SPMD
Instrumentation
PDT for automatic instrumentation of routines
MPI interposition wrappers
SAMRAI timers for interesting code segments
timers classified in groups (apps, mesh, …)
timer groups are managed by TAU groups

25. Full Profile Window (Exclusive Time)
512 processes

26. Node / Context / Thread Profile Window

27. Derived Metrics

28. Full Profile Window (Metric-specific)
512 processes

29. ParaProf Enhancements
Readers completely separated from the GUI
Access to performance profile database
Profile translators
mpiP, papiprof, dynaprof
Callgraph display
prof/gprof style with hyperlinks
Integration of 3D performance plotting library
Scalable profile analysis
Statistical histograms, cluster analysis, …
Generalized programmable analysis engine
Cross-experiment analysis

30. ParaVis Scalable parallel profile analysis
Scalable performance displays
3D graphics
Analysis across profile samples
Allow for runtime use
Animated / interactive visualization
Initially develop with SCIRun
Computational environment
Performance graphics toolkit
Portable plotting library
OpenGL
Performance
Visualizer
Performance
Analyzer
Performance
Data Reader

31. Performance Visualization in SCIRun
SCIRun program
EVH1, IBM
EVH1, Linux IA-32

32. “Terrain” Visualization (Full profile)
Uintah

33. “Scatterplot” Visualization
Each point coordinate determined by three values:
MPI_Reduce
MPI_Recv
MPI_Waitsome
Min/Max value range
Effective for cluster analysis
Uintah

34. “Bargraph” Visualization (MPI routines)
Uintah, 512 processes, ASCI Blue Pacific

35. TAU Performance Database Framework
Performance analysis programs
Performance
data description
Raw performance data
Other tools
PerfDML
translators
Performance analysis
and query toolkit
. . .
ORDB
PostgreSQL
PerfDB
profile data only
XML representation
project / experiment / trial

36. PerfDBF Browser

37. PerfDBF Cross-Trial Analysis

38. TAU Performance System Status
Computing platforms (selected)
IBM SP / pSeries, SGI Origin 2K/3K, Cray T3E / SV-1 / X1, HP (Compaq) SC (Tru64), Sun, Hitachi SR8000, NEC SX-5/6, Linux clusters (IA-32/64, Alpha, PPC, PA-RISC, Power, Opteron), Apple (G4/5, OS X), Windows
Programming languages
C, C++, Fortran 77/90/95, HPF, Java, OpenMP, Python
Thread libraries
pthreads, SGI sproc, Java,Windows, OpenMP
Compilers (selected)
Intel KAI (KCC, KAP/Pro), PGI, GNU, Fujitsu, Sun, Microsoft, SGI, Cray, IBM (xlc, xlf), Compaq, NEC, Intel

39. Selected Applications of TAU
Center for Simulation of Accidental Fires and Explosion
University of Utah, ASCI ASAP Center, C-SAFE
Uintah Computational Framework (UCF) (C++)
Center for Simulation of Dynamic Response of Materials
California Institute of Technology, ASCI ASAP Center
Virtual Testshock Facility (VTF) (Python, Fortran 90)
Los Alamos National Lab
Monte Carlo transport (MCNP) (Susan Post)
Full code automatic instrumentation and profiling
ASCI Q validation and scaling
SAIC’s Adaptive Grid Eulerian (SAGE) (Jack Horner)
Fortran 90 automatic instrumentation and profiling

40. Selected Applications of TAU (continued)
Lawrence Livermore National Lab
Overturen
Radiation diffusion (KULL)
C++ automatic instrumentation, callpath profiling
Sandia National Lab
DOE CCTTSS SciDAC project
Common component architecture (CCA) integration
Combustion code (C++, Fortran 90, GrACE, MPI)
Center for Astrophysical Thermonuclear Flashes
University of Chicago / Argonne, ASCI ASAP Center
FLASH code (C, Fortran 90, MPI)

41. Concluding Remarks
Complex ASCI parallel systems and software pose challenging performance analysis problems that require robust methodologies and tools
To build more sophisticated performance tools, existing proven performance technology must be utilized
Performance tools must be integrated with software and systems models and technology
Performance engineered software
Function consistently and coherently in software and system environments
TAU performance system offers robust performance technology that can be broadly integrated

42. Acknowledgements Department of Energy (DOE)
MICS office
DOE 2000 ACTS contract
“Performance Technology for Tera-class Parallel Computer Systems: Evolution of the TAU Performance System”
“Performance Analysis of Parallel Component Software”
University of Utah, DOE ASCI Level 1 sub-contract
DOE ASCI Level 3 (LANL, LLNL)
NSF National Young Investigator (NYI) award
Research Centre Juelich
John von Neumann Institute for Computing
Dr. Bernd Mohr
Los Alamos National Laboratory