1. Blaise Barney, LLNL ASC Tri-Lab Code Development Tools Workshop Thursday, July 29, 2010 Lawrence Livermore National Laboratory, P. O. Box 808, Livermore, CA 94551 This work performed under the auspices of the U.S. Department of Energy by Lawrence Livermore National Laboratory under Contract DE-AC52-07NA27344 mpiP: Lightweight MPI profiling LLNL-PRES-426112

2. Lawrence Livermore National Laboratory Simple, scalable, portable MPI profiling  How to simplify initial performance analysis Simple, reliable, low learning overhead Focus on important, yet easy to access information  mpiP MPI profiling library provides a portable, direct, low-overhead tool for measuring MPI behavior  To use mpiP: First: "use mpip" re-link with (or pre-load) the mpiP library run the application examine summary or detailed results  text report  GUI browser

3. Lawrence Livermore National Laboratory How does it work?  MPI calls are intercepted by mpiP wrappers  mpiP wrappers call PMPI functions  All information is task-local  Report generation is done within MPI_Finalize arbitrarily (by all tasks)  Task data can be collected quickly by a single collector task or by using MPI collectives

4. Lawrence Livermore National Laboratory Usage  First: "use mpip"  Re-link application with mpiP library. Example: -L/usr/local/tools/mpip/lib –lmpiP –lbfd –liberty -lunwind  Run-time insertion srun-mpip or mpirun-mpip script LD_PRELOAD=/usr/local/tools/mpip/lib/libmpiP.so  Run-time parameters set in MPIP environment variable. Some examples: FlagEffectDefault -k [#]Stack trace depth for MPI call sites1 -t [#]MPI time % thresholdNo filtering -cConcise report (omit call site detail)Verbose

5. Lawrence Livermore National Laboratory  Various report formats Verbose, task-specific information Concise, omits task-specific detail  Modify report output Use MPIP environment variable  Filter MPI call sites  Select concise or verbose format or both  Omit task-specific data How to examine performance data?

6. Lawrence Livermore National Laboratory How to examine performance data?  Various report sections Per-task MPI use

7. Lawrence Livermore National Laboratory How to examine performance data?  Various report sections Indexed list of MPI routine call locations

8. Lawrence Livermore National Laboratory How to examine performance data?  Various report sections Concise, top 20 MPI call sites, by wall clock time

9. Lawrence Livermore National Laboratory How to examine performance data?  Various report sections Concise, top 20 MPI call sites, by message volume

10. Lawrence Livermore National Laboratory  Various report sections Verbose, task-specific information, by wall clock time (shown) and message volume (similar, but not shown) How to examine performance data?

11. Lawrence Livermore National Laboratory How to examine performance data?  View report files using the Tool Gear GUI browser. "use mpipview" then "mpipview mpipoutputfile"

12. Lawrence Livermore National Laboratory  LLNL “use mpip” first Library in /usr/local/tools/mpip/lib Supported on most production platforms  SNL Module load tools/mpiP-3.2.0 /projects/tools_workshop/mpiP Availability

13. Lawrence Livermore National Laboratory Advanced Topics  MPI_Pcontrol can control mpiP behavior Example C: MPI_Pcontrol(3); MPI_Pcontrol(2); ArgumentEffect 0Disable profiling 1Enable Profiling 2Reset all call site data 3Generate verbose report 4Generate concise report

14. Lawrence Livermore National Laboratory References  Initially developed by Jeffrey Vetter and Michael McCracken at LLNL  Version 3.2 released, includes RMA support  Other project members Philip Roth (ORNL) Jeffrey Vetter (ORNL)  Tool Gear: John Gyllenhaal (LLNL) & John May (LLNL) http://computation.llnl.gov/casc/tool_gear/  Tool POC : Chris Chambreau chcham@llnl.govchcham@llnl.gov  Tool mail list : mpip-help@lists.sourceforge.netmpip-help@lists.sourceforge.net  Tool web site: http://mpip.sourceforge.net/http://mpip.sourceforge.net/