1. mpiP Evaluation Report Hans Sherburne, Adam Leko UPC Group HCS Research Laboratory University of Florida

2. 2 Basic Information Name:  mpiP Developer:  Jeffrey Vetter (ORNL), Chris Chambreau (LLNL) Current versions:  mpiP v2.8 Website:  http://www.llnl.gov/CASC/mpip/ http://www.llnl.gov/CASC/mpip/ Contacts:  Jeffrey Vetter: vetterjs@ornl.govvetterjs@ornl.gov  Chris Chambreau: chcham@llnl.govchcham@llnl.gov

3. 3 mpiP: Lightweight, Scalable MPI Profiling mpiP is a simple lightweight tool for profiling Gathers information through the MPI profiling layer  Probably not good candidate to be extended for UPC or SHMEM Supports many platforms running Linux, Tru64, AIX, UNICOS, IBM BG/L Very simple to use, and output file is very easy to understand  Provides statistics for the top twenty MPI calls based on time spent in call, and total size of messages sent, also provides statistics for MPI I/O  Callsite traceback depth is variable to allow user to differentiate between and examine the behavior of routines that are wrappers for MPI calls A mpip viewer, Mpipview, is available as part of Tool Gear Some of its functionality is extended to developers through an API: stackwalking, address-to-source translation, symbol demangling, timing routines, accessing the name of the executable  These functions might be useful is source-code correlation is to be included in a UPC or SHMEM tool

4. 4 What is mpiP Useful For? The data collect by mpiP is useful for analyzing the scalability of parallel applications. By examining the aggregate time and rank correlation of the time spent in each MPI call versus the total time spent in MPI calls while increasing the number of tasks, one can locate flaws in load balancing and algorithm design. This technique is described in [1] “Statistical Scalability Analysis of Communication Operations in Distributed Applications” –Vetter, J. & McCracken, M. The following are courtesy of [1]:

5. 5 The Downside… mpiP does provide the measurements of aggregate callsite time, and total MPI call time necessary for computing the rank correlation coefficient mpiP does NOT automate the process of computing the rank correlation, which must utilize data from multiple experiments Equations for calculation of coefficients of correlation (linear and rank), care of [1]:

6. 6 Partial Sample of mpiP Output

7. 7 Information Provided by mpiP Information displayed in terms of task assignments, and callsites, which correspond to machines and MPI calls in source code arranged in the following sections:  Time per task (AppTime, MPITime, MPI%)  Location of callsite in source code (callsite, line#, parent function, MPI call)  Aggregate time per callsite (top twenty) (time, app%, MPI%, variance)  Aggregate sent message size per callsite (top twenty) (count, total, avg. MPI%)  Time statistics per callsite per task (all) (max, min, mean, app%, MPI%)  Sent message size statistics per callsite per task (all) (count, max, min, mean, sum)  I/O statistics per callsite per task (all) (count, max, min, mean, sum)

8. 8 mpiP Overhead

9. 9 Source Code Correlation in Mpipview

10. 10 Bottleneck Identification Test Suite Testing metric: what did profile data tell us? CAMEL: TOSS-UP  Profile showed that MPI time is a small percentage of overall application time  Profile reveals some imbalance in the amount of time spent in certain calls, but doesn’t help the user understand the cause  Profile does not provide information about what occurs when execution is not in MPI calls.  Difficult to grasp overall program behavior from profiling information alone NAS LU: TOSS-UP  Profile reveals that a MPI function calls consume a significant portion of application time  Profile reveals some imbalance in the amount of time spent in certain calls, but doesn’t help the user understand the cause  Profile does not provide information about what occurs when execution is not in MPI calls.  Difficult to grasp overall program behavior from profiling information alone

11. 11 Bottleneck Identification Test Suite (2) Big message: PASSED  Profile clearly shows that Send and Recv dominate the application time  Profiles shows a large number of bytes transfered Diffuse procedure: FAIL  Profile showed large amount of time spent in barrier  Time is diffused across processes  Profile does not show that in each barrier a single process is always delaying completion Hot procedure: FAIL  No profile output, due to no MPI calls (other than setup and breakdown Intensive server: PASSED  Profile showed one process spent very little time in MPI calls, while the remaining processes spent nearly all their time in Recvs  Profile showed one process sent an order of magnitude more data than the others, and spent far more time in Send Ping pong: PASSED  Profile showed time spent in MPI function calls dominated the total application time  Profile showed excessive number of Sends and Recvs with little load imbalance Random barrier: PASS  Profile shows that the majority of execution time is spent in Barrier called by processes not holding “potato” Small messages: PASS  Profile clearly shows single process spends almost all of the total application time in Recv, and recvs an excessive amount of messages sent by all the other processes System time: FAIL  No profile output, due to no MPI calls (other than setup and breakdown Wrong way: TOSS-UP  One process spends most of the execution time in sends the other spends most of the execution time in receives  Profile does not reveal the improperly ordered communication pattern

12. 12 Evaluation (1) Available metrics: 1/5  Only provides a handful statistical information about time, message size, and frequency of MPI calls  No hardware counter support Cost: free 5/5 Documentation quality: 4/5  Though brief (a single webpage), documentation adequately covers installation and available functionality Extensibility: 2/5  mpiP is designed to use the MPI profiling layer so they would not be readily adapted to UPC or SHMEM and so it would be of little use  The source code correlation functions work well Filtering and aggregation: 2/5  mpiP was designed to be lightweight, and presents statistics for the top twenty callsites  Output size grows with number of tasks (machines) Hardware support: 5/5  64-bit Linux (Itanium and Opteron), IBM SP (AIX), AlphaServer (Tru64), Cray X1, Cray XD1, SGI Altix, IBM BlueGene/L Heterogeneity support: 0/5 (not supported)

13. 13 Evaluation (2) Installation: 5/5  About as easy as you could expect Interoperability: 1/5  mpiP has it’s own output format Learning curve: 4/5  Easy to use  Simple statistics are easily understood Manual overhead: 1/5  All MPI calls automatically instrumented for you when linking against mpiP library  No way to turn on/off tracing in places without relinking Measurement accuracy: 4/5  CAMEL overhead: ~5%  Correctness of programs is not affected  Overhead is low (less than 7% for all test suite programs)

14. 14 Evaluation (3) Multiple executions: 0/5 (not supported) Multiple analyses & views: 2/5  Statistics regarding MPI calls are displayed in output file  Source code location to callsite correlation provided by Mpipview Performance bottleneck identification: 2.5/5  No automatic methods supported  Some bottlenecks could be deduced by examining gathered statistics  Lack of trace information makes some bottlenecks impossible to detect Profiling/tracing support: 2/5  Only supports profiling  Profiling can be enabled for various regions of code by editing source code  Turning on/off profiling requires recompilation (a runtime environment variable for deactivating profiling is given in documentation, and acknowledged in the profile output file when set, but profiling is not disabled)

15. 15 Evaluation (4) Response time: 3/5  No results until after run  Quickly assembles report at the end of experimentation run Searching: 0/5 (not supported) Software support: 3/5  Supports C, C++, Fortran  Supports a large number of compilers  Tied closely to MPI applications Source code correlation: 4/5  Line numbers of source code provided for each MPI callsites in output file  Automatic source code correlation provided by Mpipview System stability: 5/5  mpiP and Mpipview work very reliably Technical support: 5/5  Co-author, Chris Chambreau, responded quickly, and provided good information allowing us to correct a problem with one of our benchmark apps