A Case Study of HPC Metrics Collection and Analysis Philip Johnson and Michael Paulding, University of Hawaii, Honolulu, Hawaii. Goals of the case study Provide a complete implementation of one Purpose Based Benchmark problem definition, called Optimal Truss Design Implement the Optimal Truss Design system in C++ using MPI on a 240 node Linux cluster at the University of Hawaii Develop and evaluate automated support for HPC process and product measurement using Hackystat Assess the utility of the metrics for understanding HPC development Metrics Collected Size (Number of files, Total SLOC, “Parallel” SLOC containing an MPI directive, “Serial” SLOC not containing an MPI directive, Test code) Active Time (amount of time spent editing Optimal Truss Design files) Performance (wall clock time on 1, 2, 4, 8, 16, and 32 processors) Milestone Tests (indicates functional completeness) Command Line Invocations Results: Basic Process and Product Measures Total Source Lines of Code3320 LOC Total Test Lines of Code901 LOC Total MPI Lines of Code1032 LOC Total Days (Calendar time)1 year Total Days (with Development Activity)88 days Total Active Time152 Hours Total Distinct MPI Directives60 Directives Total Files56 Files Total Sequential Files (no MPI)51 Files Total Parallel Files (containing MPI)5 Files Execution Time126 sec. (1 processor) 66 sec. (2 processors) 33 sec. (4 processors) 27 sec. (8 processors) 39 sec. (16 processors) 43 sec. (32 processors) Results: Derived Process and Product Measures Derived MetricDefinitionValue Productivity Proxy(LOC / Active Time)22 LOC/hour Average Daily Active Time(Total Active Time / Total Days)1.73 hours/day Test Code Density Percentage(Total Test LOC / Total LOC)27% MPI Code Density Percentage(Total MPI LOC / Total LOC)31% MPI File Density Percentage(Total MPI Files / Total Files)9% MPI Directive Frequency Ratio(Total MPI Directives : Total MPI LOC)1 Directive : 17 LOC Speedup(Execution Time (1 Proc.) / Exec. Time (n processors) 1.0 (1 processor) 1.9 (2 processors) 3.7 (4 processors) 4.5 (8 processors) 3.2 (16 processors) 2.9 (32 processors) Results: Process and Product Telemetry Charts For More Information Understanding HPCS development through automated process and product measurement with Hackystat, Philip M. Johnson and Michael G. Paulding, Proceedings of the Second Workshop on Productivity and Performance in High-End Computing. Results: Daily Diary with CLI and Most Active File Insights and Lessons Learned Productivity (22 LOC/hour) and test code density (27%) seem in line with traditional software engineering metrics. Speedup data indicates almost linear speedup to 4 processors, then falls off sharply, indicating that current solution is not scalable. Parallel and serial LOC were equal at start of project, then most effort was devoted to serial code, with some final enhancements to parallel code at end of project. Performance data was not comparable over course of project (only final numbers available; no telemetry) Hackystat provides effective infrastructure for collection of process and product metrics. This case study provides useful baseline data to compare with future studies. Future research: Compare to OpenMP or JavaParty implementation. Gather metrics while improving scalability of system Compare metrics against other application types. Analyze CLI data for patterns, bottlenecks Thanks to our sponsors