1. SC’07 7 th International APART Workshop Panel 11 November 2007 David Koester, Ph.D. dkoester@mitre.org

2. Panel Question "HPC Performance Metrics: Should We Drop FLOPS?" As computer architectures have advanced, making floating- point operations relatively inexpensive, it is still very ingrained in HPC to compare systems by their FLOPS rates, and to optimize algorithms to minimize their floating-point operation count even at the cost of increased memory bandwidth demands or programming effort. Should we drop FLOPS as a metric? If the answer is yes, then is there a way to gracefully (or perhaps suddenly) move to a metric other than FLOPS that has the predictive value and ease-of-analysis that FLOPS had in the 1960s and 1970s? The panelists will give their position on this question and their suggestions on the general issue of using simplistic measures like FLOPS as a performance metric.

3. Déjà Vu All Over Again Workshop on Performance Characterization, Modeling and Benchmarking for HPC Systems –Emeryville, CA 5-7 May 2003 Panel: Performance Metrics for HPCS: Holy Grail or Fata Morgana 1.Why don’t we have a good metric for HPC performance? 2.Is there any chance to define a single metric for HPC performance? Does everybody needs his/her own metric? 3.What are the requirements (theoretical, practical, and political) for such metrics? 4.What needs to be done to get new metrics accepted? Is it even possible? 5.Sustained vs Peak? Repercussions? 6.Implications of performance metrics on the political level

4. A Single “Good” Metric? (1 & 2) Parameters Performance (Theoretical Peak Capability) Peak Processor Performance # Processors System Memory Bisection BW Clock rate Ops/cycle # nodes Processors per node Parameter Space § (Examples) i.e., characteristics § Thanks to Thomas Sterling circa 2002

5. A Single “Good” Metric? (1 & 2) Parameters versus Metrics Performance (Theoretical Peak Capability) Peak Processor Performance # Processors System Memory Bisection BW Clock rate Ops/cycle # nodes Processors per node Parameter Space (Examples) Metric Space (Examples) User Applications Representative Benchmarks (HPC Challenge) Wall Clock Time Derived Metrics Sustained Flop/s Sustained (Memory) Bandwidth Sustained GUPS Sustained Intop/s

6. A Single “Good” Metric? (1 & 2) Why don’t we have a good metric for HPC performance? –We do have good metrics — well sort of… Wall clock time to solution for your application (or representative benchmark) Derived metrics of sustained performance for your application (or representative benchmark) –Flop/s, Intop/s, GUPS (Giga Updates per Second), Bandwidth –Because multiple derived metrics, can impose limitations with comparisons Maybe not… Is there any chance to define a single metric for HPC performance? –No, everybody needs his/her own — applications and workflows vary too greatly for a single performance metric Scientific calculations — sustained Flop/s Sustained GUPS Sustained transactions etc. –Yes, we must define a single metric The elevator sound bite metric The metric for the Gordon Bell award

7. As of 1 November 2007 –165 base runs –21 optimized runs http://icl.cs.utk.edu/hpcc/

8. Requirements for Metrics (3) “Demonstrate (not claim) performance” (Kuck) “You get what you measure” Theoretical –Stable –Quantifiable –Reasonable Practical –K.I.S.S. Political –Understandable by someone outside the community with limited time, limited knowledge of HPC, …, (who controls your funding)

9. Accepting New Metrics (4) What needs to be done? –View as a science Perform research and publish! Fund work in this area! –Work toward consensus Is it possible? –Most definitely!

10. Sustained or Peak? Repercussions (5) Sustained Performance  Honest  Application dependent Peak Performance  Dishonest  Misleading  Processing in HPC will be free! Peak Macho Flop/s % Peak

11. Metrics — The Layer 9 Protocol Implications (6) National level politics and the media Metric(s) need to be understandable –By individuals outside the community with limited time, limited knowledge of HPC, …, (who controls your funding) –The New York Times level of in- depth media coverage –The elevator soundbite  Need main stream acceptance and understanding of HPC metrics Business Metric(s) need to be useful –For procurement planning and decision-making by various organizations and business entities

12. HPC Performance Metrics: Should We Drop Flop/s? Clearly identify when you are referring to parameters versus metrics Drop Flops/? –Yes… –No… Simple alternative to compare systems? –Utilized bandwidth…

13. HPC Challenge Benchmarks and Multiprocessor Architecture

14. HPC Challenge Performance System Comparisons

15. New HPC Metrics? A Knight of The Order of the Grail bemusedly calls upon seekers of the Holy Grail to 'choose wisely.' Is utilized bandwidth the Grail? The true Grail was a simple wooden cup that stood out among the false golden, jewel encrusted grails — remember, choose wisely! Indiana Jones and the Last Crusade