1. Copyright 2004 David J. Lilja1 Measuring Computer Performance SUMMARY

2. Copyright 2004 David J. Lilja2 Fundamental Solution Techniques Measurement Simulation Analytical modeling

3. Copyright 2004 David J. Lilja3 Performance Metrics Characteristics of good metrics Processor and system metrics Speedup and relative change

4. Copyright 2004 David J. Lilja4 Measurement Tools and Techniques Strategies Interval timers Program profiling Tracing Indirect measurement

5. Copyright 2004 David J. Lilja5 Statistical Interpretations of Measured Data What do all of these means mean? Sources of measurement errors Confidence intervals Statistically comparing alternatives

6. Copyright 2004 David J. Lilja6 Design of Experiments Terminology One-factor ANOVA Two-factor ANOVA Generalized m-factor experiments Fractional factorial designs m2 n designs Multifactorial designs Plackett and Burman

7. Copyright 2004 David J. Lilja7 Simulation Types of simulations Random number generation Verification and validation

8. Copyright 2004 David J. Lilja8 References Sources of additional information

9. Copyright 2004 David J. Lilja9 Performance Bookshelf Suggested books on computer systems performance measurement and analysis Comprehensive performance books Experimental design Modeling and queuing analysis Simulation and random number generation Software suggestions and reference books http://labq.com/bookstore.shtml

10. Copyright 2004 David J. Lilja10 References Comprehensive Performance Analysis David J. Lilja, Measuring Computer Performance: A Practitioner's Guide, Cambridge University Press, 2000, http://labq.com/perf-book.shtml. Experimental Design Joshua J. Yi, David J. Lilja, and Douglas M. Hawkins, “A Statistically Rigorous Approach for Improving Simulation Methodology,” International Symposium on High-Performance Computer Architecture (HPCA), February, 2003. R. Plackett and J. Burman, “The Design of Optimum Multifactorial Experiments,” Biometrika, Vol. 33, Issue 4, June, 1946, pp. 305-325. D. C. Montgomery, Design and Analysis of Experiments (5 th ed), Wiley & Sons, 2000, http://labq.com/bookstore.shtml#experiments.

11. Copyright 2004 David J. Lilja11 References MinneSPEC AJ KleinOsowski and David J. Lilja, “MinneSPEC: A New SPEC Workload for Simulation-Based Computer Architecture Research,” Computer Architecture Letters, Vol. 1, June, 2002, pp. 10-13. http://www.arctic.umn.edu/~lilja/minnespec/ http://www.arctic.umn.edu/~lilja/minnespec/ L. Eeckhout et al, “Designing Computer Architecture Workloads,” IEEE Computer, Feb., 2003, pp. 65-71. Sampling J. Haskins and K. Skadron, “Minimal Subset Evaluation: Rapid Warm-up for Simulated Hardware State,” Intl. Conf. Computer Design, 2001. R. E. Wunderlich, T. F. Wenisch, B. Falsafi, J. C. Hoe, “SMARTS: Accelerating Microarchitecture Simulation via Rigorous Statistical Sampling,” Intl. Symp. Computer Architecture, 2003, pp. 84-95.

12. Copyright 2004 David J. Lilja12 References SimPoint T. Sherwood, E. Perelman, G. Hamerly, and B. Calder, “Automatically Characterizing Large Scale Program Behavior,” Intl. Conf. Architectural Support for Programming Languages and Operating Systems, 2002.

13. Copyright 2004 David J. Lilja13 “Measurements are not to provide numbers but insights.” Ingrid Bucher Questions?