Copyright 2004 David J. Lilja1 Measuring Computer Performance SUMMARY
Copyright 2004 David J. Lilja2 Fundamental Solution Techniques Measurement Simulation Analytical modeling
Copyright 2004 David J. Lilja3 Performance Metrics Characteristics of good metrics Processor and system metrics Speedup and relative change
Copyright 2004 David J. Lilja4 Measurement Tools and Techniques Strategies Interval timers Program profiling Tracing Indirect measurement
Copyright 2004 David J. Lilja5 Statistical Interpretations of Measured Data What do all of these means mean? Sources of measurement errors Confidence intervals Statistically comparing alternatives
Copyright 2004 David J. Lilja6 Design of Experiments Terminology One-factor ANOVA Two-factor ANOVA Generalized m-factor experiments Fractional factorial designs m2 n designs Multifactorial designs Plackett and Burman
Copyright 2004 David J. Lilja7 Simulation Types of simulations Random number generation Verification and validation
Copyright 2004 David J. Lilja8 References Sources of additional information
Copyright 2004 David J. Lilja9 Performance Bookshelf Suggested books on computer systems performance measurement and analysis Comprehensive performance books Experimental design Modeling and queuing analysis Simulation and random number generation Software suggestions and reference books
Copyright 2004 David J. Lilja10 References Comprehensive Performance Analysis David J. Lilja, Measuring Computer Performance: A Practitioner's Guide, Cambridge University Press, 2000, Experimental Design Joshua J. Yi, David J. Lilja, and Douglas M. Hawkins, “A Statistically Rigorous Approach for Improving Simulation Methodology,” International Symposium on High-Performance Computer Architecture (HPCA), February, R. Plackett and J. Burman, “The Design of Optimum Multifactorial Experiments,” Biometrika, Vol. 33, Issue 4, June, 1946, pp D. C. Montgomery, Design and Analysis of Experiments (5 th ed), Wiley & Sons, 2000,
Copyright 2004 David J. Lilja11 References MinneSPEC AJ KleinOsowski and David J. Lilja, “MinneSPEC: A New SPEC Workload for Simulation-Based Computer Architecture Research,” Computer Architecture Letters, Vol. 1, June, 2002, pp L. Eeckhout et al, “Designing Computer Architecture Workloads,” IEEE Computer, Feb., 2003, pp Sampling J. Haskins and K. Skadron, “Minimal Subset Evaluation: Rapid Warm-up for Simulated Hardware State,” Intl. Conf. Computer Design, R. E. Wunderlich, T. F. Wenisch, B. Falsafi, J. C. Hoe, “SMARTS: Accelerating Microarchitecture Simulation via Rigorous Statistical Sampling,” Intl. Symp. Computer Architecture, 2003, pp
Copyright 2004 David J. Lilja12 References SimPoint T. Sherwood, E. Perelman, G. Hamerly, and B. Calder, “Automatically Characterizing Large Scale Program Behavior,” Intl. Conf. Architectural Support for Programming Languages and Operating Systems, 2002.
Copyright 2004 David J. Lilja13 “Measurements are not to provide numbers but insights.” Ingrid Bucher Questions?