1. Lecture 2b: Performance Metrics

2. Performance Metrics Measurable characteristics of a computer system: Count of an event Duration of a time interval Size of a parameter Rate: Operations executed per second

3. Performance Mertrics Clock Speed Clock speed/frequency (f): the rate of clock pulses (ex: 1GHz) Cycle time (T c ): time between two clock pulses (T c = 1/f) TcTc

4. Performance Mertrics Instruction Execution Rate Cycles per Instruction (CPI): is an average depends on the design of micro-architecture (hardwired/microprogrammed, pipelined) Number of instructions: is the number of instructions executed at runtime Depends on instruction set architecture (ISA) compiler CPI = CPI i : number of cycles required for instruction i I i : number of executed instructions of type i

5. Performance Metrics CPU Performance CPU time of a program (T) = instructions x cycles x time program instruction cycle CPI (cycles per instruction) T = instruction count x CPI x 1 f

6. Performance Metrics CPU Performance Drawbacks: In modern computers, no program runs without some operating system running on the hardware Comparing performance between machines with different operating systems will be unfair

7. Performance Metrics Execution time Elapsed time / wall-clock time / response time Latency to complete a task, including disk access, memory access, I/O, operating system overhead, and everything (includes time consumed by other programs in a time-sharing system) CPU time The time CPU is computing, not including I/O time or waiting time User time / user CPU time CPU time spent in the program System time / system CPU time CPU time spent in the operating system performing tasks requested by the program

8. Performance Metrics Performance Comparison Relative performance Performance x = 1. Execution time X Performance Ratio = Performance X = Execution time Y Performance Y Execution time X

9. Performance Metrics Relative Performance If workload consists of more than one program, total execution time may be used. If there are more than one machine to be compared, one of them must be selected as a reference.

10. Performance Metrics Throughput Total amount of work done in a given time Measured in tasks per time unit Can be used for Operating system performance Pipeline performance Multiprocessor performance

11. Performance Metrics MIPS (Million instructions per second) Includes both integer and floating point performance Number of instructions in a program varies between different computers Number of instructions varies between different programs on the same computer MIPS = Instruction count = Clock rate Execution time x 10 6 CPI x 10 6

12. Performance Metrics MFLOPS (Million floating point operations per second) Give performance of only floating-point operations Different mixes of integer and floating-point operations may have different execution times: Integer and floating-point units work independently Instruction and data caches provide instruction and data concurrently

13. Performance Metrics Utilization Speciality ratio 1  general purpose Utilization = Busy time. Total time Speciality ratio = Maximum performance. Minimum performance

14. Performance Metrics Asymptotic and Half performance r  – asymptotic performance n 1/2 – half performance T = r  (n + n 1/2 ) r  = 1/t n 1/2 = t 0 /t Slope = r  -1 t0t0 -n 1/2 n 1/2 2t 0

15. Performance Metrics Speedup Express how much faster is system 2 than system 1 Calculated directly from execution time Performance x = 1 = 1 Execution time X T X Speedup 2,1 = Performance 2 = T 1 Performance 1 T 2

16. Performance Metrics Relative Change It expresses the performance of system 2 relative to system 1 Performance x = 1 = 1 Execution time X T X Relative change 2,1 = Performance 2 - Performance 1 = T 1 - T 2 = Speedup 2,1 - 1 Performance 1 T 2

17. Performance Metrics Statistical Analysis Used to compare performance Workload consists of many programs Depends on the nature of the data as well as distribution of the test results

18. Performance Metrics Indices of Central Tendency Used to summarize multiple measurements Mean Median Mode

19. Performance Metrics Mean (average) Gives equal weight to all measurements Arithmetic mean =  x i, 1 ≤ i ≤ n n MeasurementExecution time X110 X220 X315 X418 X516 Mean15.8

20. Performance Metrics Median 1. Order all n measurements 2. The middle value is the median. If n is even, median is the mean of the middle 2 values Using Median instead of Mean reduces the skewing effect of the outliers. MeasurementExecution time X110 X220 X315 X418 X516 X6200 Mean46.5 MeasurementExecution time X110 X315 X516 X418 X220 X6200 Median = = 17

21. Performance Metrics Mode Mode is the value that occurs most frequently If all values occur once, there is no mode If there are several samples that all have the same value, there would be several modes MeasurementExecution time X110 X220 X336 X420 X520 X620 Mode = 20

22. Mean, Median, Mode Mean Incorporates information from the entire measured values Sensitive to outliers Median and Mode Less sensitive to outliers Do not effectively use all information ex

23. Performance Metrics Arithmetic mean (average) May be misleading if the data are skewed or scattered Arithmetic mean =  x i, 1 ≤ i ≤ n n MAMA MBMB MCMC Prog150100500 Prog2400800 Prog3555051004700 Average2000

24. Performance Metrics Weighted average weight is the frequency of each program in daily processing Results may change with a different set of execution frequencies Weighted average = ∑ w i. x i 1 ≤ i ≤ n weightMAMA MBMB MCMC Prog160%50100500 Prog230%400800 Prog310%555051004700 Average7058101010

25. Performance Metrics Geometric mean Results are stated in relation to the performance of a reference machine Geometric mean = (  x i ) 1/n, 1 ≤ i ≤ n MAMA Normalized to M A M B (reference) Normalized to M B MCMC Normalized to M C Prog150210015000.2 Prog240028001 1 Prog355500.925100147001.085 Average1.5410.60 Results are consistent no matter which system is chosen as reference

26. Performance Metrics Harmonic mean Used to compare performance results that are expressed as a rate (e.g. operations per second, throughput, etc.) Slowest rates have the greatest influence on the result  It identifies areas where performance can be improved Harmonic mean = n, 1 ≤ i ≤ n ∑ 1/x i

27. Performance Metrics Characteristics of a good performance metric If the time values are averaged, then the resulting mean value must be directly proportional to the total time. If the rate values are averaged, then the resulting mean value must be inversely proportional to the total time.

28. Performance Metrics Ex n benchmark programs T i is the execution time of program i F floating-point operations in each program M i = F / T i is the execution rate of program i (MFLOP/s) Arithmetic mean Inappropriate for summarizing rates T A = T A is directly proportional to the total execution time M A = = M A is inversely proportional to the total execution time

29. Performance Metrics Harmonic mean Inappropriate for summarizing execution times Appropriate for summarizing rates T H = T H is not directly proportional to the total execution time M H = = M H is inversely proportional to the total execution time

30. Performance Metrics Ex

31. Performance Metrics Geometric mean Inappropriate for summarizing execution times Inappropriate for summarizing rates T G = T G is not directly proportional to the total execution time M H = = M H is not inversely proportional to the total execution time

32. Performance Metrics Geometric mean Produces a consistent ordering of the systems but it is the wrong ordering M1M2M3 Prog1417244134 Prog28370 Prog366153135 Prog4394993352766000 Prog5772368369 Geometric mean (Normalized wrt S1)1.00.860.84 Geometric mean (Normalized wrt S2)1.171.00.99 Rank321 Total time407873436266798 Arithmetic mean8157687213342 Rank213

33. Performance Metrics Histogram Used to display the distribution of a set of measured values (variability) First find the minimum and maximum values. Then divide the range into b subranges, called cells.

34. Histogram Message size (kbytes) Network ANetwork B 0 < xi ≤ 51139 5 < xi ≤ 102725 10 < xi ≤ 154118 15 < xi ≤ 20325 20 < xi ≤ 252119 25 < xi ≤ 301242 30 < xi ≤ 3540

35. Performance Metrics Index of Dispersion Index of dispersion is used to compare the spread of measurements around the mean value Range is the simplest metric for an index of dispersion Range is sensitive to a few extreme values

36. Performance Metrics Index of Dispersion Maximum of the absolute values of the difference of each measurement from the mean It is also sensitive to extreme values

37. Performance Metrics Index of Dispersion Sample variance is the simplest metric for an index of dispersion Requires 2 passes through the data to calculate first x and then s 2 Requires 1 pass

38. Performance Metrics Index of Dispersion Standard deviation Coefficient of variance (COV): normalizes standard deviation wrt the mean