1. Chapter 1 Section 1.4 Dr. Iyad F. Jafar Evaluating Performance

2. Outline Introduction CPU Execution Time The Performance Equation Determinants of Performance SPEC Benchmark Other Performance Metrics Examples 2

3. Introduction 3 Given a collection of computers, which one to buy? Best Performance ? Least Cost ? Best Cost/Performance? How to define performance? Time required to finish a task (individual users) Number of tasks executed per time (throughput) Less time to finish implies better performance If X is n times faster than Y, then Performance = 1 / Execution Time

4. CPU Execution Time 4 How to measure the execution time? Almost all modern computers are based on a clock The clock is a periodic square wave with known period (cycle time) Period = 1 / Frequency The base unit in measuring time is the cycle time, thus one clock cycle Time = cycles * cycle time Time = cycles / clock frequency

5. CPU Execution Time 5 Time required to execute a program is essentially the time required to execute its instructions! However, not all instructions take the same time! One way to think about execution time is that it equals the number of instructions executed multiplied by the average time per instruction Time = #instructions x cycle time Time = #instructions x average cycles per instruction x cycle time Time = IC x CPI x CC

6. CPU Execution Time 6 The average CPI is computed by Where l Where IC k is the number of instructions of class k executed l CPI k is the number of clock cycles per instruction for that instruction class l N is the number of instruction classes Note: The overall effective CPI varies by instruction mix – a measure of the dynamic frequency of instructions across one or many programs

7. The Performance Equation 7 Notes Three key factors for performance: IC, CPI, and CC CC: The clock rate is usually given IC: Overall instruction count (executed instructions) by using profilers/ simulators CPI: varies by instruction type and ISA implementation

8. The Performance Equation Example 1. In a certain program 1000 instructions were executed on CPU running at 1 GHz. If the instruction counts and CPI for each class are given below, how long does it take to execute the program? 8 Instruction Class Instruction Count Class CPI 12002 23003 35001 Effective CPI = (200x2+300x3+500x1)/1000 = 1.8 Time = 1000 x 1.8 x 1 ns = 1.8 us

9. The Performance Equation Example 2. Suppose the computer A has clock cycle of 250 ps and CPI 2.0 for some program, and computer B has clock cycle time of 500 ps and CPI of 1.2 for the same program, then which computer is faster ? 9 Time A = IC x 2 x 250 ps = 500 IC ps Time B = IC x 1.2 x 500 ps = 600 IC ps Performance A Time B 600 IC Performance B Time A 500 IC -------------------- = ---------- = --------- = 1.2 Computer A is 1.2 faster than B

10. The Performance Equation Example 3. A certain processor that has four instruction classes is to be modified using different approaches. The details of the program used in evaluating different approaches are given in the table below. What is the effective CPI for The original processor Approach 1. A cache is added and it reduces the average load time to 2 cycles. Approach 2. A branch prediction scheme is used and it cuts the branch time by 1 cycle. Approach 3. A second ALU is added to execute two ALU instructions at once. 10 ClassFrequencyClass CPI ALU50%1 Load20%5 Store10%3 Branch20%2 CPI k x F 0.5 1.0 0.3 0.4 CPI k x F 0.5 0.4 0.3 0.4 CPI k x F 0.5 1.0 0.3 0.2 CPI k x F 0.25 1.0 0.3 0.4 2.21.62.01.95Effective CPI Original App1App2App3 Speed up 1.3751.101.128

11. ICCPICC Algorithm Programming Language Compiler ISA Processor Organization Technology Determinants of Performance 11 X XX XX X X X X X X X

12. SPEC Benchmark What programs can be used to evaluate different computers? Can we cheat? Need a standard! SPEC Benchmark Standard Performance Evaluation Corp (SPEC) Programs used to measure performance (CPU, Web, I/O…) Typical actual workloads SPEC CPU2006 Elapsed time to execute a selection of programs Negligible I/O, so focuses on CPU performance Summarize as geometric mean of performance ratios CINT2006 (integer) and CFP2006 (floating-point) 12

13. SPEC Benchmark 13 NameDescriptionIC×10 9 CPITc (ns)Exec timeRef timeSPECratio perlInterpreted string processing2,1180.750.406379,77715.3 bzip2Block-sorting compression2,3890.850.408179,65011.8 gccGNU C Compiler1,0501.720.47248,05011.1 mcfCombinatorial optimization33610.000.401,3459,1206.8 goGo game (AI)1,6581.090.4072110,49014.6 hmmerSearch gene sequence2,7830.800.408909,33010.5 sjengChess game (AI)2,1760.960.483712,10014.5 libquantumQuantum computer simulation1,6231.610.401,04720,72019.8 h264avcVideo compression3,1020.800.4099322,13022.3 omnetppDiscrete event simulation5872.940.406906,2509.1 astarGames/path finding1,0821.790.407737,0209.1 xalancbmkXML parsing1,0582.700.401,1436,9006.0 Geometric mean11.7 CINT2006 for Opteron X4 2356 High cache miss rates

14. SPEC Benchmark 14 CINT2000 Results for Various Processors Clock Speed (GHz) CINT2000

15. SPEC Benchmark 15 CFP2000 CFP2000 Results for Various Processors Clock Speed (GHz)

16. Other Performance Factors Power consumption is one factor in evaluating performance This is specifically important in the embedded market where battery life is important (and passive cooling) 16

17. Examples Example 4. given a program with 10 6 instructions with the following mix: 10% class A, 20% class B, 50% class C, and 20% class D. If this program is executed on two different processors with the specifications given below, then What is the effective CPI for the program for each implementation? Which implementation is faster? What is the speedup? 17 Processor CR (GHz) CPI Class ACPI Class BCPI Class CCPI Class D 11.51234 222222

18. Examples Example 5. The information for some program that is executed on some processor is given below. If the processor is modified such that the CPI for Class 2 instructions is reduced to 2, then w ould it be beneficial to adopt this modification if this modification requires increasing the clock cycle by 10%? modification does not affect the clock cycle but requires twice the amount of power to execute the program 18 Class i CPI i Frequency i 120.3 250.2 330.5