1 CPS4150 Chapter 4 Assessing and Understanding Performance

2 CPS4150 Measure, Report, and Summarize Make intelligent choices See through the marketing hype Key to understanding underlying organizational motivation Why is some hardware better than others for different programs? What factors of system performance are hardware related? (e.g., Do we need a new machine, or a new operating system?) How does the machine's instruction set affect performance? 4.1 Performance

3 CPS4150 For example, to improve the performance of a software system, we may need to understand what factors in the hardware contribute to the overall system performance and the relative importance of these factors. The factors may include how well the program uses the instructions of the computer, how well the underlying hardware implements the instructions, and how well the memory and I/O system performance. Understanding how to determine the performance impact of these factors is crucial to understanding the motivation behind the design of particular aspects of the computer. Performance

4 CPS4150 Which of these airplanes has the best performance? AirplanePassengersRange (mi)Speed (mph) Boeing Boeing BAC/Sud Concorde Douglas DC How much faster is the Concorde compared to the 747? How much bigger is the 747 than the Douglas DC-8? If we define performance in term of speed. There are two possibility: The fastest one as the one with the highest cruising speed/one passenger The one transporting 450 passengers from place to another place

5 CPS4150 Response Time/execution Time (latency) (The total time required for the computer to complete a task.) — How long does it take for my job to run? — How long does it take to execute a job? — How long must I wait for the database query? — Individual computer user will be interested in this value (how fast the computer can get my job done) Throughput — How many jobs can the machine run at once? — What is the average execution rate? — How much work is getting done in a given time? — Data center managers are often interested in increasing this value If we upgrade a machine with a new processor what do we increase? If we add a new machine to the lab what do we increase? Example p.242 Computer Performance: TIME, TIME, TIME

6 CPS4150 For some program running on machine X, Performance X = 1 / Execution time X "X is n times faster than Y" Performance X / Performance Y = n OR Excution time y / Excution time x = n Example: –machine A runs a program in 20 seconds –machine B runs the same program in 25 seconds How much faster is A than B? Do Example: Page 243 Book's Definition of Performance

7 CPS4150 Elapsed Time (or response time, system performance)– The total time to complete a task –counts everything (disk and memory accesses, I/O, operating system overhead, etc.) –computer is often shared and a processor may work on several programs simultaneously  optimize throughput rather than minimize the elapsed time for one program –a useful number, but often not good for comparison purposes CPU time (CPU performance) –doesn't count I/O or time spent running other programs –can be broken up into system CPU time (operating system performing tasks on behalf of the program), and user CPU time (CPU time spent in the program) Our focus: user CPU time –time spent executing the lines of code that are "in" our program Execution Time

8 CPS4150 Clock Cycles –A measure that relates to how fast the hardware can perform basic functions –A clock that runs at a constant rate and determines when events tale place in the hardware. –These discrete time intervals are called clock cycle. –The length of a clock period both as the time for a complete clock cycle and as the clock rate which is the inverse of the clock period. –Example: clock cycle: 0.25 nanoseconds (ns) clock rate: 4 gigahertz (GHz)

9 CPS Clock Cycles Instead of reporting execution time in seconds, we often use cycles Our goal  reduce either the length of the clock cycle or the number of clock cycles BUT, a trade-ff between the number of clock cycles needed for a program and the length of each cycle.

10 CPS Clock Cycles Instead of reporting execution time in seconds, we often use cycles Clock “ticks” indicate when to start activities (one abstraction): cycle time = time between ticks = seconds per cycle clock rate (frequency) = cycles per second (1 Hz. = 1 cycle/sec) A 4 Ghz. clock has a cycle time time

11 CPS4150 So, to improve performance (everything else being equal) you can either (increase or decrease?) ________ the # of required cycles for a program, or ________ the clock cycle time or, said another way, ________ the clock rate. How to Improve Performance

12 CPS4150 Could assume that number of cycles equals number of instructions This assumption is incorrect, different instructions take different amounts of time on different machines. Why? hint: remember that these are machine instructions, not lines of C code time 1st instruction2nd instruction3rd instruction4th 5th6th... How many cycles are required for a program?

13 CPS4150 Multiplication takes more time than addition Floating point operations take longer than integer ones Accessing memory takes more time than accessing registers Important point: changing the cycle time often changes the number of cycles required for various instructions (more later) time Different numbers of cycles for different instructions

14 CPS4150 Our favorite program runs in 10 seconds on computer A, which has a 4 GHz. clock. We are trying to help a computer designer build a new machine B, that will run this program in 6 seconds. The designer can use new (or perhaps more expensive) technology to substantially increase the clock rate, but has informed us that this increase will affect the rest of the CPU design, causing machine B to require 1.2 times as many clock cycles as machine A for the same program. What clock rate should we tell the designer to target?" Don't Panic, can easily work this out from basic principles Example

15 CPS Clock Cycles CUP time A = (CPU clock cycles A ) / (clock rate A ) 10 seconds = (CPU clock cycles A ) /(4 * 10 9 cycles/seconds) CPU clock cycles A = 10 * 4 * 10 9 cycles = 40 * 10 9 cycles CUP clock cycles A = 40 * 10 9 cycles CUP time B = 6 seconds = (1.2 * CPU clock cycles A ) / (clock rate B ) clock rate B = (1.2* 40 * 10 9 cycles)/ 6 seconds =8 GHz

16 CPS4150 A given program will require –some number of instructions (machine instructions) –some number of cycles –some number of seconds We have a vocabulary that relates these quantities: –cycle time (seconds per cycle) –clock rate (cycles per second) –CPI (cycles per instruction) (average number of clock cycle each instruction takes to execution) a floating point intensive application might have a higher CPI –MIPS (millions of instructions per second) this would be higher for a program using simple instructions Now that we understand cycles

17 CPS4150 Performance Performance is determined by execution time Do any of the other variables equal performance? –# of cycles to execute program? –# of instructions in program? –# of cycles per second? –average # of cycles per instruction? –average # of instructions per second? Common pitfall: thinking one of the variables is indicative of performance when it really isn’t.

18 CPS4150 CPI Example Suppose we have two implementations of the same instruction set architecture (ISA). For some program, Machine A has a clock cycle time of 250 ps and a CPI of 2.0 Machine B has a clock cycle time of 500 ps and a CPI of 1.2 What machine is faster for this program, and by how much? See page 248 If two machines have the same ISA which of our quantities (e.g., clock rate, CPI, execution time, # of instructions, MIPS) will always be identical?

19 CPS4150 Example: A compiler designer is trying to decide between two code sequences for a particular machine. Based on the hardware implementation, there are three different classes of instructions: Class A, Class B, and Class C, and they require one, two, and three cycles (respectively). The first code sequence has 5 instructions: 2 of A, 1 of B, and 2 of C The second sequence has 6 instructions: 4 of A, 1 of B, and 1 of C. Which sequence will be faster? How much? (CUP clock cycle 1 = (2X1) + (1X2) +(2X3) = = 10 cycles) What is the CPI for each sequence? (CPU clock cycle 2 = (4X1) +(1X2) +(1X3) = = 9 cycles) # of Instructions Example

20 CPS4150 Workload: A set of programs run on a computer that is either the actual collection of applications run by a user or is constructed from real programs to approximate such as mix. Benchmark: Programs specially chosen to measure performance. Small benchmarks –nice for architects and designers –easy to standardize –can be abused (specific benchmark for special purpose and is not for general case), for example engineering and developing machine –May be misleading –Different classes and applications of computers will require different types of benchmarks. SPEC (System Performance Evaluation Cooperative) –companies have agreed on a set of real program and inputs –valuable indicator of performance (and compiler technology) –can still be abused 4.3 Benchmarks

21 CPS4150 Benchmarks Reproducibility- see page 256, Figure 4-3. –Input data will affect the memory system Total execution Time –Directly proportional to execution time - our final measure of performance. –The average of the execution times that is directly proportional to total execution time is the arithmetic mean (AM): where Time i is the execution time for the ith program of a total of n in the workload. A weight w i can be assigned to each program to indicate the frequency of the program in that workload. For example 20% of program1 and 80% of program2. By summing the products of weighting factors and execution times, we can obtain clear picture of the performance of the workload.  weighted arithmetic mean.

22 CPS4150 Check your self –See page 258. Check your self

23 CPS4150 Benchmark Games An embarrassed Intel Corp. acknowledged Friday that a bug in a software program known as a compiler had led the company to overstate the speed of its microprocessor chips on an industry benchmark by 10 percent. However, industry analysts said the coding error…was a sad commentary on a common industry practice of “cheating” on standardized performance tests…The error was pointed out to Intel two days ago by a competitor, Motorola …came in a test known as SPECint92…Intel acknowledged that it had “optimized” its compiler to improve its test scores. The company had also said that it did not like the practice but felt to compelled to make the optimizations because its competitors were doing the same thing…At the heart of Intel’s problem is the practice of “tuning” compiler programs to recognize certain computing problems in the test and then substituting special handwritten pieces of code… Saturday, January 6, 1996 New York Times

24 CPS4150 SPEC ‘89 Compiler “enhancements” and performance

25 CPS4150 SPEC CPU2000

26 CPS4150 SPEC CPU2000 The execution time measurement are first normalized by dividing the execution time on Sum Ultra 5_10 with 300 MHz processor by the execution time on the measured computer.  SPEC ratio. The bigger the faster. For a given instruction set architecture, increase in CPU performance can come from three sources: –Increase in clock rate –Improvements in processor organization that lower the CPI.\ –Compiler enhancements that lower the instruction count or generate instructions with a lower average CPI. (e.g. By using simpler instruction sets).

27 CPS4150 SPEC 2000 Does doubling the clock rate double the performance? Can a machine with a slower clock rate have better performance?