1. Lecture 2: Performance Evaluation
Performance definition, benchmark, summarizing performance, Amdahl’s law, and CPI

2. What Does Performance Mean?
Response time
A simulation program finishes in 5 minutes
Throughput
A web server serves 5 million request per second
Other metrics
MIPS (million instruction per second)
MFLOPS
Clock frequency
Meaning: how fast the computer respond to user input or finish a program; the sooner the better
Examples: A user sees the result in 5 second after inputting the keyword to a library database system; A simulation program finishes in one hour
how much work a computer can finish for a given time; the higher the better
MFLOPS:
A clustered-base web server serves up to 5 million requests per second
Why throughput: a system runs multiple jobs simultaneously.
Other measures MIPS, MFLOPS, clock frequency
Meaningful when directly related to response time or throughput

3. Execution Time
Processor design is concerned with processor consumed by program execution. Shorter execution time=>
Shorter response time
Higher throughput
Execution time = #inst×CPI×Cycletime
What affects #inst, CPI, and cycle time?
Almost all designs can be interpreted
Any other metrics is meaningful only if consistent with execution time

4. Performance of Computers
Performance is defined for a program and a machine.
How to compare computers? Need benchmark programs:
Real applications: scientific programs, compilers, text-processing software, image processing
Modified applications: providing portability and focus
Kernels: good to isolate performance of individual features
Lmbench: measure latency and bandwidth of memory, file system, networking, etc.
Toy benchmarks
Synthetic benchmarks: matching average execution profile

5. Performance Comparison
“X is n times faster than Y”:
n: speedup if we are considering an enhancement, optimization, etc.
What does “improving” mean?
Improve performance: decrease execution time, increase throughput
Improve execution time: decrease execution time
Degrade performance: the reverse of the above; brings negative speedup
Give examples:
Given machine M1, program A, Program B
Execution Time Performance
A 5
B 10

6. Benchmark Suite
Benchmark suite is a collection of benchmarks with a variety of applications
Alleviating weakness of a single benchmark
More representative for computer designers to evaluate their design
Benchmarks test both computer and compilers, and OS in many cases
Desktop benchmarks: CPU, memory, and graphics performance
Sever benchmarks: throughput-oriented, I/O and OS intensive
Embedded benchmarks: measuring the ability to meet deadline and save power

7. Summarizing Performance
Given the performance of a set of programs, how to evaluate the performance of machines?
A B C
P1 (secs)
P2 (secs)
Total (secs)
Which computer is the “best” one?

8. Arithmetic Mean Total execution time / (number of programs)
Simple and intuitive
Representative if the user run the programs an equal number of times

9. Weighted Arithmetic Mean
Give (different) weights to different programs
Considering the frequencies of programs in the workload

10. Geometric Means Based on relative performance to a reference machine
Relative performance is consistent with different reference machines
If C is 2x faster than B (using B as the reference), B is 2x faster than A (A as the reference), then C is 4x faster than A (A as the reference)

11. Harmonic Mean
Given speedups s1, s2, …, s_n, the average speedup by harmonic mean is
n / (1/s1 + 1/s2 + … + 1/s_n)
Why not arithmetic mean?

12. Amdahl’s Law
We know about performance: defining, measuring, and summarizing
How to maximize performance gains from the beginning in our design?
Principle: Make the Common Case Fast!

13. Amdahl’s Law
Predict overall speedup from “local speedup” by an enhancement, provided the frequency to use the enhancement is know.
“Local speedup” is related to design and optimization objectives, like to double CPU frequency, to reduce cache latency by half

14. Amdahl’s Law

15. Equation Based on Instruction Types

16. Make Design Choice Using CPU Time Equation
Assume we need to improve the performance of a graphics engine:
FP FPSQR Other
Frequency 25% 2% 75%
CPI
Alternative 1: CPIFPSQR 20  2
Alternative 2: CPIFP  2.5
Which one is better? Calculate speedups.

17. Amdahl’s Law Choice one: Speed up FP Square root by 10x
Choice two: Speed up all FP instruction by 1.6x
20% time are used by FP Square root, 50% for all FP inst
Which choice is better?
Implication: Optimizing for the common case first

18. SPEC CPU Benchmark SPEC: Standard Performance Evaluation Corporation
CPU-intensive benchmark for evaluating processor performance of workstation
Four generations: SPEC89, SPEC92, SPEC95, and SPEC2000
Two types of programs: INT and FP
Emphasizing memory system performance in SPEC2000

19. SPEC CPU2000 Profiling Dynamic instruction mix
Instruction Int avg FP avg
Load int 26% 15%
Store int 10% 2%
Load fp %
Store fp - 7%
Add 19% 23%
All fp inst - 41%
Cond br. 12% 4%
All ctrl inst 16% 4%

20. Other SPEC Benchmarks SPECviewperf and SPEapc: 3D graphics performance
SPEC JVM98: performance of client-side Java virtual machine
SPEC JBB2000: Server-cline Java application
SPEC WEB99: evaluating WWW servers
SPEC HPC96: parallel and distributed computing

21. Server Benchmarks SPEC CPU2000, WBB99, SFS97
TPC Measuring the ability of a system to handle transactions
TPC-C: online transaction processing (OLTP) benchmark (for bank systems)
TPC-H: ad hoc decision make support
TPC-R: decision make support with standard queries
TPC-W: simulating business-oriented transactional web server

22. Embedded Benchmark
EEMBC (Embedded Microprocessor Benchmark Consortium) benchmarks
Based on kernel performance
Five classes: automotive/industrial, consumer networking, office automation, and telecommunications
Embedded benchmarks are not mature