1. BİL 221 Bilgisayar Yapısı Lab. – 1: Benchmarking

2. 2 Basic Performance Metrics Time related: –Execution time [seconds] wall clock time system and user time –Latency –Response time Rate related: –Rate of computation floating point operations per second [flops] integer operations per second [ops] –Data transfer (I/O) rate [bytes/second] Effectiveness: –Efficiency [%] –Memory consumption [bytes] –Productivity [utility/($*second)] Modifiers: –Sustained –Peak –Theoretical peak

3. 3 What Is a Benchmark? The term “benchmark” also commonly applies to specially-designed programs used in benchmarking A benchmark should: –be domain specific (the more general the benchmark, the less useful it is for anything in particular) –be a distillation of the essential attributes of a workload –avoid using single metric to express the overall performance Computational benchmark kinds –synthetic: specially-created programs that impose the load on the specific component in the system –application: derived from a real-world application program Benchmark: a standardized problem or test that serves as a basis for evaluation or comparison (as of computer system performance) [Merriam-Webster]

4. 4 Purpose of Benchmarking To define the playing field To provide a tool enabling quantitative comparisons Acceleration of progress –enable better engineering by defining measurable and repeatable objectives Establishing of performance agenda –measure release-to-release or version-to-version progress –set goals to meet –be understandable and useful also to the people not having the expertise in the field (managers, etc.)

5. 5 Properties of a Good Benchmark Relevance: meaningful within the target domain Understandability Good metric(s): linear, orthogonal, monotonic Scalability: applicable to a broad spectrum of hardware/architecture Coverage: does not over-constrain the typical environment Acceptance: embraced by users and vendors Has to enable comparative evaluation Limited lifetime: there is a point when additional code modifications or optimizations become counterproductive Adapted from: Standard Benchmarks for Database Systems by Charles Levine, SIGMOD ‘97

6. Metrics of Performance Compiler Programming Language Application Datapath Control TransistorsWiresPins ISA Function Units (millions) of Instructions per second: MIPS (millions) of (FP) operations per second: MFLOP/s Cycles per second (clock rate) Megabytes per second Answers per month Operations per second

7. 7 MIPS ve MFLOPS  Machines with different instruction sets?  Programs with different instruction mixes?  Uncorrelated with performance Marketing metric Marketing metric  “Meaningless Indicator of Processor Speed”  Popular in supercomputing community  Often not where time is spent  Not all FP operations are equal “Normalized” MFLOP/s “Normalized” MFLOP/s  Can magnify performance differences A better algorithm (e.g., with better data reuse) can run faster even with higher FLOP count A better algorithm (e.g., with better data reuse) can run faster even with higher FLOP count MFLOPS MIPS

8. 8 Benchmarklar  To increase predictability, collections of benchmark applications, called benchmark suites, are popular  SPECCPU: popular desktop benchmark suite CPU only, split between integer and floating point programs CPU only, split between integer and floating point programs SPECint2000 has 12 integer, SPECfp2000 has 14 integer pgms SPECint2000 has 12 integer, SPECfp2000 has 14 integer pgms SPECCPU2006 was announced Spring 2006 SPECCPU2006 was announced Spring 2006 SPECSFS (NFS file server) and SPECWeb (WebServer) added as server benchmarks SPECSFS (NFS file server) and SPECWeb (WebServer) added as server benchmarks www.spec.org www.spec.org www.spec.org  Transaction Processing Council measures server performance and cost-performance for databases TPC-C Complex query for Online Transaction Processing TPC-C Complex query for Online Transaction Processing TPC-H models ad hoc decision support TPC-H models ad hoc decision support TPC-W a transactional web benchmark TPC-W a transactional web benchmark TPC-App application server and web services benchmark TPC-App application server and web services benchmark

9. SPEC CPU (integer) “Representative” applications keeps growing with time!

10. SPEC CPU (floating point)

11. 11 Performans Nasıl Özetlenir?  Arithmetic average of execution times?? But they vary in basic speed, so some would be more important than others in arithmetic average But they vary in basic speed, so some would be more important than others in arithmetic average  Could add weights per program, but how to pick weight? Different companies want different weights for their products Different companies want different weights for their products  SPECRatio: Normalize execution times to reference computer, yielding a ratio proportional to performance = time on reference computer / time on computer being rated time on reference computer / time on computer being rated Spec uses an older Sun machine as reference Spec uses an older Sun machine as reference

12. 12 Oranlar  If program SPECRatio on Computer A is 1.25 times bigger than Computer B, then  Note that when comparing 2 computers as a ratio, execution times on the reference computer drop out, so choice of reference computer is irrelevant

13. 13 Ortalamalar

14. 14 Geometrik Ortalama  Since ratios, proper mean is geometric mean (SPECRatio unitless, so arithmetic mean meaningless) 1. Geometric mean of the ratios is the same as the ratio of the geometric means 2. Ratio of geometric means = Geometric mean of performance ratios  choice of reference computer is irrelevant!  These two points make geometric mean of ratios attractive to summarize performance

15. SPECjvm2008

16. Workloads

18. Workload Descriptions