1. Thoughts on Metrics & Benchmarking Randal E. Bryant

2. – 2 – Program Complexity Complexity Sources Control Concurrency Error handling & mitigation Data Complex data structures Data structure invariants Data Control Embedded Systems My Systems

3. – 3 – Role of Benchmarks Good Provides objective standard for comparison Competition drives progressBad Can become goal rather than means to achieve goal Poorly selected set can thwart progress E.g., DIMACS SAT benchmarks

4. – 4 – Benchmarking Desiderata Organic Process E.g., SPEC, SAT Continually update Large community involvedSelection Range of sizes From different sources (applications, organizations) Desirable, but Uncommon: Parameterized Access to originating application

5. – 5 – Some Ideas Select Small Set of Driving Applications Control intensive embedded system(s) Data intensive Create Supporting Benchmarks Scaled down versions or components of driving applicationsIssues Language Ancilliary code: Libraries, OS Specs.