1. Benchmarking for [Physical] Synthesis Igor Markov and Prabhakar Kudva The Univ. of Michigan / IBM

2. In This Talk … Benchmarking vs benchmarks Benchmarking vs benchmarks Benchmarking exposes new research Qs Benchmarking exposes new research Qs Why industry should care about benchmarking Why industry should care about benchmarking What is (and is not) being done to improve benchmarking infrastructure What is (and is not) being done to improve benchmarking infrastructure Not in this talk, but in a focus group Not in this talk, but in a focus group Incentives for verifying published work Incentives for verifying published work How to accelerate a culture change How to accelerate a culture change

3. Benchmarking Design benchmarks Design benchmarks Data model / representation; Instances Data model / representation; Instances Objectives (QOR metrics) and constraints Objectives (QOR metrics) and constraints Algorithms, methodologies; Implementations Algorithms, methodologies; Implementations Solvers: ditto Solvers: ditto Empirical and theoretical analyses, e.g., Empirical and theoretical analyses, e.g., Hard vs easy benchmarks (regardless of size) Hard vs easy benchmarks (regardless of size) Correlation between different objectives Correlation between different objectives Upper / lower bounds for QOR, statistical behavior, etc Upper / lower bounds for QOR, statistical behavior, etc Dualism between benchmarks and solvers Dualism between benchmarks and solvers For more details, see http://gigascale.org/bookshelf For more details, see http://gigascale.org/bookshelf

4. Industrial Benchmarking Growing size & complexity of VLSI chips Growing size & complexity of VLSI chips Design objectives Design objectives Area / power / yield / etc Area / power / yield / etc Design constraints Design constraints Timing / FP + fixed-die partitions / fixed IPs / routability / pin access / signal integrity… Timing / FP + fixed-die partitions / fixed IPs / routability / pin access / signal integrity… Can the same algo excel in all contexts? Can the same algo excel in all contexts? Sophistication of layout and logic motivate open benchmarking for Synthesis and P&R Sophistication of layout and logic motivate open benchmarking for Synthesis and P&R

5. Design Types ASICs ASICs Lots of fixed I/Os, few macros, millions of standard cells Lots of fixed I/Os, few macros, millions of standard cells Design densities : 40-80% (IBM) Design densities : 40-80% (IBM) Flat and hierarchical designs Flat and hierarchical designs SoCs SoCs Many more macro blocks, cores Many more macro blocks, cores Datapaths + control logic Datapaths + control logic Can have very low design densities : < 20% Can have very low design densities : < 20% Micro-Processor (  P) Random Logic Macros(RLM) Micro-Processor (  P) Random Logic Macros(RLM) Hierarchical partitions are LS+P&R instances (5-30K) Hierarchical partitions are LS+P&R instances (5-30K) High placement densities : 80%-98% (low whitespace) High placement densities : 80%-98% (low whitespace) Many fixed I/Os, relatively few standard cells Many fixed I/Os, relatively few standard cells Note: “Partitioning w Terminals” DAC`99, ISPD `99, ASPDAC`00

6. Why Invest in Benchmarking Academia Academia Benchmarks can identify / capture new research problems Benchmarks can identify / capture new research problems Empirical validation of novel research Empirical validation of novel research Open-source tools/BMs can be analyzed and tweaked Open-source tools/BMs can be analyzed and tweaked Industry Industry Evaluation and transfer of academic research Evaluation and transfer of academic research Support for executive decisions (which tools are relatively week & must be improved) Support for executive decisions (which tools are relatively week & must be improved) Open-source tools/BMs can be analyzed and tweaked Open-source tools/BMs can be analyzed and tweaked When is an EDA problem (not) solved? When is an EDA problem (not) solved? Are there good solver implementations? Are there good solver implementations? Can they “solve” existing benchmarks? Can they “solve” existing benchmarks?

7. Participation / Leadership Necessary Activity 1: Benchmarking platform / flows Activity 1: Benchmarking platform / flows Activity 2: Establishing common evaluators Activity 2: Establishing common evaluators Static timing analysis Static timing analysis Congestion / yield prediction Congestion / yield prediction Power estimation Power estimation Activity 3: Standard-cell libraries Activity 3: Standard-cell libraries Activity 4: Large designs w bells & whistles Activity 4: Large designs w bells & whistles Activity 5: Automation of benchmarking Activity 5: Automation of benchmarking

8. Activity 1: Benchmarking Platform Benchmarking “platform”: a reasonable subset of Benchmarking “platform”: a reasonable subset of data model data model specific data representations (e.g., file formats) specific data representations (e.g., file formats) access mechanisms (e.g., APIs) access mechanisms (e.g., APIs) reference implementation (e.g., a design database) reference implementation (e.g., a design database) design examples in compatible formats design examples in compatible formats Base platforms available (next slide) Base platforms available (next slide) More participation necessary More participation necessary regular discussions regular discussions additional tasks / features outlined additional tasks / features outlined

9. Common Methodology Platform Synthesis (SIS, MVSIS…) Placement (Capo, Dragon, Feng Shui, mPl,…) Common Model (Open Access?) Blif  Bookshelf format Blue Flow exists, Common model hooks: To be Done

11. Placement Utilities http://vlsicad.eecs.umich.edu/BK/PlaceUtils/ http://vlsicad.eecs.umich.edu/BK/PlaceUtils/ Accept input in the GSRC Bookshelf format Accept input in the GSRC Bookshelf format Format converters Format converters LEF/DEF  Bookshelf LEF/DEF  Bookshelf Bookshelf  Kraftwerk (DAC98 BP, E&J) Bookshelf  Kraftwerk (DAC98 BP, E&J) BLIF(SIS)  Bookshelf BLIF(SIS)  Bookshelf Evaluators, checkers, postprocessors and plotters Evaluators, checkers, postprocessors and plotters Contributions in these categories are welcome Contributions in these categories are welcome

12. Placement Utilities (cont’d) Wirelength Calculator (HPWL) Wirelength Calculator (HPWL) Independent evaluation of placement results Independent evaluation of placement results Placement Plotter Placement Plotter Saves gnuplot scripts ( .eps,.gif, …) Saves gnuplot scripts ( .eps,.gif, …) Multiple views (cells only, cells+nets, rows,…) Multiple views (cells only, cells+nets, rows,…) Probabilistic Congestion Maps (Lou et al.) Probabilistic Congestion Maps (Lou et al.) Gnuplot scripts Gnuplot scripts Matlab scripts Matlab scripts better graphics, including 3-d fly-by views better graphics, including 3-d fly-by views.xpm files ( .gif,.jpg,.eps, …).xpm files ( .gif,.jpg,.eps, …)

13. Placement Utilities (cont’d) Legality checker Legality checker Simple legalizer Simple legalizer Layout Generator Layout Generator Given a netlist, creates a row structure Given a netlist, creates a row structure Tunable %whitespace, aspect ratio, etc Tunable %whitespace, aspect ratio, etc All available in binaries/PERL at All available in binaries/PERL at http://vlsicad.eecs.umich.edu/BK/PlaceUtils/ Most source codes are shipped w Capo Most source codes are shipped w Capo

14. Activity 2: Creating Evaluators Contribute measures/analysis tools for: Contribute measures/analysis tools for: Timing Analysis Timing Analysis Congestion/Yield Congestion/Yield Power Power Area Area Noise…. Noise….

15. Challenges for Evaluating Timing-Driven Optimizations QOR not defined clearly QOR not defined clearly Max path-length? Worst set-up slack? Max path-length? Worst set-up slack? With false paths or without?... With false paths or without?... Evaluation methods are not replicable (often shady) Evaluation methods are not replicable (often shady) Questionable delay models, technology params Questionable delay models, technology params Net topology generators (MST, single-trunk Steiner trees) Net topology generators (MST, single-trunk Steiner trees) Inconsistent results: path delays <  gate delays Inconsistent results: path delays <  gate delays Public benchmarks?... Public benchmarks?... Anecdote: TD-place benchmarks in Verilog (ISPD `01) Anecdote: TD-place benchmarks in Verilog (ISPD `01) Companies guard netlists, technology parameters Companies guard netlists, technology parameters Cell libraries; area constraints Cell libraries; area constraints

16. Metrics for Timing + Reporting STA non-trivial: use PrimeTime or PKS STA non-trivial: use PrimeTime or PKS Distinguish between optimization and evaluation Distinguish between optimization and evaluation Evaluate setup-slack using commercial tools Evaluate setup-slack using commercial tools Optimize individual nets and/or paths Optimize individual nets and/or paths E.g., net-length versus allocated budgets E.g., net-length versus allocated budgets Report all relevant data Report all relevant data How was the total wirelength affected? How was the total wirelength affected? Were per-net and per-path optimizations successful? Were per-net and per-path optimizations successful? Did that improve worst slack or did something else? Did that improve worst slack or did something else? Huge slack improvements reported in some 1990s papers, but wire delays were much smaller than gate delays Huge slack improvements reported in some 1990s papers, but wire delays were much smaller than gate delays

17. Benchmarking Needs for Timing Opt. A common, reusable STA methodology A common, reusable STA methodology High-quality, open-source infrastructure High-quality, open-source infrastructure False paths; realistic gate/delay models False paths; realistic gate/delay models Metrics validated against phys. synthesis Metrics validated against phys. synthesis The simpler the better, but must be good predictors The simpler the better, but must be good predictors Buffer insertion profoundly impacts layout Buffer insertion profoundly impacts layout The use of linear wirelength in timing-driven layout assumes buffers insertion (min-cut vs quadratic) The use of linear wirelength in timing-driven layout assumes buffers insertion (min-cut vs quadratic) Apparently, synthesis is affected too Apparently, synthesis is affected too

18. Vertical Benchmarks “Tool flow” “Tool flow” Two or more EDA tools, chained sequentially (potentially, part of a complete design cycle) Two or more EDA tools, chained sequentially (potentially, part of a complete design cycle) Sample contexts: physical synthesis, place & route, retiming followed by sequential verification Sample contexts: physical synthesis, place & route, retiming followed by sequential verification Vertical benchmarks Vertical benchmarks Multiple, redundant snapshots of a tool flow sufficient info for detailed analysis of tool performance Multiple, redundant snapshots of a tool flow sufficient info for detailed analysis of tool performance Herman Schmit @CMU is maintaining a resp. slot in the VLSI CAD Bookshelf Herman Schmit @CMU is maintaining a resp. slot in the VLSI CAD Bookshelf See http://gigascale.org/bookself See http://gigascale.org/bookself Include flat gate-level netlists Include flat gate-level netlists Library information ( < 250nm) Library information ( < 250nm) Realistic timing & fixed-die constraints Realistic timing & fixed-die constraints

19. Infrastructure Needs Need common evaluators of delay / power Need common evaluators of delay / power To avoid inconsistent / outdated results To avoid inconsistent / outdated results Relevant initiatives from Si2 Relevant initiatives from Si2 OLA (Open Library Architecture) OLA (Open Library Architecture) OpenAccess OpenAccess For more info, see http://www.si2.org For more info, see http://www.si2.orghttp://www.si2.org Still: no reliable public STA tool Still: no reliable public STA tool Sought: OA-based utilities for timing/layout Sought: OA-based utilities for timing/layout

20. Activity 3 : Standard-cell Libraries Libraries carry technology information Libraries carry technology information Impact of wirelength delays increases in recent technology generations Impact of wirelength delays increases in recent technology generations Cell characteristics must be compatible Cell characteristics must be compatible Some benchmarks in the Bookshelf use 0.25  m and 0.35  m libraries Some benchmarks in the Bookshelf use 0.25  m and 0.35  m libraries Geometry info is there, + timing (in some cases) Geometry info is there, + timing (in some cases) Cadence test library? Cadence test library? Artisan libraries? Artisan libraries? Use commercial tools to create libraries Use commercial tools to create libraries Prolific, Cadabra,… Prolific, Cadabra,…

21. Activity 4: Need New Benchmarks To Confirm / Defeat Tool Tuning Data on tuning from the ISPD03 paper “Benchmarking for Placement”, Adya et al. Data on tuning from the ISPD03 paper “Benchmarking for Placement”, Adya et al. Observe that Observe that Capo does well on Cadence-Capo, grid-like circuits Capo does well on Cadence-Capo, grid-like circuits Dragon does well on IBM-Place (IBM-Dragon) Dragon does well on IBM-Place (IBM-Dragon) FengShui does well on MCNC benchmarks FengShui does well on MCNC benchmarks mPL does well on PEKO mPL does well on PEKO This is hardly a coincidence This is hardly a coincidence Motivation for more / better benchmarks Motivation for more / better benchmarks P.S. Most differences above have been explained, all placers above have been improved P.S. Most differences above have been explained, all placers above have been improved

22. Activity 4: Large Benchmark Creation has large designs www.opencores.org has large designs www.opencores.org May be a good starting point – use vendor tools to create blif files (+post results) May be a good starting point – use vendor tools to create blif files (+post results) Note: there may be different ways to convert Note: there may be different ways to convert A group of design houses (IBM, Intel, LSI, HP) is planning a release of new large gate-level benchmarks for layout A group of design houses (IBM, Intel, LSI, HP) is planning a release of new large gate-level benchmarks for layout Probably no logic information Probably no logic information

23. Activity 5: Benchmarking Automation Rigorous benchmarking is laborious. Risk of errors is high Rigorous benchmarking is laborious. Risk of errors is high How do we keep things simple / accessible? How do we keep things simple / accessible? Encapsulate software management in an ASP Encapsulate software management in an ASP Web uploads for binaries and source in tar.gz w Makefiles Web uploads for binaries and source in tar.gz w Makefiles Web uploads for benchmarks Web uploads for benchmarks GUI interface for NxM simulations; tables created automatically GUI interface for NxM simulations; tables created automatically GUI interface for composing tool-flows; flows can be saved/reused GUI interface for composing tool-flows; flows can be saved/reused Distributed back-end includes job scheduling Distributed back-end includes job scheduling Email notification of job completion Email notification of job completion All files created are available on the Web (permissions & policies) All files created are available on the Web (permissions & policies) Anyone can re-run / study your experiment or interface with it Anyone can re-run / study your experiment or interface with it

31. Follow-on Action Plan Looking for volunteers to  -test Bookshelf.exe Looking for volunteers to  -test Bookshelf.exe Particularly, in the context of synthesis & verification Particularly, in the context of synthesis & verification Contact: Igor imarkov@eecs.umich.edu Contact: Igor imarkov@eecs.umich.eduimarkov@eecs.umich.edu Create a joint benchmarking group from industry and academia Create a joint benchmarking group from industry and academia Contact: Prabhakar kudva@us.ibm.com Contact: Prabhakar kudva@us.ibm.comkudva@us.ibm.com Regular discussions Regular discussions Development based on common infrastructure Development based on common infrastructure