1. Mutation Analysis with Coverage Discounting Peter Lisherness, Nicole Lesperance, and Kwang-Ting (Tim) Cheng University of California – Santa Barbara

2. Motivation 1 Functional Coverage Mutation Analysis Coverage Discounting + Functionally Meaningful + Evaluates design activation -Ignores propagation and checker quality + Evaluates propagation and checker quality - Mutants hard to analyze + Functionally Meaningful + Evaluates design activation, propagation and checkers

3. Coverage Discounting: The Main Idea Use fault insertion to discount (revise downward) coverage scores 2 Detected + undetected mutants Functional Coverage Mutation Analysis Coverage Discounting Undetected Mutants Coverage Score 100% 0% Discounted Coverage Score

4. An Existing Solution: Mutation Analysis DUT Tests Checker Undetected Fix Add/Fix Detected Benefits – Evaluates quality of testbench checkers – Indicates if tests fail to propagate potential errors to the checker or fail to activate mutants 3

5. Mutation Analysis Drawbacks 1.Long runtime for simulation 2.Many man-hours required to analyze results: 2.1 Mutants are synthetic - difficult to identify testbench improvements to target mutant detection 2.2Redundant mutants never detectable: Some undetected faults are redundant if (a > 0) b=1+a; else b=1-a; if (a >= 0) b=1+a; else b=1-a; Example: 4

6. Premise of Coverage Discounting 1) A mutant may change design functionality 2) A coverpoint, covered by testbench in the original design, may be suppressed (i.e. no longer be covered) in the mutated design 3) If the mutant suppressing the coverpoint is not detected, then the coverpoint is no longer considered covered in the original design 5

7. Coverage Discounting Flow 6 DUT Tests Checker Detected Undetected Add more tests No Yes Fix Coverage Changed?

8. An Example Original Design if (remaining_length >= 5) begin burst_en=true; length_next:=remaining_length - 4 end else begin burst_en=false; length_next:=remaining_length - 1 end... BURST_MODE : coverpoint burst_en; 7 Number of packets sent in burst mode increased from 4 to 5 in the condition, but designers forgot to increase packet number everywhere else

9. An Example Mutated Design false if (false) begin burst_en=true; length_next:=remaining_length - 4 end else begin burst_en=false; length_next:=remaining_length - 1 end... BURST_MODE : coverpoint burst_en; *The mutant goes undetected because the checkers are only sensitive to bus content and ordering, not timing 8

10. An Example Since the coverpoint BURST_MODE is suppressed by the mutant and the mutant is undetected, BURST_MODE is discounted It is now clear that the checker must check for timing in order to adequately cover burst mode functionality If timing is checked, the real bug is caught 9

11. Experimental Results 10 Experimental Results

12. OpenRISC CPU Experiment #1: Measure test quality – Simulation Infrastructure Functional simulator (ISS) Ad-hoc fault insertion – Tests: Instruction-Set v. Random – Coverpoints: Opcodes Experiment #2: Identify checker weaknesses – Simulation Infrastructure: SoC full-chip RTL simulation Certitude fault insertion – Tests: SoC Regression Suite – Coverpoints: CPU top-level signals 11

13. Measure Test Quality Functional Tests Original Discounted Random Tests 12

14. Identify Checker Weaknesses Tests 13

15. Identify Checker Weaknesses 14

16. UART Illustrates success of coverage discounting applied to a sophisticated testbench Simulation Infrastructure: – Design RTL w/OVM Testbench – SystemVerilog coverpoints (ModelSim) – Certitude fault insertion Tests: OVM Sequences Coverpoints: Hand-written spec-based 15

17. UART Results Reduces debug effort from analysis of 146 mutants to examining 3 functional coverpoints 1588 Mutants: – 7 not activated – 106 not propagated – 33 not detected – Total 146 mutants demand attention 846 Coverpoints: – 4 uncovered – 3 discounted – All discounted relate to specific unchecked functions Loopback, timeout interrupt identification register 16

18. Results Examined Coverage discounting: – Identified "good" tests (propagation) – Diagnosed checker problems – Identified coverpoints vacuously covered However: – Runtime of mutation analysis still an issue – Technique sensitive to the quantity and quality of faults 17

19. Confidence Metric Aims to answer: 1.Is a coverpoint adequately challenged by the set of mutants? 2.When have enough faults been inserted (when can simulation be stopped)? 3.What is the optimal simulation ordering of mutants for coverage discounting? 18

20. Detection Confidence (DECO) Score Discounting relies on coverpoints being suppressed by faults Point confidence: the number of times a coverpoint is suppressed DECO(n): computes the percentage of coverpoints with point confidence greater than n 19

21. DECO-Directed Ordering Optimize test/mutant simulation order to increase DECO score 1.Test Selection: Choose test covering the most low confidence points 2.Mutant Selection: Select mutant activated by the fewest tests 20

22. Quicker Confidence CPUUART 21

23. Quicker Discounting CPUUART 22

24. Summary – DECO Provides feedback earlier Improves confidence (both over time and at termination) Enables early termination 23

25. Summary Analyzed information gained from coverage discounting for two designs Developed a confidence metric to gauge mutant effectiveness and an ordering heuristic to reduce runtime 24