# 1 Applications of Optimization to Logic Testing Gary Kaminski and Paul Ammann ICST 2010 CSTVA Workshop.

## Presentation on theme: "1 Applications of Optimization to Logic Testing Gary Kaminski and Paul Ammann ICST 2010 CSTVA Workshop."— Presentation transcript:

1 Applications of Optimization to Logic Testing Gary Kaminski and Paul Ammann ICST 2010 CSTVA Workshop

2 Outline Motivation Test Sets and Coverage Criteria Should We Apply Well Known Optimization Techniques? Testing DNF Predicates For Logic Faults Minimal DNF Summary DNF Faults and Fault Hierarchy Coverage Criteria Optimizing DNF Criteria Results Optimized Test Set Sizes Benefit, Computational Cost Optimizing Individual Tests Implications For Regression Testing Thoughts On Where to Go from Here

3 Motivation Test Sets and Coverage Criteria Problem 1: Find Test Set that Satisfies Criterion Very Common Approach Often Tests Are Redundant Problem 2: Find Minimal Test Set that Satisfies Criterion Given A Test Set, Throw Out Redundant Tests Now Every Test Is There For a Reason! Problem 3: Find Minimum Test Set that Satisfies Criterion Generally Much Harder Can’t Be Found With “Local” Information Need To Make Globally Optimal Choices Question: Is It Useful To Recast Coverage As Standard Optimization Problem? Gain? Cost? Here We Address This Question In Context Of DNF Logic Testing

4 Minimal DNF Summary Terms separated by OR, literals by AND ab + a!c vs. a(b + !c) Make each term true and other terms false ab + ac vs. ab + abc Impossible to remove a literal without changing the predicate semantics ab vs. abc + ab!c Green – in minimal DNF Red – not in minimal DNF

5 Nine Minimal DNF Logic Faults Original: ab + bc Expression Negation Fault (ENF) !(ab + bc) Operator Reference Fault (ORF) abbc Operator Reference Fault (ORF) a + b + bc Term Negation Fault (TNF) !(ab) + bc Term Omission Fault (TOF) bc Literal Negation Fault (LNF) !ab + bc Literal Insertion Fault (LIF) abc + bc Literal Reference Fault (LRF) ac + bc Literal Omission Fault (LOF) a + bc

Fault Detection Hierarchy Expression Negation Fault ENF Literal Insertion Fault LIF Term Omission Fault TOF Literal Reference Fault LRF Literal Negation Fault LNF Operator Reference Fault ORF+ Literal Omission Fault LOF Term Negation Fault TNF Operator Reference Fault ORF* 6 Key Fact: Detection Relations Depend on Whether Fault is Feasible

7 Coverage Criteria Preamble : Unique True Points and Near False Points UTP: An assignment of values such that only one term evaluates to true. ab + bc: TTF is a UTP for term ab NFP: An assignment of values such that the predicate evaluates to false but when a literal is omitted, it evaluates to true. ab + bc: TFF is an NFP for b in term ab Closely Related to MCDC Notion

For each implicant find unique true points (UTPs) so that –Literals not in term take on values T and F Consider the DNF predicate: –f = ab + cd For term ab –Choose TTFT, TTTF For term cd –Choose FTTT, TFTT MUTP test set –{TTFT, TTTF, FTTT, TFTT} MUTP Criterion (Chen, Lau, Yu) 01 00 10110100 ab cd t t tt 11 10 t t t

CUTPNFP Criterion (Chen, Lau, Yu) Consider f = ab + cd For term ab –For a, choose UTP, NFP pair TTFF, FTFF –For b, choose UTP, NFP pair TTFT, TFFT For term cd –For c, choose UTP, NFP pair FFTT, FFFT –For d, choose UTP, NFP pair FFTT, FFTF Possible CUTPNFP test set –{TTFF, TTFT, FFTT //UTPs FTFF, TFFT, FFFT, FFTF} //NFPs 01 00 10110100 ab cd t t tt 11 10 t t t

Find NFP tests for each literal such that all literals not in the term attain F and T Consider the DNF predicate: – f = ab + cd For implicant ab –Choose FTFT, FTTF for a –Choose TFFT, TFTF for b For implicant cd –Choose FTFT, TFFT for c –Choose FTTF, TFTF for d MNFP test set –{TFTF, TFFT, FTTF, TFTF} Example is small, but generally MNFP is large MNFP Criterion (Chen, Lau, Yu) 01 00 10110100 ab cd t t tt 11 10 t t t

11 MUMCUT/Minimal-MUMCUT Criteria MUMCUT is simply MUTP + CUTPNFP + MNFP Minimal-MUMCUT omits tests via low level criterion feasibility analysis –Adds CUTPNFP and MNFP tests only when necessary Minimal-MUMCUT guarantees detecting LIF, LRF, LOF –And thus all 9 faults in the hierarchy, even if some faults infeasible Key Point: Minimal != Minimum Local vs. Global Perspective CUTPNFP feasible? MNFP Test Set = MUTP + MNFP For Each Literal In Term Test Set = MUTP + CUTPNFP MUTP feasible? Test Set = MUTP + NFP For Each Term

12 MUTP Optimization Minimize test set size subject to detecting LRFs with UTPs –LIFs Guaranteed Be Detected When LRFs are Feasible Each UTP is a 0-1 variable (integer programming) Each constraint represents the UTPs that detect a LRF Can reduce number of variables and constraints by focusing on a fault called the TRF-LIF instead of the LRF (see paper for details) ab + cd Minimize x1100 + x1101 + x1110 + x0011 + x0111 + x1011 Subject to x1100 + x1101 >= 1 (cb + cd) x1110 >= 1 (!cb + cd) x1100 + x1110 >= 1 (db + cd) x1101 >= 1 (!db + cd) …….

13 CUTPNFP Optimization Minimize test set size subject to finding UTP – NFP corresponding pairs Each point is a 0-1 variable Constraints represent an implication that the UTP and NFP be corresponding Additional constraint to reflect that an UTP must be selected ab + cd Minimize x1100 + x0100 + x1000 + x1101 + … Subject To x1100 <= M1; -- if 1100 is selected as an UTP then M1 must = 1 x0100 >= M1; -- if M1 = 1, 0100 must be chosen as a NFP x1000 >= M1; -- if M1 = 1, 1000 must be chosen as a NFP x1101 <= M2; -- if 1101 is selected as an UTP then M2 must = 1 x0101 >= M2; -- if M2 = 1, 0101 must be chosen as a NFP x1001 >= M2; -- if M2 = 1,1001 must be chosen as a NFP x1110 <= M3; -- if 1110 is selected as an UTP then M3 must = 1 x0110 >= M3; -- if M3 = 1, 0110 must be chosen as a NFP x1010 >= M3; -- if M3 = 1, 1010 must be chosen as a NFP x1100 + x1101 + x1110 >= 1; -- an UTP must be selected

14 MNFP Optimization Minimize test set size subject to detecting LRFs with NFPs Each NFP is a 0-1 variable Each constraint represents the NFPs that detect a LRF ab + cd Minimize x0100 + x0101 + x0110 + x1000 + x1001 + x1010... Subject to x0110 >= 1 (cb + cd) x0100 + x0101 >= 1 (!cb + cd) x1001 >= 1 (db + cd) x1000 + x1010 >= 1 (!db + cd) …….

15 Minimal-MUMCUT Optimization Applies optimization techniques for MUTP, CUTPNFP, and MNFP above based on criterion feasibility When MUTP is feasible, can select any NFP When MUTP is not feasible, but CUTPNFP is feasible, apply CUTPNFP optimization Otherwise, apply MNFP optimization ab + cd Minimize x0100 + x0101 + x0110 + x1000 + x1001 + x1010 + … Subject to x0100 + x0101 + x0110 >= 1 (any NFP for a works as MUTP is feasible) x1000 + x1001 + x1010 >= 1 (any NFP for b works as MUTP is feasible) …….

16 Case Study Analyzed 19 Boolean predicates in an avionics software system (Weyuker, Chen, Lau, and Yu) Number of unique literals range: 5 to 13 Built a tool in Java to automatically: -generate MUTP, CUTPNFP, MNFP, Minimal-MUMCUT tests based on heuristics -generate an optimization model that when solved by CPLEX yields minimized test sets Examined test set size and computation time difference between heuristics and optimization

17 Average Test Set Size Results CriterionMinimized Test Set Size Heuristic Test Set Size Percentage MUTP15.00 100.00% CUTPNFP41.1644.7991.90% MNFP37.8946.1682.08% Minimal- MUMCUT 40.3743.9591.85%

18 Average Computation Time Results (sec) CriterionHeuristicsOptimization (Model Generation and Solver) MUTP0.070.23 CUTPNFP0.111.33 MNFP0.225.95 Minimal- MUMCUT 5.156.09

19 Average Optimization Model Size CriterionNumber of Variables Number of Constraints MUTP15 CUTPNFP561336 MNFP414245 Minimal- MUMCUT 392106

20 Interpretation of Results For a small difference in computation time, optimization produces smaller test sets Running tests is time-consuming and expensive, so reducing test set size by one can be significant Choose optimization over heuristics

21 Fault Detection Maximization Testers have limitations on the number of tests they can run and thus it may be impossible to satisfy a criterion. In this case, it would help to know what tests maximize fault detection subject to a given test set size. Each point (test) is a 0-1 variable - x1100 represents the point (test) where a is T, b is T, c is F, d is F - x1100 will be 0 if it is not in the test set and 1 if it is in the test set Each fault is a 0-1 variable - LIF_1_c represents a fault where literal c is inserted into term 1 - LIF_1_c will be 0 if it is not detected and 1 if it is detected - objective function is to maximize the sum of all the fault variables - a fault variable exists for each fault based on the 9 DNF fault types

22 Fault Detection Maximization Consider ab + cd Maximize …. + LIF_1_c +... Subject to x1101 + x1100 >= M_LIF_1_c; (1) LIF_1_c <= M_LIF_1_c; (2) …… LIF_1_c represents a fault where c is inserted into term 1 (ab). If neither 1101 nor 1100 is selected, the x1101 and x1100 variables will be 0. For constraint (1) to hold, the variable M_LIF_1_c must then be 0. Thus, variable LIF_1_c must be 0 as well and the fault goes undetected. If 1101 or 1100 is selected, either x1101 and x1100 will be 1. This allows the variable M_LIF_1_c to be 1, which in turn allows LIF_1_c to be 1 and thus the fault is detected.

23 Case Study Used same predicates as in the first study Built a tool in Java to automatically: -generate an optimization model that when solved yields a test set that maximizes fault detection subject to a given test set size -generate an optimization model that when solved yields a test set that minimizes fault detection subject to a given test set size (by flipping constraints in the model) Used the tool to compare test set size between a best, random, and worst case scenario

24 Case Study Results Amongst all 19 predicates: As many as 288 times more faults are detected in the best case than the worst case for a single test. As many as 7 times more faults are detected in the best case than the random case for a single test. Implication for regression testing ScenarioFault Detection Percentage for 1 test Worst0.35% Random6.13% Best15.97%

25 Case Study Results

26 Fraser and Gargantini Study Found fault detecting tests for the same 19 predicates by formalizing the problem as a logic satisfiability problem which can be solved by a SAT algorithm (not optimization) 3 approaches 1)Minimal-MUMCUT heuristic  every test needed for fault detection 2)Satisfiability Algorithm  improves on heuristic 3)Optimization  guaranteed smallest test set size Minimal- MUMCUT Satisfiability Algorithm Optimization Avg Test Set Size43.9542.3740.37

27 Conclusion Applied optimization techniques in the Minimal DNF logic testing domain to: 1)Minimize test set size subject to guaranteeing fault detection (by satisfying a given criterion) 2)Maximize fault detection subject to a certain test set size Case study showed for a small sacrifice in computation time, optimization produced smaller test sets Case study showed large difference in fault detection between a best, random, and worst case scenario Applications in testing of programs with large predicates

28 Thoughts On Where To Go Next Reformulating Model To Handle Infeasible Tests –Crucial for Handling Predicates Embedded In Code –Requires Removal Of Some Constraints Some Faults Become May Become Equivalent Hence Constraint To Trigger Fault is Infeasible –Also Requires Adding Additional Constraints Fault Hierarchy Only Applies For Feasible Faults Applying Optimization To Other Coverage Criteria –Eg: Branch Coverage Building Optimal “Test Databases” –Given a Query With A Particular “Where” Clause –What Minimal Database Guarantees Detecting All 9 Faults in Query? Tool Available at Ammann/Offutt Book Web SiteBook Web Site –http://cs.gmu.edu/~offutt/softwaretest

Download ppt "1 Applications of Optimization to Logic Testing Gary Kaminski and Paul Ammann ICST 2010 CSTVA Workshop."

Similar presentations