Software Logic Mutation Testing Presented by Gary Kaminski

Overview Introduction and Motivation Logic Faults and Detection Case Study Comparison Context Conclusion

Logic Mutation Testing Process 1)Systematically create program variations (mutants) each containing a single typical logic fault if (a && b)  if (a || b) 2)For each mutant, find an input that distinguishes the original program from the mutant program (killing) - a && b and a || b evaluate to different truth values Focus is on step 1)

Logic Mutation Testing Logic mutants are created by applying mutation operators to Boolean predicates Boolean predicates may come from: 1)UML Diagrams 2) Security Requirements 3) Looping constructs (while / do-while / for) 4) Preconditions for methods (if - then)

Motivation Current mutation systems: –generate excess logic mutants –do not detect certain classes of logic faults Goal: - generate fewer logic mutants - detect more logic faults Assumption: - formally restrict attention to minimal Disjunctive Normal Form (DNF) Boolean predicates

Minimal DNF Terms separated by OR, literals by AND ab + a!c vs. a(b + !c) Make each term true and other terms false ab + ac vs. ab + abc Impossible to remove a literal or term without changing the predicate ab vs. abc + ab!c

Excess Logic Mutants - Example Original: ab + b!c + !bc 1) mutant generated multiple times: ab + _ + !bc 2) equivalent mutants: ac + b!c + !bc 3) mutants guaranteed to be killed by an input that kills some other mutant: !ab + b!c + !bc and cb + b!c + !bc 4) highly effective mutation operators not used: abc + b!c + !bc

Minimal DNF Logic Faults Original: ab + b!c + !bc Expression Negation Fault: !(ab + b!c + !bc) Expression Stuck At Fault 0: FALSE Expression Stuck At Fault 1: TRUE Term Negation Fault: !(ab) + b!c + !bc Term Omission Fault: ab + _ + !bc Scalar Variable Replacement TOF: ab + c!c + !bc TOF generates same mutant twice: ab + _ + !bc

Minimal DNF Logic Faults Original: ab + b!c + !bc Operator Reference Fault+: abb!c + !bc Operator Reference Fault.: a + b + b!c + !bc Operator Reference Fault+0: FALSE + !bc Operator Reference Fault+1: TRUE + !bc Operator Reference Fault.0: FALSE + b!c + !bc Operator Reference Fault.1: TRUE + b!c + !bc

Minimal DNF Logic Faults Original: ab + b!c + !bc Literal Negation Fault: !ab + b!c + !bc Literal Stuck At Fault 0: FALSEb + b!c + !bc Literal Stuck At Fault 1: TRUEb + b!c + !bc Literal Omission Fault: _b + b!c + !bc SVR LOF: bb + b!c + !bc LOF generates same mutant twice original: abc  _bc, a_c, a_c, ab_

Minimal DNF Logic Faults Original: ab + b!c + !bc Literal Reference Fault: ac + b!c + !bc Literal Reference Fault: a!c + b!c + !bc Literal Insertion Fault: abc + b!c + !bc Literal Insertion Fault: ab!c + b!c + !bc LRF and LIF generate equivalent mutants

Logic Fault Minimal DNF Detection Green Arrow – Detection may not hold Black Arrow – Detection always holds Red Box – Mutation Operator does not exist Blue Box – Mutation Operator partially exists Black Box – Mutation Operator exists LOF ORF. LRF LNF TNF LIF TOF LSTF0 ORF.0 ESTF0 ORF+ ORF+0 ENF LSTF1 ORF.1 ORF+1 ESTF1 SVRTOF SVRLOF

Logic Mutation Comparison Original: ab + b!c + !bc 61 mutants vs. 4 LIFs, 3 LOFs, 3 LRFs Equivalent LIF: abc + b!c + !bc LRF: cb + b!c + !bc & ac + b!c + !bc LOF: a_ + b!c + !bc Equivalent LIF: ab + b!c!a + !bc LRFs: ab + !a!c + !bc & ab + b!a + !bc No Equivalent LIFs for term !bc LOFs: ab + b!c + _c & ab + b!c + !b_

Mutation Efficiency Measures Mutation Efficiency Score = Faults Detected - Mutants Generated Mutation Efficiency Ratio = Faults Detected / Mutants Generated FD assumes an input is found to kill the mutant FD includes equivalent mutants detected automatically ab + b!c + !bc Current Mutation Systems: MES: 60 FD - 61 MG = -1 MER: 60 FD / 61 MG =.98 LIF-LRF-LOF MES: 73 FD – 10 MG = 63 MER: 73 FD / 10 MG = 7.30

Case Study Analyzed 19 Boolean predicates in an avionics software system Number of unique literals range: 5 to 13 Compared MES and MER for current mutation systems vs. LIF-LRF-LOF Examined how MES and MER vary based on number of unique literals

Case Study Results LIF-LRF-LOF generated 18% of the mutants current mutation systems would (150 vs. 819) Current mutation systems guarantee detecting 75% of faults LIF-LRF-LOF does (819 vs. 1082) Number of literals matters for MES but not for MER MES = FD – MG and MER = FD / MG CurrentLIF-LRF-LOF MES 0932 MER

Mutation Efficiency Score (FD – MG) vs. Number of Unique Literals

Mutation Efficiency Ratio (FD/MG) vs. Number of Unique Literals

Context: Internal Variable Problem What input values kill a mutant? Mutation may be buried deep in the code Must reach predicate and have mutant predicate and original predicate evaluate to different truth values Partial solutions using constraints exist

Context: Minimal DNF in Practice 1)95% of 20,256 Boolean predicates in avionics software were in minimal DNF* 2) Detected > 99% of corresponding faults in non-minimal DNF Boolean predicates* *Source: Y.T Yu and M.F. Lau. Comparing Several Coverage Criteria for Detecting Faults in Logical Decisions. In Proceedings QSIC 2004: 4th International Conference on Quality Software, Pages

Context: Industry Audience LIF-LRF-LOF for software with Boolean predicates having >= 5 unique literals Exhaustive coverage for < 5 unique literals ab + b!c + bc (8 tests vs. 10 mutants) Avionics software* *Source: J.J Chilenski and S.P. Miller. Applicability of modified condition/decision coverage to software testing. IEE/BCS Software Engineering Journal, 9(5): , September 1994.

Related Work 1)J.J. Chilenski. An Investigation of Three Forms of the Modified Condition Decision Coverage (MCDC) Criterion. Final Technical Report, DOT/FAA/AR-01/18, April Non-Minimal DNF based logic coverage criteria 2) T.Y. Chen, M.F. Lau, and Y.T. Yu. MUMCUT: A Fault-Based Strategy for Testing Boolean Specifications. Software Engineering Conference, (APSEC ’99) Proceedings. Sixth Asia Pacific. Pages Minimal DNF based logic coverage criteria and Double Faults 3) D. Richard Kuhn. Fault Classes and Error Detection Capability of Specification Based Testing. ACM Transactions on Software Engineering and Methodology, 8(4): , October Seminal work in logic fault hierarchy

Conclusion Current logic mutation testing is inefficient Excess mutants generated and faults are missed Fault hierarchy approach based on minimal DNF: detect more faults with less mutants Applications in avionics software

Other Research and Publications Logic Coverage Criteria: Reduce test set size and maintain fault detection Use feasibility of smaller, less expensive component criteria Internal variable problem CoJava and constraint programming Publications: G. Kaminski, G. Williams, and P. Ammann. Reconciling Perspectives of Logic Testing for Software. Submitted to Journal of Software Testing, Verification, and Reliability, January Online.