1. An Analysis and Survey of the Development of Mutation Testing by Yue Jia and Mark Harmon A Quick Summary For SWE6673

2. Introductory Comments Mutation testing is a fault-based technique (testing to show existence or absence of specific faults)of developing “mutants” to be tested by a set of test cases. – The type of faults are mostly ---- syntax based faults Test case is ran against several mutant programs. The result is kept and a “Mutation Adequacy Score” is kept: Mutation Adequacy Score = (# of defect found)/( total # of seeded defects) or = (# of mutants killed)/(# of seeded non-equivalent mutants) ≤ 1 In a sense, Mutation Test is evaluating how good is the set of test cases An area that was started in 1970’s by Lipton, DeMillo & Hamlet

3. Mutation Testing (Theory & Process) 2 Underpinning Hypotheses (assumptions): 1.Programmers are competent and make simple errors 2.Errors and faults (defects) are coupled – detection of simple faults lead to detection of many complex faults Mutation Analysis & Execution Process: A.For a program P, generate (develop) mutation P’ of P B.Run the original program P with test cases T, fixing all the bugs in P. C.Run the mutation program P’ with test cases T Consider P’ “killed” (mutation detected) if results of running T against P and P’ are different D.Continue running all the P’s and score the “killed” P’s versus all the developed P’s ----- this “score” gives us the Mutation Adequacy Score. But --- Mutation Testing has Problems/Weaknesses: – There may be a “high number” of mutants and the cost of running them all. – High amount of “human effort” required in the analysis of mutants is costly.

4. Mutation Test Case Problem: Mutants Reduction Given a set of Mutants, M and a set of test cases T – let MS T (M) stand for mutation score of M with T – Then the problem is to find a subset of mutants M’ from M where: MS T (M) ≈ MS T (M’) or (# of killed M/non-equivalent M)≈ (# of killed M’/non-equivalent M’) What do you think ---? Is this really the same?

5. Mutation Test Case Problem: Mutants Reduction (cont.) Many different approaches have been tried to reduce the number of mutants for testing: – Random Sampling : found that randomly selecting 10% of mutants only reduces the effectiveness 16% – Mutation clustering: grouping mutants into clusters by traditional clustering techniques such as K-means and selecting representative mutants from each cluster – Selective Mutation: reducing the number of mutation operators to reduce the number of mutants generated without significant loss of effectiveness. – High Order Mutation: use higher order mutants (via multiple applications of mutation operators) to replace number of first order mutants

6. Techniques for Reducing Execution (Running) time and effort 1.Consider Strong, Weak, and Firm ways to analyze the killing of mutants during the execution process: – Strong: execute the whole program and if the mutant results differently, then it is “killed” – Weak: execute up to and include the part that is mutant and check the result – Firm: execute somewhere between Strong and Weak

7. Techniques for Reducing Execution (Running) time and effort (cont.) 2.Use different tools : – interpreter (expensive) – compiler (fast) – Compiler based (compile the whole original program P, but only compile the mutation part for each program from M – byte code translation to different platforms (for platform testing) – aspect oriented mutation – generate & compile different mutants 3.Use multiple platform such as distributed processors to simultaneously execute the mutants

8. Detection of “Equivalent Mutant” Problem Detecting that a program P and one of its mutant is equivalent is theoretically undecidable: Program P “Equivalent” Mutant P’ for (int i=0; i<5; i++) for (int i=0; i != 5; i++) { no change to { no change to value of i within loop value of i within loop } } P’ is a mutant of program P if P’ is syntactically different from P but is functionally same as P (e.g. produces same results). Mutation Score based on non-equivalent mutants without complete detection of “equivalent” mutants implies that we can never get to 100% on mutation score. How much does this bother you ---?

9. Mutation Testing May be Applied: to Various Artifacts To Program Source Code in following languages : – FROTRAN (22 mutant operators) – ADA – C (77 mutant operators) – JAVA (include OO mutant operators) – C++; SQL; PHP; AOP; COBOL; Spreadsheet Language To Specifications in : – Finite State Machine – Star Chart – Petri Net – Web services in XML – Description of runtime environments

10. Mutation Testing May also be Applied to ---- Constraint-based test data generation (found that 75% of the mutants can be killed with automatically generated test data.) [e.g. conditions in which mutants will die are written as algebraic constraints on test cases and then generate the test cases] Regression testing where we reuse the test data: – Reschedule the test sequence based on the mutant killing score – Minimize the test cases that need to be re-run for regression based on the mutant killing score of the test cases

11. Some Empirical Results Most of the earlier work dealt with code size of < 50 loc and increased in size as non-academic code was used Mutation testing developed test cases, when compared against “all-use” data flow test cases, actually subsumes the all use-test cases. (16% more defects were found with mutation generated test cases than all-use data flow) “Real world” software errors were compared against mutants, and found 85% of mutants were also “real world” faults. Human errors and mutants are different enough that both automated and human generated faults are needed for testing

12. Some Concluding Remarks Tools for Mutation Testing have increased in number since 2000. (30 years after its inception ---- technology maturation takes that long) Continued Barrier: – Perception of complexity and high cost – Decreasing (can not be solved)Equivalent Mutants problem is gaining some momentum Does thinking about Mutants provide us insight into defects and help us generate better test cases? ----- Your thoughts?

13. Now that you have read about Mutation Testing --- consider the following pseudo – code Input pairs as test cases Integer m, n, max ; {5,2 } max = 5 Input m, n; {10, 305} max = 305 If (m ≥ n) {20, 20} max = 20 max = m; {-23, -5} max = -5 else { 0, -4} max = 0 max = n; print max; Is this test set pretty good --- what mutant will not be killed? ----- what type of defect was this?