Yue Jia, Mark Harman King’s College London CREST Constructing Subtle Faults Using Higher Order Mutation Testing Higher Order Mutation Testing

Agenda Background Problems Solution Empirical Study Summary Mark Harman and Yue Jia King’s College London, CREST Centre BackgroundProblemsSolutionEmpirical StudySummary

Mutation Testing Mark Harman and Yue Jia King’s College London, CREST Centre ProblemsSolutionEmpirical StudySummary Background

Mutation Testing Which test case is better? Mark Harman and Yue Jia King’s College London, CREST Centre ProblemsSolutionEmpirical StudySummary Background

Mutation Testing Mark Harman and Yue Jia King’s College London, CREST Centre ProblemsSolutionEmpirical StudySummary Background

Mutation Testing Mark Harman and Yue Jia King’s College London, CREST Centre ProblemsSolutionEmpirical StudySummary Background

Mutation Testing Generate Mutants > First Order MutantOriginal Program Mark Harman and Yue Jia King’s College London, CREST Centre ProblemsSolutionEmpirical StudySummary Background

Mutation Testing Generate Mutants > + > Higher Order MutantFirst Order MutantOriginal Program Mark Harman and Yue Jia King’s College London, CREST Centre ProblemsSolutionEmpirical StudySummary Background

Mutation Testing Generate Mutants > + > Higher Order MutantFirst Order MutantOriginal Program > > First Order MutantHigher Order Mutant Mark Harman and Yue Jia King’s College London, CREST Centre ProblemsSolutionEmpirical StudySummary Background

Problem 1970s Computational cost Mark Harman and Yue Jia King’s College London, CREST Centre BackgroundSolutionEmpirical StudySummary Problems

Problem 1970s Computational cost e.g. Triangle LoC 50 FOM 500 HOM Mark Harman and Yue Jia King’s College London, CREST Centre BackgroundSolutionEmpirical StudySummary Problems

Problem 1970s Computational cost e.g. Triangle LoC 50 FOM 500 HOM Mark Harman and Yue Jia King’s College London, CREST Centre BackgroundSolutionEmpirical StudySummary Problems

Problem 1970s Computational cost e.g. Triangle LoC 50 FOM 500 HOM cost Mark Harman and Yue Jia King’s College London, CREST Centre BackgroundProblemsSolutionEmpirical StudySummary Problems

Solution 1980s Coupling effect hypothesis cost Mark Harman and Yue Jia King’s College London, CREST Centre BackgroundSolutionEmpirical StudySummary Problems

Solution 1980s Coupling effect hypothesis cost Mark Harman and Yue Jia King’s College London, CREST Centre BackgroundSolutionEmpirical StudySummary Problems

Solution 1980s Simple Complex cost If a test set kills simple faults, it also kills the complex faults Mark Harman and Yue Jia King’s College London, CREST Centre BackgroundSolutionEmpirical StudySummary Problems

Solution 1980s Coupling effect hypothesis cost Mark Harman and Yue Jia King’s College London, CREST Centre BackgroundSolutionEmpirical StudySummary Problems

Solution 1990s Selective mutation / Mutation sampling cost Mark Harman and Yue Jia King’s College London, CREST Centre BackgroundSolutionEmpirical StudySummary Problems

Selected First Order Mutants > > < < a a b b Trivial Fault Mark Harman and Yue Jia King’s College London, CREST Centre BackgroundSolutionEmpirical StudySummary Problems

Subtle Fault 90% of the real faults are complex faults (Purushothaman and Perry) HOMs -> Subtle faults Mark Harman and Yue Jia King’s College London, CREST Centre BackgroundSolutionEmpirical StudySummary Problems

Subtle Fault Solution Programmer Mark Harman and Yue Jia King’s College London, CREST Centre BackgroundSolutionEmpirical StudySummary Problems

Subtle Fault Solution Programmer 1 Mark Harman and Yue Jia King’s College London, CREST Centre BackgroundSolutionEmpirical StudySummary Problems

Subtle Fault Solution Programmer 2 Mark Harman and Yue Jia King’s College London, CREST Centre BackgroundSolutionEmpirical StudySummary Problems

Subtle Fault Solution Programmer 3 Mark Harman and Yue Jia King’s College London, CREST Centre BackgroundSolutionEmpirical StudySummary Problems

Subtle Fault 4 Mark Harman and Yue Jia King’s College London, CREST Centre BackgroundSolutionEmpirical StudySummary Problems Solution Programmer

Solution 1990s Selective mutation / Mutation sampling cost Mark Harman and Yue Jia King’s College London, CREST Centre BackgroundSolutionEmpirical StudySummary Problems

Solution 1980s Coupling effect hypothesis cost Mark Harman and Yue Jia King’s College London, CREST Centre BackgroundSolutionEmpirical StudySummary Problems

cost Problem 1970s Computational cost Mark Harman and Yue Jia King’s College London, CREST Centre BackgroundSolutionEmpirical StudySummary Problems

Solution Higher order mutation testing cost Mark Harman and Yue Jia King’s College London, CREST Centre BackgroundProblemsEmpirical StudySummary Solution

Higher order mutation testing …Search Based Optimization cost Mark Harman and Yue Jia King’s College London, CREST Centre BackgroundProblemsEmpirical StudySummary Solution

Higher Order Mutation Testing Increase the subtlety Reduce the Oracle effort Reduced number of equivalent mutants Mark Harman and Yue Jia King’s College London, CREST Centre BackgroundProblemsEmpirical StudySummary Solution

Higher Order Mutant Most common case Test set T Ta Tb Tc Tabc Mark Harman and Yue Jia King’s College London, CREST Centre BackgroundProblemsEmpirical StudySummary Solution

Higher Order Mutant Uncommon case Ta Tb Tc Tabc Mark Harman and Yue Jia King’s College London, CREST Centre BackgroundProblemsEmpirical StudySummary Solution

Higher Order Mutant Subsuming higher order mutant Ta Tb Tc Tabc Mark Harman and Yue Jia King’s College London, CREST Centre BackgroundProblemsEmpirical StudySummary Solution

Higher Order Mutant Subsuming higher order mutant Ta Tb Tc Tabc Mark Harman and Yue Jia King’s College London, CREST Centre BackgroundProblemsEmpirical StudySummary Solution

Higher Order Mutant Strongly Subsuming higher order mutant Ta Tb Tc Tabc If a test case kills HOM M abc, it also kills the FOM F a, F b and F c Mark Harman and Yue Jia King’s College London, CREST Centre BackgroundProblemsEmpirical StudySummary Solution

Higher Order Mutation Testing Increase the subtlety Reduce the Oracle effort Reduced number of equivalent mutants Mark Harman and Yue Jia King’s College London, CREST Centre BackgroundProblemsEmpirical StudySummary Solution

Higher Order Mutation Testing Increase the subtlety Reduce the Oracle effort Mutants Reduced number of equivalent mutants Mark Harman and Yue Jia King’s College London, CREST Centre BackgroundProblemsEmpirical StudySummary Solution

Higher Order Mutation Testing Increase the subtlety Reduce the Oracle effort Mutants Test cases Reduced number of equivalent mutants Mark Harman and Yue Jia King’s College London, CREST Centre BackgroundProblemsEmpirical StudySummary Solution

Higher Order Mutation Testing Increase the subtlety Reduce the Oracle effort Mutants Test cases Reduced number of equivalent mutants Mark Harman and Yue Jia King’s College London, CREST Centre BackgroundProblemsEmpirical StudySummary Solution

Results Mutants LoC FOM sHOM Triangle Tcas Schedule Totinfo Printtokens Space Mark Harman and Yue Jia King’s College London, CREST Centre BackgroundProblemsSolution Empirical Study Summary

Algorithms Greedy Hill Climbing Genetic Algorithm Mark Harman and Yue Jia King’s College London, CREST Centre BackgroundProblemsSolutionSummary Empirical Study

Results Mark Harman and Yue Jia King’s College London, CREST Centre BackgroundProblemsSolutionSummary Empirical Study

Triangle Program Classify triangle by the lengths of the sides Equilateral Isosceles Scalene Mark Harman and Yue Jia King’s College London, CREST Centre BackgroundProblemsSolutionSummary Empirical Study

Strongly Subsuming HOM if(trian == 1 && a + b > c) if(trian > 1 && a + b > c) if(trian == 1 && a + b <= c) a == c && a + b > c && a + c <= b b == c && a + b > a && b + c <= a a == b && a + b > c a == b && a + b <= c Mark Harman and Yue Jia King’s College London, CREST Centre BackgroundProblemsSolutionSummary Empirical Study

Tool MiLu Deer Horse Cow Donkey rarevaluable Strongly Subsuming HOM (Père David's Deer) Mark Harman and Yue Jia King’s College London, CREST Centre BackgroundProblemsSolutionSummary Empirical Study

Mark Harman and Yue Jia King’s College London, CREST Centre BackgroundProblemsSolutionSummary Empirical Study

Summary Higher order mutation testing Quality of mutant Computational Cost Question Are real faults higher order mutants? Thanks Mark Harman and Yue Jia King’s College London, CREST Centre BackgroundProblemsSolutionEmpirical Study Summary