1. Yue Jia, Mark Harman King’s College London CREST Constructing Subtle Faults Using Higher Order Mutation Testing Higher Order Mutation Testing

2. Agenda Background Problems Solution Empirical Study Summary Mark Harman and Yue Jia King’s College London, CREST Centre BackgroundProblemsSolutionEmpirical StudySummary

3. Mutation Testing Mark Harman and Yue Jia King’s College London, CREST Centre ProblemsSolutionEmpirical StudySummary Background

4. Mutation Testing Which test case is better? Mark Harman and Yue Jia King’s College London, CREST Centre ProblemsSolutionEmpirical StudySummary Background

5. Mutation Testing Mark Harman and Yue Jia King’s College London, CREST Centre ProblemsSolutionEmpirical StudySummary Background

6. Mutation Testing Mark Harman and Yue Jia King’s College London, CREST Centre ProblemsSolutionEmpirical StudySummary Background

7. Mutation Testing Generate Mutants ----------------- ----------------- ----------------- + > First Order MutantOriginal Program Mark Harman and Yue Jia King’s College London, CREST Centre ProblemsSolutionEmpirical StudySummary Background

8. Mutation Testing Generate Mutants ----------------- ----------------- ----------------- + > + > Higher Order MutantFirst Order MutantOriginal Program Mark Harman and Yue Jia King’s College London, CREST Centre ProblemsSolutionEmpirical StudySummary Background

9. Mutation Testing Generate Mutants ----------------- ----------------- ----------------- + > + > Higher Order MutantFirst Order MutantOriginal Program ----------------- ----------------- ----------------- + > ----------------- ----------------- ----------------- ----------------- ----------------- ---------------- + ----------------- ----------------- ----------------- > First Order MutantHigher Order Mutant Mark Harman and Yue Jia King’s College London, CREST Centre ProblemsSolutionEmpirical StudySummary Background

10. Problem 1970s Computational cost Mark Harman and Yue Jia King’s College London, CREST Centre BackgroundSolutionEmpirical StudySummary Problems

11. Problem 1970s Computational cost e.g. Triangle LoC 50 FOM 500 HOM 2 500 Mark Harman and Yue Jia King’s College London, CREST Centre BackgroundSolutionEmpirical StudySummary Problems

12. Problem 1970s Computational cost e.g. Triangle LoC 50 FOM 500 HOM 2 500 Mark Harman and Yue Jia King’s College London, CREST Centre BackgroundSolutionEmpirical StudySummary Problems

13. Problem 1970s Computational cost e.g. Triangle LoC 50 FOM 500 HOM 2 500 cost Mark Harman and Yue Jia King’s College London, CREST Centre BackgroundProblemsSolutionEmpirical StudySummary Problems

14. Solution 1980s Coupling effect hypothesis cost Mark Harman and Yue Jia King’s College London, CREST Centre BackgroundSolutionEmpirical StudySummary Problems

15. Solution 1980s Coupling effect hypothesis cost Mark Harman and Yue Jia King’s College London, CREST Centre BackgroundSolutionEmpirical StudySummary Problems

16. Solution 1980s Simple Complex cost If a test set kills simple faults, it also kills the complex faults Mark Harman and Yue Jia King’s College London, CREST Centre BackgroundSolutionEmpirical StudySummary Problems

17. Solution 1980s Coupling effect hypothesis cost Mark Harman and Yue Jia King’s College London, CREST Centre BackgroundSolutionEmpirical StudySummary Problems

18. Solution 1990s Selective mutation / Mutation sampling cost Mark Harman and Yue Jia King’s College London, CREST Centre BackgroundSolutionEmpirical StudySummary Problems

19. Selected First Order Mutants + + - - > > < < a a b b Trivial Fault Mark Harman and Yue Jia King’s College London, CREST Centre BackgroundSolutionEmpirical StudySummary Problems

20. Subtle Fault 90% of the real faults are complex faults (Purushothaman and Perry) ------------------------------ ------------------------------ ------------------------------ ------------------------------ ------------------------------ ------------------------------ ------------------------------ ------------------------------ ------------------------------ ---------------------------- HOMs -> Subtle faults Mark Harman and Yue Jia King’s College London, CREST Centre BackgroundSolutionEmpirical StudySummary Problems

21. Subtle Fault Solution Programmer Mark Harman and Yue Jia King’s College London, CREST Centre BackgroundSolutionEmpirical StudySummary Problems

22. Subtle Fault Solution Programmer 1 Mark Harman and Yue Jia King’s College London, CREST Centre BackgroundSolutionEmpirical StudySummary Problems

23. Subtle Fault Solution Programmer 2 Mark Harman and Yue Jia King’s College London, CREST Centre BackgroundSolutionEmpirical StudySummary Problems

24. Subtle Fault Solution Programmer 3 Mark Harman and Yue Jia King’s College London, CREST Centre BackgroundSolutionEmpirical StudySummary Problems

25. Subtle Fault 4 Mark Harman and Yue Jia King’s College London, CREST Centre BackgroundSolutionEmpirical StudySummary Problems Solution Programmer

26. Solution 1990s Selective mutation / Mutation sampling cost Mark Harman and Yue Jia King’s College London, CREST Centre BackgroundSolutionEmpirical StudySummary Problems

27. Solution 1980s Coupling effect hypothesis cost Mark Harman and Yue Jia King’s College London, CREST Centre BackgroundSolutionEmpirical StudySummary Problems

28. cost Problem 1970s Computational cost Mark Harman and Yue Jia King’s College London, CREST Centre BackgroundSolutionEmpirical StudySummary Problems

29. Solution Higher order mutation testing cost Mark Harman and Yue Jia King’s College London, CREST Centre BackgroundProblemsEmpirical StudySummary Solution

30. Higher order mutation testing …Search Based Optimization cost Mark Harman and Yue Jia King’s College London, CREST Centre BackgroundProblemsEmpirical StudySummary Solution

31. Higher Order Mutation Testing Increase the subtlety Reduce the Oracle effort Reduced number of equivalent mutants Mark Harman and Yue Jia King’s College London, CREST Centre BackgroundProblemsEmpirical StudySummary Solution

32. Higher Order Mutant Most common case Test set T Ta Tb Tc Tabc Mark Harman and Yue Jia King’s College London, CREST Centre BackgroundProblemsEmpirical StudySummary Solution

33. Higher Order Mutant Uncommon case Ta Tb Tc Tabc Mark Harman and Yue Jia King’s College London, CREST Centre BackgroundProblemsEmpirical StudySummary Solution

34. Higher Order Mutant Subsuming higher order mutant Ta Tb Tc Tabc Mark Harman and Yue Jia King’s College London, CREST Centre BackgroundProblemsEmpirical StudySummary Solution

35. Higher Order Mutant Subsuming higher order mutant Ta Tb Tc Tabc Mark Harman and Yue Jia King’s College London, CREST Centre BackgroundProblemsEmpirical StudySummary Solution

36. Higher Order Mutant Strongly Subsuming higher order mutant Ta Tb Tc Tabc If a test case kills HOM M abc, it also kills the FOM F a, F b and F c Mark Harman and Yue Jia King’s College London, CREST Centre BackgroundProblemsEmpirical StudySummary Solution

37. Higher Order Mutation Testing Increase the subtlety Reduce the Oracle effort Reduced number of equivalent mutants Mark Harman and Yue Jia King’s College London, CREST Centre BackgroundProblemsEmpirical StudySummary Solution

38. Higher Order Mutation Testing Increase the subtlety Reduce the Oracle effort Mutants Reduced number of equivalent mutants Mark Harman and Yue Jia King’s College London, CREST Centre BackgroundProblemsEmpirical StudySummary Solution

39. Higher Order Mutation Testing Increase the subtlety Reduce the Oracle effort Mutants Test cases Reduced number of equivalent mutants Mark Harman and Yue Jia King’s College London, CREST Centre BackgroundProblemsEmpirical StudySummary Solution

40. Higher Order Mutation Testing Increase the subtlety Reduce the Oracle effort Mutants Test cases Reduced number of equivalent mutants Mark Harman and Yue Jia King’s College London, CREST Centre BackgroundProblemsEmpirical StudySummary Solution

41. Results Mutants LoC FOM sHOM Triangle 50 584 47 Tcas 150 679 98 Schedule2 350 1014 78 Totinfo 500 2570 320 Printtokens 750 866 67 Space 6000 7570 522 Mark Harman and Yue Jia King’s College London, CREST Centre BackgroundProblemsSolution Empirical Study Summary

42. Algorithms Greedy Hill Climbing Genetic Algorithm Mark Harman and Yue Jia King’s College London, CREST Centre BackgroundProblemsSolutionSummary Empirical Study

43. Results Mark Harman and Yue Jia King’s College London, CREST Centre BackgroundProblemsSolutionSummary Empirical Study

44. Triangle Program Classify triangle by the lengths of the sides Equilateral Isosceles Scalene Mark Harman and Yue Jia King’s College London, CREST Centre BackgroundProblemsSolutionSummary Empirical Study

45. Strongly Subsuming HOM if(trian == 1 && a + b > c) if(trian > 1 && a + b > c) if(trian == 1 && a + b <= c) a == c && a + b > c && a + c <= b b == c && a + b > a && b + c <= a a == b && a + b > c a == b && a + b <= c Mark Harman and Yue Jia King’s College London, CREST Centre BackgroundProblemsSolutionSummary Empirical Study

46. Tool MiLu Deer Horse Cow Donkey rarevaluable Strongly Subsuming HOM (Père David's Deer) Mark Harman and Yue Jia King’s College London, CREST Centre BackgroundProblemsSolutionSummary Empirical Study

47. www.dcs.kcl.ac.uk/pg/jiayue/milu Mark Harman and Yue Jia King’s College London, CREST Centre BackgroundProblemsSolutionSummary Empirical Study

48. Summary Higher order mutation testing Quality of mutant Computational Cost Question Are real faults higher order mutants? Thanks Mark Harman and Yue Jia King’s College London, CREST Centre BackgroundProblemsSolutionEmpirical Study Summary