1. Analyzing the Validity of Selective Mutation with Dominator Mutants
FSE 2016, Seattle, Washington, USA
Bob Kurtz, Paul Ammann, Jeff Offutt, Marcio E. Delamaro, Mariet Kurtz, and Nida Gökçe
George Mason University, USA
Universidade de São Paulo, Brazil
The MITRE Corporation
Muğla Sıtkı Koçman University, Turkey

2. Mutation testing in 2016
MASON
GMU
FSE 2016
ACME Co.

3. GMU Mutation testing in 2016 int max(int i, int j) { if (i > j) {
return i; }
else {
return j;
int max(int i, int j) {
if (i <= j) {
return i; }
else {
return j;
int max(int i, int j) {
if (i > j) {
return j; }
else {
int max(int i, int j) {
if (i > j) {
return i; }
else {
return j++;
Mutation
Operators
ROR
LVR
AOI
MASON
GMU
FSE 2016
ACME Co.

4. ≠ GMU   Mutation testing in 2016 Original Mutants assertEquals(2,
int max(int i, int j) {
if (i > j) {
return i; }
else {
return j; ≠  
int max(int i, int j) {
if (i <= j) {
return i; }
else {
return j;
int max(int i, int j) {
if (i > j) {
return j; }
else {
int max(int i, int j) {
if (i > j) {
return i; }
else {
return j++;
int max(int i, int j) {
if (i > j) {
return i; }
else {
int max(int i, int j) {
if (i <= j) {
return i; }
else {
return j;
int max(int i, int j) {
if (i > j) {
return j; }
else {
int max(int i, int j) {
if (i > j) {
return i; }
else {
return j++;
int max(int i, int j) {
if (i > j) {
return i; }
else {
int max(int i, int j) {
if (i >= j) {
return i; }
else {
return j;
int max(int i, int j) {
if (i > i) {
return i; }
else {
return j;
int max(int i, int j) {
if (i) {
return i; }
else {
return j;
int max(int i, int j) {
if (i > j) {
return --i; }
else {
return j;
assertEquals(2,
max(2,1));
int max(int i, int j) {
if (i >= j) {
return i; }
else {
return j;
int max(int i, int j) {
if (i > i) {
return i; }
else {
return j;
int max(int i, int j) {
if (i) {
return i; }
else {
return j;
int max(int i, int j) {
if (i > j) {
return --i; }
else {
return j;
MASON
GMU
FSE 2016
ACME Co.

5. Mutation testing in 2016 20,000 Mutants 2,000 SLOC ACME Co. FSE 2016
int max(int i, int j) {
if (i > j) {
return i; }
else {
return j;
2,000 SLOC
FSE 2016
ACME Co.

6. Mutation testing in 2016
FSE 2016
ACME Co.

7. =  Mutation testing in 2016 int max(int i, int j) { if (i > j) {
return i; }
else {
return j;
if (i >= j) { = 
FSE 2016
ACME Co.

8. Mutation testing in 2016
FSE 2016
ACME Co.

9. Path too long, no end in sight!
Mutation testing in 2016
???
Equivalent mutants!
You can’t kill ‘em!
How many?
10-20%
Path too long, no end in sight!
MASON
GMU
FSE 2016
ACME Co.

10. Mutation testing in 2016
FSE 2016
ACME Co.

11. =    What went wrong? Equivalent mutants
Syntactically different but semantically identical to the original program
Cannot be killed by tests, must be manually evaluated one-by-one
Requires unrealistic amounts of work!
int max(int i, int j) {
if (i > j) {
return i; }
else {
return j;
int max(int i, int j) {
if (i >= j) {
return i; }
else {
return j;
int max(int i, int j) {
if (i > j) {
return i; }
else {
return j++; =   
FSE 2016

12. What went wrong? Redundant mutants
A mutant is redundant if it is always killed when some other mutant is killed
≈98% of non-equivalent mutants
How far along is testing?
FSE 2016

13. Mutant reduction strategies
int max(int i, int j) {
if (i > j) {
return i; }
else {
return j;
Selective mutation
Use the “best” operators to produce fewer mutants
CCDL
ORAN
VLSR …
OAAN
SRSR
FSE 2016

14. Mutant reduction strategies
int max(int i, int j) {
if (i > j) {
return i; }
else {
return j;
Random mutant selection
Typically select ≈5% of all mutants
CCDL
OAAN
ORAN
SRSR
VLSR …
FSE 2016

15. Research questions
Selective mutation has generally been thought of as the best approach since 1993
Random mutant selection has been used since 1980
How effective are these techniques?
Can we improve on them?
FSE 2016

16. mi → mj Mutant subsumption
Given a set of mutants M, mutant mi subsumes mutant mj (mi → mj) iff:
Some test kills mi
All tests that kill mi also kill mj
All Tests +
Tests that kill mj
Tests that kill mk
Tests that kill mi
mi → mj
FSE 2016
[Ammann, et al., ICST 2014]

17. Subsumption graph m1 m1 m2 m3 m2 m3 m4 m4 m5 m5 m6 m7 m6 m7 m8 m8
Shows the subsumption relationship between mutants
Leaf nodes not subsumed by any other mutant are “dominator mutants”
Tests that kill all these dominators will also kill all other mutants
All the other mutants are redundant! m1 m1 m2 m3 m2 m3 m4 m4 m5 m5 m6 m7 m6 m7 m8 m8
FSE 2016
[Kurtz, et al., Mutation 2014]

18. Using selective mutation, random mutants, or some other technique
Testing model
Begin
Using selective mutation, random mutants, or some other technique
Mutants
All Mutants
Select some mutant set M
Mutants
All Tests
Select minimal test set T that kills M
Find all mutants killed by T
Mutation Scores
Dominator Scores
End
FSE 2016

19. Nondeterministic process
Testing model
Begin
Mutants
All Mutants
Select some mutant set M
Nondeterministic process
Mutants
All Tests
Select minimal test set T that kills M
Find all mutants killed by T
Mutation Scores
Dominator Scores
End
FSE 2016

20. Testing model Begin Mutants All Mutants Select some mutant set M
#𝑚𝑢𝑡𝑎𝑛𝑡 𝑠 𝑘𝑖𝑙𝑙𝑒𝑑 #𝑚𝑢𝑡𝑎𝑛𝑡 𝑠 𝑡𝑜𝑡𝑎𝑙 −#𝑚𝑢𝑡𝑎𝑛𝑡 𝑠 𝑒𝑞𝑢𝑖𝑣𝑎𝑙𝑒𝑛𝑡
Mutants
All Tests
Select minimal test set T that kills M
#𝑑𝑜𝑚𝑖𝑛𝑎𝑡𝑜 𝑟𝑠 𝑘𝑖𝑙𝑙𝑒𝑑 #𝑑𝑜𝑚𝑖𝑛𝑎𝑡𝑜 𝑟𝑠 𝑡𝑜𝑡𝑎𝑙
Find all mutants killed by T
Mutation Scores
Dominator Scores
End
FSE 2016

21. Siemens suite scores
Mutation and dominator score using the 5 E-Selective mutation operators from Mothra [Offutt, et al., 1993]
FSE 2016
[Ammann, et al., ICST 2014]

22. The Siemens suite? Really??
REAL SOFTWARE
SIEMENS SUITE
FSE 2016

23. The Siemens suite? Really!
The Siemens suite is useful here because:
It has a thorough test suite that elicits rich subsumption relationships between mutants
The Proteum mutation tool for C has a very large set of mutation operators
If we can show that selective mutation is not a good solution for any particular program, then it is not a good solution for programs in general
FSE 2016

24. Mutation vs. Dominator Score
Mutation score for the Siemens replace program (1,000 iterations)
FSE 2016
[Kurtz, et al., Mutation 2016]

25. Mutation vs. Dominator Score
Mutation and dominator score for the Siemens replace program
FSE 2016
[Kurtz, et al., Mutation 2016]

26. 1-4 Operators for print_tokens
Our Goal
We define work as the number of mutants the engineer must evaluate (write a killing test or to show it’s equivalent)
FSE 2016

27. 1-4 Operators for print_tokens
231,525 operator combinations
We define work as the number of mutants the engineer must evaluate (to write a killing test or to show it’s equivalent)
FSE 2016

28. 1-4 Operators for print_tokens
Optimal selective operator combinations for this program
231,525 operator combinations
FSE 2016

29. 1-4 Operators for Siemens suite
Operator combination are far less optimal for all programs
There is no set of up to 4 operators that is good for every program
489,504 operator combinations
FSE 2016

30. Selective and random
Traditional mutant reduction approaches are not optimal, consistent with prior research that selective and random are similar
FSE 2016

31. More than 4 operators m1 m1 m2 m3 m2 m3 m4 m4 m5 m6 m5 m5 m6 m7 m6 m7
We’d like to expand to sets of mutation operators larger than 4
Brute force approach is infeasible
1-4 operators = 489,504 sets ≈ 24 hours
1-59 operators ≈ 6x1017 sets ≈ 3x109 years
Approximate test results based on subsumption
Spearman’s rank ρ=0.966
Recalculate best operator combinations in the usual way m1 m1 m2 m3 m2 m3 m4 m4 m5 m6 m5 m5 m6 m7 m6 m7 m8 m8
FSE 2016

32. Optimal selective solutions
Best 1-program solutions
Best 1-59 operator combinations
Even the optimal operator combinations are not very good
FSE 2016

33. Is mutation testing dead?
I don’t think so, but we need to do something better!
FSE 2016

34. Mutation testing in 2020?
Use machine learning to generate optimized mutants based on program features
int max(int i, int j) {
if (i > j) {
return j; }
else {
int max(int i, int j) {
if (i > j) {
return i; }
else {
return j++;
int max(int i, int j) {
if (i > j) {
return i; }
else {
return j;
int max(int i, int j) {
if (i > j) {
return i; }
else {
int max(int i, int j) {
if (i > i) {
return i; }
else {
return j;
int max(int i, int j) {
if (i > j) {
return i; }
else {
return j;
int max(int i, int j) {
if (i) {
return i; }
else {
return j;
int max(int i, int j) {
if (i > j) {
return --i; }
else {
return j;
FSE 2016
[Future work]

35. Mutation testing in 2020?
Use machine learning to generate optimized mutants based on program features
Use static analysis to determine partial subsumption & tests
int max(int i, int j) {
if (i > j) {
return i; }
else {
return j;
int max(int i, int j) {
if (i > j) {
return j; }
else {
int max(int i, int j) {
if (i > j) {
return i; }
else {
return j++;
int max(int i, int j) {
if (i > j) {
return i; }
else {
int max(int i, int j) {
if (i > i) {
return i; }
else {
return j;
int max(int i, int j) {
if (i) {
return i; }
else {
return j;
int max(int i, int j) {
if (i > j) {
return --i; }
else {
return j;
FSE 2016
[Kurtz, et al., Mutation 2015]

36. Mutation testing in 2020?
Use machine learning to generate optimized mutants based on program features
Use static analysis to determine partial subsumption & tests
Execute tests to refine subsumption and kill mutants
int max(int i, int j) {
if (i > j) {
return i; }
else {
return j;
FSE 2016
[Kurtz, et al., Mutation 2015]

37. Mutation testing in 2020?
Use machine learning to generate optimized mutants based on program features
Use static analysis to determine partial subsumption & tests
Execute tests to refine subsumption and kill mutants
Remove subsumed mutants and redundant tests 
int max(int i, int j) {
if (i > j) {
return i; }
else {
return j;
FSE 2016
[Kurtz, et al., Mutation 2014]

38. Mutation testing in 2020?
Use machine learning to generate optimized mutants based on program features
Use static analysis to determine partial subsumption & tests
Execute tests to refine subsumption and kill more mutants
Remove subsumed mutants and redundant tests
Output a set of tests and a FEW probable-high-value mutants for the engineer to kill
int max(int i, int j) {
if (i > j) {
return i; }
else {
return j;
int max(int i, int j) {
if (i > j) {
return i; }
else {
return --i;
return j;
FSE 2016

39. Conclusions
Mutation score is an imprecise metric inflated by redundant mutants
Researchers should use dominator score instead
Current mutant selection techniques are not optimal
There are no mutation operator combinations that are effective for a range of programs
To optimize dominator score per unit of work, we need to customize mutants to the program under test!
FSE 2016

40. Backup Slides
FSE 2016

41. Future work
Develop a mutant selection approach that is customized for a program
Can’t tell if a mutation is good or bad without program context
Use machine learning
Correlate program features and mutation operators with mutant “goodness”
Select more dominator and near-dominator mutants
Select fewer highly- subsumed and equivalent mutants 
FSE 2016

42. Mutant Subsumption Graphs
Given the following score function: m1
m3,m6 m5 m2 t1 t2 t3 t4 m1  m2 m3 m4 m5 m6 m7 m8 m9
m10
m7,m9 m4
m8,m10
When we construct the mutant subsumption graph from the score function, we see three root nodes that are not subsumed by any other mutants. One mutant from each of these nodes forms a dominator set:
{ m1, m3, m5 }, { m1, m6, m5 }
All other mutants are redundant!
FSE 2016

43. Dominator mutation score
Assume we execute test t1 m1
m3,m6 m5 m2 t1 t2 t3 t4 m1  m2 m3 m4 m5 m6 m7 m8 m9
m10
m7,m9 m4
m8,m10
Killed mutants are shown in gray
Mutation score:
7 of 9 killable mutants = 0.78
Dominator score:
1 of 3 mutants in a dominator set = 0.33
FSE 2016

44. Redundancy and equivalency
We want to investigate how the accuracy changes as the number of redundant and equivalent mutants change, we need a way to measure redundancy and equivalency, preferably in a decoupled manner
FSE 2016

45. Work and Normalized Work
How much effort is required?
We use a simple definition: the number of mutants that a tester must examine
To effectively compare work between different programs with different numbers of mutants, we define equivalent work:
FSE 2016

46. Redundancy and Work
FSE 2016

47. Equivalency and Work
FSE 2016

48. Average / worst-case correlation
Spearman’s rank correlation ρ=0.966
FSE 2016

49. What’s an optimal solution?
We score non-optimal points using the Hausdorff distance (dH), the distance from the nearest optimal point
Hausdorff Distance dh
FSE 2016

50. Today’s common techniques
Mutant Selection Technique
Hausdorff Distance
(smaller is better)
SSDL
0.104
E-Selective
0.134
5% Random
0.150
10% Random
0.274
15% Random
0.147
20% Random
0.229
25% Random
0.111
FSE 2016

51. Incrementally better Operator set “A” Operator set “B”
9 mutation operators
Same dominator score as E- Selective
Only 42% of the work
Operator set “B”
8 mutation operators
Same work as E-Selective
29% higher dominator score
Operator set “C”
14 mutation operators
A knee in the curve
None are as good as the best solutions for a single program!
FSE 2016

52. Determining local context
FSE 2016

53. Threats to validity
The Siemens suite is a small number of small programs
Are they really representative of real-world programs?
Existence of unkilled (but killable) mutants injects errors into determining dominator scores
Specific operator sets identified as optimal may not be optimal, but broader points are not affected
FSE 2016