1 Software Testing and Quality Assurance Lecture 9 - Software Testing Techniques

2 Lecture Outline Mutation Analysis Theory of Mutation analysis and concept of mutant. Mutation Operator Equivalent mutants Comparing coverage criteria.

3 Example Void squeeze (char s[],int c) { int i,j; for (i = j = 0; s[i] != ‘\0’; i++) if (s[i] != c}{ s[ j++ ] = s[i]; } s[i] = ‘\0’; }

4 Example Void squeeze (char s[],int c) { int i,j; for (i = j = 0; s[i] != ‘\0’; i++) if (s[i] != c}{ s[ j ] = s[i]; } s[i] = ‘\0’; } Now, function places all references to ‘c’ at position 0 in the array. (FAULT)

5 Test Cases for the Example Test cases (Test suite) is executed on the faulty version; two possibilities An error is encountered, and the programmer can debug the program to find the fault and repair it. An error is not encountered, and all the test cases pass.

6 Coverage Based Testing Techniques To date, we have discussed methods that aim to find faults by achieving coverage. Little idea as to how good the resulting test suite is.

7 Mutation Analysis Method for measuring the effectiveness of test suites; and As a result, we produce new test cases to be added to that test suite. A mutant of a program is a copy of the program, but with one slight syntactic change.

8 Fault is a mutant. In this case, the slight syntactic change is the removal of ++ operator in the array index. Mutant Example Void squeeze (char s[],int c) { int i,j; for (i = j = 0; s[i] != ‘\0’; i++) if (s[i] != c}{ s[ j ] = s[i]; } s[i] = ‘\0’; }

9 Mutation Analysis Given a program, A test input for that program, and a mutant, We say that test case kills the mutant if and only if The output of the program and the mutant differ for the test inputs. A mutant that has been killed is said to be dead. A mutant that has not been killed is said to be alive.

10 Coupling Effect - Theory behind Mutation Analysis A test case that distinguishes a small fault in a program by identifying unexpected behavior is so sensitive that it will distinguish more complex faults. More complex faults are coupled with simple faults. This is our general observation as well. Many of the faults produced by test suites are a result of one minor fault.

11 Coupling Effect, Mutant & Testing Any test suite that does not uncover the mutant is inadequate. For example,we test the original squeeze function with a test suite,and all test cases pass. Now, we deliberately insert the fault (to evaluate the quality of test suites). If fault is uncovered, mutant is killed. Otherwise, test suites are inadequate. Mutation Analysis Process

12 Mutation Analysis Used to guide test input generation. For example, Find a test case that kills every mutant. It is a way of assessing test suite quality. Test suites that kill more mutants is of higher quality.

13 Mutation Analysis via Mutant Operator Mutant operator is a transformation rule that, given a program, generates a mutant for that program. For example, For ‘if (x < y)’ statement, possible mutants are If (x ≤ y) If (x > y) If (x ≥ y) If (x = y) If (x ≠ y) …..

14 Mutation Analysis via Mutant Operator Mutants are syntactic changes to a program, so mutant operators are depend on the syntax of programming languages. The mutant operators of one programming language may not necessarily apply to other languages.

15 Equivalent Mutants Given a program and a mutant of that program, The mutant is said to be equivalent mutant if, for every input, the program and the mutant produce the same output. A major problem with mutation analysis

16 Equivalent Mutants - example Squeeze function S [ j ++] = s [ i ]; S [ j ++] = s [abs[ i ]]; Absolute value function Variable i only ever takes on value between 0 and the size of the array. Hence, abs[i] is irrelevant. This mutant is equivalent can not be killed. In industry, testers aim to kill 95% of the mutants.

17 Comparing Coverage Criteria Which criteria gives the best coverage? Frankl et al. [ An applicable Family of Data-Flow Testing Criteria, IEEE Transactions on Software Engineering, 14(10), 1988 ] compared paths that each criteria selects.

18 Comparing Coverage Criteria All-Paths All-Du-Paths Multiple condition All-C-Use/Some-P-Use All Uses All-P-Uses/Some C-Uses All-Defs All-P-Uses All-Edges All-Nodes Decision/condition Condition

19 Comparing Coverage Criteria If (y==1) { x = 2; } If (z==1) { a = f(x); } Multiple condition coverage enforces that y == 1 and z == 1 are both executed for the true and false. To achieve any data-flow criteria, at least One test must execute a definition path From the statement x = 2 to a = f(x).

20 Key points Mutation analysis is a method of measuring effectiveness of test suites. And as a result, we produce test cases to improve program testing. The set of paths that satisfy ‘data-flow’ and ‘coverage based testing’ are not necessarily unique.