1. MUTACINIS TESTAVIMAS Benediktas Knispelis, IFM-2/2 Mutation testing

2. Chapters About mutation testing Weak mutation testing Strong mutation testing Various mutation operators Tools for mutation testing Mutacinis testavimas 2 / 19

3. The aim of mutation testing Locate and expose weaknesses in test suites Mutacinis testavimas 3 / 19

4. Definition Mutacinis testavimas 4 / 19

5. About Mutation testing is also a type of white-box testing methods It involves modifying source code in small ways Mainly used for unit test cases Mutacinis testavimas 5 / 19

6. Mutation testing overview Mutation testing is done by selecting a set of mutation operators and then applying them to the source program one at a time for each applicable piece of the source code. The result of applying one mutation operator to the program is called a mutant. Mutacinis testavimas 6 / 19

7. Mutation testing overview (2) If the test suite is able to detect the change in the mutant code, then the mutant is said to be killed. When this happens, the mutant is considered dead and no longer needs to remain in the testing process since the faults represented by that mutant have been detected, and more importantly, it has satisfied its requirement of identifying a useful test case. Mutacinis testavimas 7 / 19

8. Mutation testing overview (3) If a mutant gives the same outcome as the original with the test data, then it is said to be live. The quality of a set of test cases is measured by the percentage of mutants killed by the test data (mutation score) MS = 100 * MK / (MA – ME) MK – quantity of mutants killed MA – overall quantity of mutants ME – quantity of equivalent mutants Mutacinis testavimas 8 / 19

9. Example The original code: if(a && b)c = 1; else c = 0; We replace '&&' with '||' and produce the following mutant: if(a || b)c = 1; else c = 0; Now, for the test to kill this mutant, the following condition should be met: 1) Test input data should cause different program states for the mutant and the original program. For example, a test with a=1 and b=0 would do this (enough for Weak mutation testing). 2) The value of 'c' should be propagated to the program's output and checked by the test (necessary for Strong mutation testing). Mutacinis testavimas 9 / 19

10. Weak mutation testing Weak mutation is an approximation technique that compares the internal states of the mutant and original program immediately after execution of the mutated portion of the program. It requires much less computing power to ensure that the test suite satisfies weak mutation testing. Mutacinis testavimas 10 / 19

11. Strong mutation testing When strong mutation testing is executed, the values produced by mutated code are passing further and change (or do not change) the output of the module or subroutine. If the mutant is not killed, then considerations are made whether it is possible to enhance test data to kill this live mutant or whether no test data could ever be constructed to kill the mutant. Strong mutation testing requires much more computing power. Mutacinis testavimas 11 / 19

12. Equivalent mutants The original code: int index=0; while (…){ index++; if (index == 10) break; } Replace "==" with ">=" and produce the following mutant: int index=0; while (…){ index++; if (index >= 10) break; } However, it is not possible to find a test case which could kill this mutant. Mutacinis testavimas 12 / 19

13. Traditional mutation operators 1. Statement deletion 2. Boolean expression changes (true and false) 3. Arithmetic operation changes (+ and *, - and /) 4. Boolean operator changes (> and >=, == and <=) 5. Changes of variables declared in the same scope (variable types should be the same) Mutacinis testavimas 13 / 19

14. Class-level mutation operators 1. Polymorphic 2. Method Overloading 3. Method Overriding/Hiding in Inheritance 4. Field Variable Hiding in Inheritance 5. Information Hiding (Access Control) 6. Static/Dynamic States of Objects 7. Exception Handling Mutacinis testavimas 14 / 19

15. Mutation testing tools NinjaTurtles (for mutation testing in.NET) Mutagenesis (a PHP 5.3+ mutation testing framework) Heckle (Ruby mutation tester) Jester (the JUnit test tester / JAVA) Mutandis (JavaScript) Mutacinis testavimas 15 / 19

16. NinjaTurtles Open-source Integrates with your existing unit test framework Mutation testing variants applied at IL level: Sequence point deletion Arithmetic operator substitution (*, /, +, - and %) Bitwise operator substitution (&, | and ^) Branch substitution (condition, always and never branch) Conditional boundary substitution ( and >=) Substitution of reads from variables, parameters and fields of the same type Substitution of writes to variables of the same type Mutacinis testavimas 16 / 19

17. NinjaTurtles Running mutation tests via console: NinjaTurtles.Console run -c ClassToTest TestAssembly.dll Supplementing existing test suites with mutation tests (C#): [Test, MutationTest] public void MethodUnderTest_Mutation_Tests() { MutationTestBuilder.For("MethodUnderTest").Run(); } Mutacinis testavimas 17 / 19

18. Questions 1. Aim of the mutation testing? 2. Mutation testing types? 3. Differences between strong and weak mutation testing? 4. What kind of traditional mutation operators do you know? 5. Is it possible to automate mutation testing? Mutacinis testavimas 18 / 19

19. Thank you Mutacinis testavimas 19 / 19