1. Software Testing Part II March, 2001

2. Fault-based Testing Methodology (white-box) 2 Mutation Testing

3. 3 Klaidos įvedimas l Automatinis programos testavimas l Klaidos įvedimas l Automatinio testavimo pakartojimas

4. 4 Strong mutation testing l Called “fault-based” because it’s directed at the possible faults that can occur in a program l Provides a measure of test-data adequacy l The technique begins by generating test data by some means; it does not really matter how it is generated (can be randomly generated)

5. 5 Strong mutation testing (cont) l Approach: generate variations of the original program, called “mutants” l The original test data is run through both the original program and the mutant l If there is a difference in the output, then the mutant is dead

6. 6 Strong mutation testing (cont) l If the mutant is not killed, then the test data is not adequate to reveal the fault and distinguish the mutant from the original l If this is the case, then the test data must be augmented to reveal the fault and kill the live mutant l The obvious problem: too many simple faults that can be introduced, O(n^2) for an n line program

7. 7 List of possible faults, Budd (1983) l constant replacement l scalar replacement l scalar variable for constant replacement l constant for variable l array reference for constant replacement l array reference for scalar variable l constant for array reference replacement l scalar variable for array reference replace l array reference for array reference replace l constant replacement l data statement alteration

8. 8 List of possible faults, Budd (1983) l comparable array name replacement l arithmetic operator replacement l relational operator replacement l logical connector replacement l absolute value insert l unary operator insertion l statement analysis l statement deletion l return statement replace l goto label replace l do statement end replace

9. 9 How effective is mutation testing? l “Test data that distinguishes all programs differing from a correct one by only simple errors is so sensitive that it also finds complex errors”, DeMillo (1978) l mutation testing makes the “competent programmer” assumption: “programmers write programs that are nearly correct”!

10. 10 Example program: l string searching program: the program prompts the user for a positive integer in the range 1 to 20 and then for a string of characters of that length. The program then prompts for a character and returns the position in the string at which the character was first found or a message indicating that the character was not present in the string. The user has the option to search for more characters.

11. 11 main() { (1 ) char a[20], ch, response = ‘y’; (2) int x, i; (3) bool found; (4) cout << “Input an integer between 1 and 20: “; (5) cin >> x; (6) while ( x 20) { (7) cout << “Input an integer between 1 and 20: “; (8) cin >> x; } (9) cout << “input “ << x << “ characters: “; (10) for (int i = 0; i > a[i]; (11) cout << endl;

12. 12 (12) while ( response == ‘y’ || response == ‘Y’) { (13) cout << “input character to search: “; (14) cin >> ch; (15) found = false; (16) i = 0; (17) while (!found && i < x) (18) if (a[i++] == ch) found = true; (19) if (found) cout << ch << “ at: “ << i << endl; (20) else cout << ch << “ not in string” << endl; (21) cout << “search for another character? [y/n]”; (22) cin >> response; }

13. 13 mutation testing string example l Assume we have branch tested the program l consider lines 15 to 18 (15) found = false; (16) i = 0; (17) while (!found && i < x) (18) if (a[i++] == ch) found = true;

14. 14 Test data for branch testing

15. 15 RIP l Since the output of the mutant program differs from the output of the original program, mutant #1 is dead! l It is important to make only one small change so that faults don’t cancel each other

16. 16 mutation testing string example l Assume we have branch tested the program, l create another mutant: (15) found = false; (16) i = 0; (17) while (!found && i < x) (18) if (a[i++] == ch) found = true;

17. 17 Mutant #2 lives! l The output of the mutant is the same as the original program ==> mutant #2 is alive l Analysis of the program reveals that the test data is not adequate to reveal the fault; add a test case to search first position

18. 18 RIP

19. 19 Super mutants! l Sometimes it’s impossible to kill a mutant because no test data can be found that will reveal the newly introduced fault l an absurd example : main() { int x = 3; int y = x*3; if (y < 10) cout << “less”; }

20. 20 Suppose the string processing program had contained a fault! (15) found = false; (16) i = 1; (17) while (!found && i < x) (18) if (a[i++] == ch) found = true;

21. 21 Strengths of strong mutation testing l Shows the absence of particular faults; contrary to Dijkstra’s comment! By introducing a fault into the program and then showing that the test data reveals it, we know that the fault cannot be there! l Forces the programmer to scrutinize ( kruopščiai ištirti) and analyze the program under test to think of test data to expose the newly introduced fault, i.e., kill the mutant!

22. 22 Weaknesses of strong mutation testing l Computationally expensive! l Huge number of mutants can be generated l There are systems that can help generate mutant programs and test data; but much work must be done by hand.

23. 23 Weak mutation testing l Less demanding that strong mutation l take some construct of the program and create some mutants of this construct l consider: if (A && B || C), possible mutants: n if (A || B || C) n if (A && B && C) n if ((A && B) || C) n etc.

24. 24 Weak mutation testing (cont) l Howden (1981) identified five candidate constructs: n data access (or variable reference) n data storage (or variable assignment) n arithmetic expression n arithmetic relation n boolean expression: same as multiple condition coverage l Howden developed reliable tests for the five constructs

25. 25 Consider data access or variable reference l We’re considering the r-value here. l Aim is to ensure that the correct variable is being accessed. l consider the variables in string processing program, type-by-type n integer variables are referenced on lines 9, 10, 17, 19 and 20; make sure we have adequate test data to ensure that they are getting correct values n bool variable is referenced on line 15

26. 26 Consider arithmetic relations: l they occur on lines 6 and 17; test data that ensures that x has values 0, 1, 2, 1, 20 and 21 would test x and i

27. 27 Strengths of weak mutation l Like strong mutation testing, it promotes a thorough analysis of the program under test l computationally cheaper l not necessary to physically generate the mutants

28. 28 weaknesses of weak mutation l Unlike strong mutation, it is not reliable for the program as a whole; adequacy of the data is local to the construct being mutated l Girgis and Woodward (1986) found that weak mutation was outperformed by data flow testing; they also found that different faults were found by each technique and promoted a combination of testing techniques