1. Mutation Testing
The Mutants are Coming!
Copyright © 2017 – Curt Hill

2. Introduction
There are several things that we consider worth saving in version control
Code
Documentation
One of these are the tests that we created for the project
Mutation testing is testing the tests
Copyright © 2017 – Curt Hill

3. Mutations
What we want to do is create a suite of variants of the programs in question
This is done after every test has been passed
Take a piece of code
Modify the code in a small way
Bebugging
Now run the tests and see if the tests finds that there are defects
Do this many times with many different small modifications
Copyright © 2017 – Curt Hill

4. Failures A mutation that goes undetected is always cause for alarm
It is either:
Inside dead code – which is its own problem
An indication that the tests are inadequate
Something else is correcting the error before it manifests – highly unlikely
After a failure we must write new tests or enhance old ones
Copyright © 2017 – Curt Hill

5. Small Modifications
A small modification must not provoke a syntax error
Typically only one error
So that one error does not mask another
There are numerous classes of errors that could be considered
These are on next slide
Copyright © 2017 – Curt Hill

6. Classes of Errors Changing an operator
Deleting a statement or portion of it
Changing a variable in an expression to one with similar scope and type
Duplicating a statement
Replacing a sub-expression with a constant
Changing the type of a variable
Changing a condition
Others are possible as well
Copyright © 2017 – Curt Hill

7. Suites
One small error in a program or class hardly constitutes a test of the tests
It is better to generate many such tests
Perhaps several different ones for each routine
We run all of these tests and keep score of how many errors are found
The statistics from this are a measure of the goodness of our testing
Copyright © 2017 – Curt Hill

8. Theory There are two underlying theories behind mutation testing
The competent programmer
A programmer is more likely to make a simple typo
The coupling effect
Simple defects cascade into a serious fault
Copyright © 2017 – Curt Hill

9. Kill the Mutant!
Copyright © 2017 – Curt Hill

10. Requirements In order to kill a mutant three things must happen
The test data must execute the mutated statement
The test data should distinguish between good and bad
The incorrect data must change something that is displayed or tested
Copyright © 2017 – Curt Hill

11. Example Original: if(a && b) c = 1; else c = 0; Mutant: if(a || b)
The test must execute the statements
The test must be able to distinguish between the two: a = 1, b = 1 will not a = 1, b = 0 will
The variable c must be have an effect on something displayed or tested
Copyright © 2017 – Curt Hill

12. Automation
Mutation testing is only effective if the testing is automated
Unit tests are typically automated so this is the most common way mutation testing occurs
Mutation testing is only effective if many mutants are created
This typically also involves a program that generates many one error mutants
Starts with “correct” code and inserts one randomly selected fault per mutant
Copyright © 2017 – Curt Hill

13. Nice Picture Copyright © 2017 – Curt Hill
See:
Copyright © 2017 – Curt Hill

14. Automation What must a mutation testing scheme involve?
Many iterations of:
Randomly mutate the code
Modify the make to run the mutant instead of original
Make the unit test
Run the unit test
Count the results
Copyright © 2017 – Curt Hill

15. History The term bebugging refers to injecting errors into programs
The Psychology of Computer Programming by Gerald Weinberg in 1970
First used to test programmer’s ability to find bugs
Derived from a technique used on radar operators
Proposed by Richard Lipton as a testing scheme in 1971
An implementation by Timothy Budd as a part of his PhD in 1980
Copyright © 2017 – Curt Hill

16. Conclusion Mutation testing is testing the tests
A white box testing of the white box testing
Does not expose requirements problems or omissions as black box testing should
It should expose unit testing omissions
It needs an automated approach like unit testing to be practical
Copyright © 2017 – Curt Hill

17. Exercise Time to make mutants
Extract from the unittest assignment the CurtDate.cpp
You will now produce mutants by hand
Each mutant will have the file name:
CurtDate.NX
Where N is the initial of your first name
Where X is a sequence number starting with 1
Copyright © 2017 – Curt Hill

18. The Code Most of the routines have no flow of control
These should have one mutant per method
Those that have flow of control should have one mutant for the Cyclomatic Complexity Number
Each student will get one of the following:
Several small routines without flow
One or two with flow
Correct will be split by two
Copyright © 2017 – Curt Hill

19. Rules Each mutant will have one and only one modification
Mutant must compile
The modification will be one of these:
Changing an operator (arithmetic or conditional)
Deleting a statement or portion of it
Changing a variable in an expression to one with similar scope and type
Zip the mutants and
Copyright © 2017 – Curt Hill