Mutation Testing The Mutants are Coming! Copyright © 2017 – Curt Hill

Introduction There are several things that we consider worth saving in version control Code Documentation One of these are the tests that we created for the project Mutation testing is testing the tests Copyright © 2017 – Curt Hill

Mutations What we want to do is create a suite of variants of the programs in question This is done after every test has been passed Take a piece of code Modify the code in a small way Bebugging Now run the tests and see if the tests finds that there are defects Do this many times with many different small modifications Copyright © 2017 – Curt Hill

Failures A mutation that goes undetected is always cause for alarm It is either: Inside dead code – which is its own problem An indication that the tests are inadequate Something else is correcting the error before it manifests – highly unlikely After a failure we must write new tests or enhance old ones Copyright © 2017 – Curt Hill

Small Modifications A small modification must not provoke a syntax error Typically only one error So that one error does not mask another There are numerous classes of errors that could be considered These are on next slide Copyright © 2017 – Curt Hill

Classes of Errors Changing an operator Deleting a statement or portion of it Changing a variable in an expression to one with similar scope and type Duplicating a statement Replacing a sub-expression with a constant Changing the type of a variable Changing a condition Others are possible as well Copyright © 2017 – Curt Hill

Suites One small error in a program or class hardly constitutes a test of the tests It is better to generate many such tests Perhaps several different ones for each routine We run all of these tests and keep score of how many errors are found The statistics from this are a measure of the goodness of our testing Copyright © 2017 – Curt Hill

Theory There are two underlying theories behind mutation testing The competent programmer A programmer is more likely to make a simple typo The coupling effect Simple defects cascade into a serious fault Copyright © 2017 – Curt Hill

Kill the Mutant!   Copyright © 2017 – Curt Hill

Requirements In order to kill a mutant three things must happen The test data must execute the mutated statement The test data should distinguish between good and bad The incorrect data must change something that is displayed or tested Copyright © 2017 – Curt Hill

Example Original: if(a && b) c = 1; else c = 0; Mutant: if(a || b) The test must execute the statements The test must be able to distinguish between the two: a = 1, b = 1 will not a = 1, b = 0 will The variable c must be have an effect on something displayed or tested Copyright © 2017 – Curt Hill

Automation Mutation testing is only effective if the testing is automated Unit tests are typically automated so this is the most common way mutation testing occurs Mutation testing is only effective if many mutants are created This typically also involves a program that generates many one error mutants Starts with “correct” code and inserts one randomly selected fault per mutant Copyright © 2017 – Curt Hill

Nice Picture Copyright © 2017 – Curt Hill See: https://www.guru99.com/mutation-testing.html Copyright © 2017 – Curt Hill

Automation What must a mutation testing scheme involve? Many iterations of: Randomly mutate the code Modify the make to run the mutant instead of original Make the unit test Run the unit test Count the results Copyright © 2017 – Curt Hill

History The term bebugging refers to injecting errors into programs The Psychology of Computer Programming by Gerald Weinberg in 1970 First used to test programmer’s ability to find bugs Derived from a technique used on radar operators Proposed by Richard Lipton as a testing scheme in 1971 An implementation by Timothy Budd as a part of his PhD in 1980 Copyright © 2017 – Curt Hill

Conclusion Mutation testing is testing the tests A white box testing of the white box testing Does not expose requirements problems or omissions as black box testing should It should expose unit testing omissions It needs an automated approach like unit testing to be practical Copyright © 2017 – Curt Hill

Exercise Time to make mutants Extract from the unittest assignment the CurtDate.cpp You will now produce mutants by hand Each mutant will have the file name: CurtDate.NX Where N is the initial of your first name Where X is a sequence number starting with 1 Copyright © 2017 – Curt Hill

The Code Most of the routines have no flow of control These should have one mutant per method Those that have flow of control should have one mutant for the Cyclomatic Complexity Number Each student will get one of the following: Several small routines without flow One or two with flow Correct will be split by two Copyright © 2017 – Curt Hill

Rules Each mutant will have one and only one modification Mutant must compile The modification will be one of these: Changing an operator (arithmetic or conditional) Deleting a statement or portion of it Changing a variable in an expression to one with similar scope and type Zip the mutants and email Copyright © 2017 – Curt Hill