1. Dr. Rob Hasker Dr. Brad Dennis

2. Coverage  Exercise: Each participant: write down 4 instructions Input to procedure: value given by someone, which person (1-4) gave you the value (source) Instruction set: ○ Add one to value ○ Subtract one from value ○ If expression then instructions else instructions where expression is a Boolean expr over value, source ○ Give value to destination – each path must end this way

3. Coverage  See se3800/samples/triangle.htmlse3800/samples/triangle.html Coverage from triangle 3 2 1 Working with partner… ○ Write tests which pass ○ Identify changes to code which pass those tests but fail to meet specification  What is statement coverage?  How does this differ from branch coverage?

4. Coverage  Decision: Boolean expression  Condition: (atomic) component of a Boolean expression  Decision coverage: every Boolean expression evaluates to both true and false How different from statement coverage?  Condition coverage: every condition evaluates to both true and false Too easy: consider cases for a && b  Multiple condition coverage: all combinations executed How many tests needed for a && b && c ?

5. MCDC  Modified Condition Decision Coverage Every statement invoked at least once Every entry, exit point invoked at least once Every control statement taken all possible outcomes at least once Every (non-constant) Boolean expr evaluated to both true and false Every (non-constant) condition evaluated to both true and false Every non-constant condition in a Boolean expression shown to independently affect the outcome

6. MCDC  Modified Condition Decision Coverage: Every entry, exit point executed at least once Every decision has taken all possible outcomes at least once Every condition has taken all possible outcomes at least once Every condition (in a decision) has been shown to independently affect that decision’s outcome.  With this definition, number of tests is O(N)  See A Practical Tutorial on Modified Condition/Decision Coverage, NASA/TM- 2001-210876 for a tutorialA Practical Tutorial on Modified Condition/Decision Coverage, NASA/TM- 2001-210876

7. Path Testing  Statement, decision, MCDC: all about executing logic  Ultimate goal: execute every path  Enumerate paths for GCD example:GCD example printf("The gcd of %d and %d", x, y); while ( x != y ) { if ( x > y ) x = x - y; else y = y - x; } printf( " is %d.\n", x ); printf("gcd of %d and %d", x, y); while ( x != y ) { if ( x > y ) x = x - y; else y = y - x; } printf( " is %d.\n", x );

8. Mutation Testing  Change code, run tests Some test should fail  How to generate mutants? That is: what operators?  Offutt, Rothermel, Lee, Untch, and Zapf. Sufficient Mutant Operators, TOSEM, April 1996

9. Too many mutants!  Number of mutants from 22 operations:  Are the most expensive mutants necessary?  Examined impact of removing each

10. Result: minimum operator set  ABS: insert calls to an absolute value function  AOR: replace all arithmetic ops by every syntactically legal alternatives  LCR: replaces AND, OR by all logical connectors  ROR: replace (modify) relational operators  UOI: insert unary operators

11. Evaluation  Five operators, responsible for ~17% of all mutants, sufficient for 10 Fortan test programs ○ In general: get O(Lines + References) mutants ○ Constant for O is large! ○ Above 10 programs (200 lines): 231,972 mutants  Practical for critical routines

12. Acceptance Tests & APIs  How to write acceptance tests for an API? Our model: acceptance test = story/scenario Issue: an API is not a user!  Solution: Cohn: Writing User Stories for Back-end SystemsCohn: Writing User Stories for Back-end Systems Personify subsystems Epic: “As a bank, I want to receive a file showing all checks to be cleared so that I can debit and credit the right accounts.” “As a bank, I want to receive a 5300 file with correctly formatted single-line deposit entry records so that I can process them.” Write test to the resulting story.

13. What next?  How to know when we’re done? Exercise: research network  Methods & Tools: Methods & Tools No risk, no need to test! Done depends on identified risks: minimal criterion is that you are done testing when risks are reduced to an acceptable level  Is there an acceptable level of risk for safety-critical software?