Unit Testing CSSE 376, Software Quality Assurance Rose-Hulman Institute of Technology March 27, 2007

2 Outline Role of unit testing Testing strategies Black box methods White box methods

3 Role of Unit Testing Assure minimum quality of units before integration into system Focus attention on relatively small units Marks end of development step

4 Testing versus Debugging TestingDebugging plannedunplanned results are recorded unrecorded may be done by non-developers always done by developers look for errorslook for causes of errors

5 Limits of Testing Testing can never demonstrate the absence of errors Exhaustive testing is infeasible Testing may be done imperfectly

6 Strategies for Unit Testing Black box use only specification of program test implementation against its specification White box use structure or other properties of a program to generate tests

7 Black Box Methods Equivalence partitioning Boundary value analysis

8 White Box Methods

9 Coverage statement branch path Cyclomatic Dataflow Mutation Error Based

10 Coverage Methods Statement execute each statement Branch execute each branch Path execute each path

11 Statement Coverage Execute each statement in the program Considered minimum criterion for most unit testing May be difficult to achieve for error cases

12 Example Program 1: if (a < 0) { 2: return 0 } 3: r = 0; 4: c = a; 5: while (c > 0) { 6: r = r + b; 7: c = c - 1; } 8: return r;

13 Statement Tests a = 3, b = 4 executes 1, 3, 4, 5, 6, 7, 5, 6, 7, 5, 6, 7, 5, 8 a = -3, b = 2 executes 1, 2

14 Branch Coverage Execute each branch of the program at least once Differs from statement coverage only for "if" statements without "else"s and case statements without default cases.

15 Dataflow Testing More than branch coverage, but less than path coverage Uses information about references to variables to select paths

16 Definitions and Uses Defining node input statement lhs of assignment Usage node output statement rhs of assignment control statement

17 Suspicious Paths Variable is defined (set to a new value) but never referenced Variable is referenced but never defined Variable is defined twice before it is used

18 DU Paths Definition-use (du) path wrt V: a path containing a defining node for V at the beginning a usage node for V at the end, and no def's or use's in between DU testing: execute each du-path of each variable

19 Example Program 1: if (a < 0) { 2: return 0 } 3: r = 0; 4: c = a; 5: while (c > 0) { 6: r = r + b; 7: c = c - 1; } 8: return r;

20 Example DU Paths (corrected 3/28/07) Def (c) = {4, 7} Use (c) = {5, 7} Def (r) = {3, 6} Use (r) = {6, 8} du-paths for c: 4 - 5, 7 – 5 du-paths for r: , , ,

21 Test Cases for DU Paths (corrected 3/28/07) a = Covers du-paths: 4 - 5, 7 - 5, ,

22 Cartoon of the Day

23 Mutation Testing (As Applied to White-Box) Path testing exercises the control of a program, not the computations along the paths Most programs under test are "almost" right

24 Mutation Method 1.Pre-process program to generate mutants 2.Execute all the mutants and the original program 3.If a mutant differs from the original, then it is "killed" 4.If all mutants are killed, then test set is adequate.

25 Mutation Metaphor

26 Example Program function square( x: integer): integer; begin square := x * x end

27 Example Mutant function square( x: integer): integer; begin square := x + x end

28 Executing Mutants Test set {2} does not kill the mutant Test set {2, 4} does kill the mutant, and it reveals the flaw in the program

29 Which Mutants? Examples: Off by one errors (i+1, i, i-1) Different variable (i, j, k)

30 Assumptions of Mutation Testing Competent Programmer: The perfect program is very close to the program under test Oracle Hypothesis: You know the output for any input Continuity: Every complicated mistake can be caught by looking for simple mistakes.

31 Problems with Mutation Very expensive each test runs the whole program many mutants for every program Some mutants fail to halt May be difficult to detect when a mutant is really just an equivalent program Not reliable: may not try the right mutants