White-box Testing Black-box Testing Extensions Unit Testing Methods White-box Testing Black-box Testing Extensions

Software Testing Methods Testing Goals Reveal failures/faults/errors Locate failures/faults/errors Show system correctness Within the limits of optimistic inaccuracy Improve confidence that the system performs as specified (verification) Improve confidence that the system performs as desired (validation) CONTENT BLOCKED Due to copyright infringement. Program testing can be used to show the presence of bugs, but never to show their absence [Dijkstra]

Unit Testing Techniques Unit Testing checks that an individual program unit (subprogram, object class, package, module) behaves correctly. Static Testing - testing a unit without executing the unit code Dynamic Testing - testing a unit by executing a program unit using test data CONTENT BLOCKED Due to copyright infringement.

Black-Box Testing Black box testing Purpose: tests the overall functional behavior Drawback: may not reveal cause of the defect Method: Treat the module as a classic input/output module; when you provide it with certain inputs it should provide certain outputs CONTENT BLOCKED Due to copyright infringement. requirements output input events

Black-box Testing (naïve approach) Getting started: Review documentation for expected output Example: “A Fibonacci sequence starts like this: 0, 1, 1, 2, 3, 5, 8, 13, 21, 34, 55, 89” CONTENT BLOCKED Due to copyright infringement. Unit Test Procedure: Ask the user to provide a number from 0 to 11 inclusive Call the Fib function Compare result value to the corresponding value in the above list You go try it this way

Specification-Based Testing Black-box Testing is more formally known as specification-based testing Specification-based testing Test cases designed, selected, and run based on specifications Use specifications to derive test cases Requirements Design Function signature Based on some kind of input domain The approach is not naïve, but considers the range of data inputs Typical values Boundary values Special cases Invalid input values (negative testing) CONTENT BLOCKED Due to copyright infringement.

Equivalence Partitioning Equivalence partitioning is an approach to black box testing that divides the input domain of a program into classes of data from which test cases can be derived. Math review: What is an equivalence class? CONTENT BLOCKED Due to copyright infringement. Equivalence classes are based on equivalence relations: For all x, y, z in set T, the following hold: Reflexivity: x ~ x Symmetric:x ~ yiffy ~x Transitivity: if x ~ y and y ~ z, then x ~ z Examples: “=“ “has the same astrological sign” Non-examples: “>” “is taller than” How does this help? Define equivalence as “these data are expected to produce the same result” We only need to test with 1 representative data element from that class May need to remind students of the concept of an equivalence relation from discrete math (reflexive, symmetric, transitive)

Equivalence partitioning Example: Suppose a program computes the value of the function . What are the equivalence classes? X < = -2 valid -2 < X < 1 invalid X >= 1 valid Test cases would be selected from each equivalence class. Cons: Lacks causality: 2 inputs may expect to produce the same output for entirely different reasons No deterministic way to define the classes; often difficult for non-mathematical operations or multi-dimensional inputs Pros: Often one can intuitively identify representative data, so there really is no reason not to try this. Can greatly reduce the number of test cases you need to run CONTENT BLOCKED Due to copyright infringement.

Boundary Value Analysis Boundary Value Analysis - black box technique where test cases are designed to test the boundary of an input domain. Studies have shown that more errors occur on the "boundary" of an input domain than in the "center". Commonly referred to as a type of “edge case” Boundary value analysis complements and can be used in conjunction with equivalence partitioning. Example revisited: After using equivalence partitioning to generate equivalence classes, boundary value analysis would dictate that the boundary values of the three ranges be included in the test data. That is, we might choose the following test cases (for a 32 bit system): X <= -2 -231, -100, -2.1, -2 -2 < X < 1 -1.9, -1, 0, 0.9 X >= 1 1, 1.1,100, 231-1 CONTENT BLOCKED Due to copyright infringement. Edge cases are a broader term in that they imply not just data-driven boundary cases but unusual interactions with the unit under test

Example: Possible Boundaries Basis: Array length Empty,1, 2, small, or large number of elements Basis: Position of minimum score Smallest element first, second, in middle, in last Basis: Number of minima Unique minimum, a few minima, all minima Input domain: float[] Basis: array length CONTENT BLOCKED Due to copyright infringement. empty one two small large Input domain: float[] Basis: position of minimum first somewhere in middle last second Input domain: float[] Basis: number of minima 1 minimum All data equal 2 minima

Structural (White-Box) Testing Structural testing Test cases designed, selected, and run based on structure of the source code Scale: tests the nitty-gritty (line-by-line) Drawbacks: need access to the source Use source code to derive test cases Build a graph model of the system Control flow Data flow Choose test cases that guarantee different types of coverage Node coverage Edge coverage Loop coverage Condition coverage Path coverage CONTENT BLOCKED Due to copyright infringement. ... our goal is to ensure that all statements and conditions have been executed at least once ...

Exhaustive Testing CONTENT BLOCKED Due to copyright infringement. loop < 20 X 14 There are 10 possible paths! If we execute one test per millisecond, it would take 3,170 years to test this program!!

Selective Testing Selected path CONTENT BLOCKED Due to copyright infringement. Selected path loop < 20 X

Example 1 float findAverage(float[] scores) { 2 float total = 0, min = MAX_FLOAT, min2 = MAX_FLOAT; 3 for (inti = 0 ; i<scores.length ; i++) { 4 if (scores[i] < min) { 5 min2 = min; 6 min = scores[i]; 7 } 8 else if (scores[i] < min2) 9 min2 = scores[i]; 10 total += scores[i]; 11 12 if (min != MAX_FLOAT && min2 != MAX_FLOAT) 13 output(min, min2); 14 return total / scores.length; 15 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15

Node Coverage Select test cases such that every node in the graph is visited Also called statement coverage Guarantees that every statement in the source code is executed at least once Select minimal number of test cases & statements Test case: {<1,2,3,4,5,6,7,10,11,3,4,8,9,10,11,3,12,13,14,15>} 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15

Edge Coverage Select test cases such that every edge in the graph is visited Also called branch coverage Guarantees that every branch in the source code is executed at least once More thorough than node coverage More likely to reveal logical errors Test cases: {<1,2,3,4,5,6,7,10,11,3,12,13,14,15>, <1,2,3,4,8,9,10,11,3,4,8,10,11,3,12,14,15>} 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15

Other Coverage Criteria Loop coverage Select test cases such that every loop boundary and interior is tested Boundary: 0 iterations Interior: 1 iteration and> 1 iterations This was missing from our tests, and could have uncovered bugs! Watch out for nested loops Less precise than edge coverage Condition coverage Select test cases such that all conditions are tested if (min != MAX_FLOAT && min2 != MAX_FLOAT) More precise than edge coverage Our edge coverage test doesn’t say why 12  14 is traversed Loop coverage would have found cases where length is 1 or 2, then we wouldn’t have found 2 mins Condition coverage would have ensured the logic of the 2nd condition in particular on line 12 wasn’t faulty

Other Coverage Criteria Path coverage Select test cases such that every pathis traversed Loops are a problem Consider example on earlier slide – what is scores.length? Suppose it was 5, how many test cases would you need? 0, 1, average, max iterations Why are paths important? Because previous instructions may cause side effects to runtime state that require you to verify later instructions in light of those side effects Most thorough……but is it feasible? Not really, so we try instead to compute the set of linearly independent paths within the graph Called the basis paths; more to come in Metrics (McCabe) If scores.length == 5, consider how many test cases: - Simple approach: 2 – you just go around the loop 5 times (twice for the post-loop conditional) But, path implies the exact sequence all the way through, so: There are 3 paths through each loop body, so you have to choose one each time through 5 times: 6 * 6 * 6 * 6 * 6 – (3 * 2 for the conditional at the end) == 7776, and that is before you consider edge cases.

Unit Testing: OO Perspective Can apply per method But this is not considered a best practice as methods represent conceptual behaviors and as such may not be used in isolation. Unit test an entire scenario implemented as a collection of objects exchanging messages Think UML sequence diagram Use-case driven unit tests often happen at the component level OO white box testing involves verifying the object traverses a defined set of states In response to stimuli (messages) - reactive systems Verify object is used in the manner designed State machines a common modeling approach for this Polymorphism creates unique challenges Encapsulation is violated Different languages support different semantics

Unit Testing: Process Perspective Unit testing is popular (again) Promoted by XP/Agile methodologies Provides continuous feedback (“code a little, test a little”) “Testing is good so we will do it all the time” Has surpassed inspections in popularity Data still says inspections are better Unit tests struggle to provide white box capability Process Terminology Test class - set of input data that is identified for testing Test case - specific data chosen for testing a test class. Test suite - set of test cases that serve a particular goal Exhaustive testing - test all logical paths in a module

TDD: An Alternative View Test-Driven Development (TDD) Write your tests first! A promoted practice for XP “code a little, test a little” becomes “test a little, code a little” Derives from your user story acceptance tests CONTENT BLOCKED Due to copyright infringement.

Summary: Unit Testing White-box Black-box Unit-level testing process Structural evaluation Coverage difficult – set a target! Black-box Treats implementation of function as “unknown” Test for valid outputs given a range of inputs Science based on domain/range of the inputs Unit-level testing process Unit testing now considered a very agile way of coding. TDD a great way to ensure you verify& validate as you go. Many developers struggle to write complete unit tests. Many developers struggle to maintain their unit tests.