Coverage – “Systematic” Testing Chapter 20

Dividing the input space for failure search Testing requires selecting inputs to try on the program, but how we select those inputs can be important. “Exhaustive” testing uses each possible input or input combination as a test case. However, the input ranges and permutations can be prohibitive. Using subdomains can prevent this: select groups of inputs that might be considered “equivalent”, such as one test “too big” one “too small” and one “just right” (then some others)

Functional (specification-based, blackbox) testing Testing at the system level, using the specification, can be considered “black box” testing. Here, inputs are selected based only on what the specification “defines” The test would have to “guess” what values might provoke a defect. However selecting inputs based on a knowledge of “what goes on inside” (called gray-box or clear- box testing), can allow us to select interesting test inputs that might aggravate defects.

Structural (program-based) testing In addition, if the code is “open” for examination by the tester, the tester can also select values based on the way the program is put together. Some of the more interesting test cases would come from examining control flow within the program ( if, then, else, switch ) or computations, such as those that may cause arithmetic exceptions, such as overflow or divide by zero errors.

Test oracles Another automated test variation is the idea of “test oracles”. Test oracles check the output of the program against the specification. This can happen in the absence of any automation to generate complete test cases. In the case of oracles, this permits the test writer to write the test, the let the oracle determine if the outputs are valid or not.

Marick's Recommendations Brian Marick recommends the following approach: 1.Generate functional tests from requirements and design to try every function. 2.Check the structural coverage after the functional tests are adequate. 3.Where the structural coverage is imperfect, generate functional tests (not structural) that induce the additional coverage. This works because form (structure) should follow function!

Variations on Coverage Testing There are different “types” of coverage testing that cover different aspects of the program. –Control-flow testing –Path coverage –Branch Coverage The difference are subtle, but important.

Control-flow graphs and their coverage Control flow graphs reflect the number of decision points in a program that can transfer or re-direct the program flow. In control-flow coverage, the possible flow of the program through the path are checked to make sure that each statement is executed by some test (node coverage).

Path coverage In the case of path coverage, the control- flow graph is checked against the test cases to ensure that every possible path is taken. Path coverage requires that each possible path and it’s variations are checked. The notion here is that different paths can (and usually does) result in different data transforms.

Branch coverage Branch coverage requires that all branches of a program to be taken, which results in statement coverage. Unlike statement coverage, however, all branches must be covered. Think about a switch statement that has some “cases” that do not terminate in “ break ” statements. Instead, the control “drops through” to the next case statement. Statement coverage does not require each “case” to be tested, but branch coverage does!

Structural-coverage measuring tools Tools to check structural coverage typically “instrument” the code. That is, the tool ‘adds’ code to the program to leave a trail of ‘cookie crumbs’ about where the program has executed. Tools to instrument the code are readily available, such as gcov, which works with the GNU C compiler. However, incrementing code causes problems in time critical systems by increasing execution time.

Dataflow coverage A concept similar to control-flow coverage is dataflow coverage (except that it focuses on the use of data, not control.) We need to follow paths in the program from the point at which a variable is defined or set to the point in which it is used. (“def-use” pairs.) Think of the “define/set” and “use” relations as nodes, then consider the similarities to control- flow graph testing.

Mutation coverage Mutation coverage is a way to “test the tests.” By introducing mutations in the code, we determine if the test detect them. “Weak mutation” coverage occurs if changes in data state are induced by a test and a mutation. “Strong mutation” coverage occurs if a test causes a mutation to manifest itself not only as a change in data state, but a change in an output value as well!

The “subsumes” relationship When looking at testing strategy, it is important to consider which types of testing might “force” other types. For example, branch coverage attains statement coverage as well. In the case where a test strategy subsumes another, the probability of fault detection increases.

Choosing a testing method in practice Practical restraints very often influence the choice of methods: –Resources available: How much time is available, and how many people? –Feasibility: Can it be done in a reasonable amount of time? (Such as full branch coverage testing) –Process: Is the testing process mature enough to support the method?