1. CompSci 230 Software Construction
Lecture Slides #24: White Box testing S1 2016

2. Agenda Topics: White box testing
General considerations
Choosing test cases for white box testing
Statement coverage
Branch coverage
Path coverage
Condition coverage
White box testers
Myers’ principles for testing
CompSci 230: WB

3. White Box Testing – general considerations
Design tests based on implementation itself (structure of code)
When we design tests, we have access to the source code and use it to derive our test design
We try to design tests that ensure that the execution always runs correctly, but how do we do that?
Usefulness of testing vs. effort is an issue in white box testing, too
CompSci 230: WB

4. Concept: Control flow graphs (CFG)
Each statement in the IUT is a node (vertex)
If execution can move from node u to node v, we connect u and v via an arc (branch)
public void deleteCurrencyByCode(String code) {
if (currencies == null) return;
for (int i=0; i < currencies.length; i++) {
if (currencies[i].getCode().equals(code)) {
if (currencies.length == 1) {
currencies = null;
return; }
else {
Currency[] tmpCurrencies = new Currency[currencies.length-1];
for (int j=0; j<i; j++) {
tmpCurrencies[j] = currencies[j];
for (int j=i; j<tmpCurrencies.length; j++) {
tmpCurrencies[j] = currencies[j+1];
currencies = tmpCurrencies;
CompSci 230: WB

5. Statement coverage
When a test executes a particular statement (vertex) in the CFG, we say that it covers this statement
A test suite whose tests cover a (sub)set of statements in the IUT is said to have (partial/complete) statement coverage
Advantage: number of statement in IUT is finite
Problem: it may be difficult to design tests that reach every statement (e.g., consider a method that always returns before it reaches the last statements)
But then this is a defect and may be detected that way!
Can you think of a situation in which a test suite with complete statement coverage might not expose a fault in the IUT?
CompSci 230: WB

6. Branch coverage
Another approach is to design a test suite so that for every arc in the CFG, there is at least one test in which the execution follows that branch
For this purpose, we may work with a simplified CFG where we remove nodes with one incoming and one outgoing branch only
Can you think of a case where branch coverage is not possible?
Can you think of a case where a test suite providing branch coverage could fail to expose a fault?
CompSci 230: WB

7. Path coverage Can work from simplified CFG here
Each possible execution sequence of nodes in the CFG is a path
Idea: Test all possible paths in the IUT
Problem: The number of paths may be infinite (e.g., our example, it is)
Usually, this is because of loops in the CFG
Some paths may infeasible under actual execution
E.g., consider two successive if-else statements with the same condition whose variables don’t change. Then the first if-branch followed by the second else-branch is an infeasible path
Challenge: find a test set of paths that is most likely to trigger a failure
CompSci 230: WB

8. Condition coverage Consider a fault in an if-statement:
Statement should be: if (a > 0 AND b > 0) {…
Actual statement: if (a > 0 OR b > 0) {…
Testing with (a=0, b=0) and (a=1,b=1) gives statement, branch, and path coverage for the if-statement
Fault remains undetected
Condition coverage aims to find test cases such that each clause (such as “a > 0” or “b==7”) in an if-statement’s condition will become true and false, in every possible combination.
Problem: Potentially large number of combinations, not every combination is feasible
Compromise: Try and find test cases where the change in truth value of a clause causes the entire condition to change its truth value
E.g., (a > 0 AND (b > 0 OR c < 0)): If b > 0, value of c does not matter, value of a determines the overall value of the condition. Similarly, if c < 0, the value of b does not matter.
CompSci 230: WB

9. White box testing – general considerations
Black box testing more or less forces us to look at the specifications: What is that thing supposed to do?
White box testing puts the focus on deriving test cases from the code structure of the IUT as it was implemented
But what if that structure doesn’t just contain faults, but is in itself faulty?
E.g., missing code, such as an if-statement that needs an else branch
E.g., developer misunderstood specifications
What question does this raise?
Need to go back to the specifications to design good test cases!
CompSci 230: WB

10. The ideal white box tester
Must-have: Knows the programming language that the IUT is written in
Knows and understands the specifications perfectly
Was not involved in the thought processes that led to the implementation (carries no preconception as to how the problem ought to be solved, or is biased by how it was solved)
So ideally, the tester will not be the developer
CompSci 230: WB

11. Myer’s Principles for Testing
A necessary part of a test case is a definition of the expected output or result
A programmer should avoid attempting to test his or her own program
A programming organization should not test its own programs
Thoroughly inspect the results of each test
Test cases must be written for input conditions that are invalid and unexpected, as well as for those that are valid and expected
Examining a program to see if it does not do what it is supposed to do is only half the battle; the other half is seeing whether the program does what it is not supposed to do
Avoid throwaway test cases unless the program is truly a throwaway program
Do not plan a testing effort under the tacit assumption that no errors will be found
The probability of the existence of more errors in a section of a program is proportional to the number of errors already found in that section
Testing is an extremely creative and intellectually challenging task
CompSci 230: WB

12. Example Principle 4: Thoroughly inspect the results of each test
Test case input for the method: an identifier for an album in the database
Expected return: A list of song titles on the album with their durations in minutes and seconds
Get:
Sweet love in the rainy sunshine 3:12
Ginger on my mind 4:42
Oh come on (Remix) 3:65
Platitudes for platypus 3:52
Ginger on my mind (Remix) 4:20
Looks like a pass, but is it?
Any other principles at stake here?
CompSci 230: WB

13. Review Is it harder to be a black box tester or a white box tester?
Can you explain the difference between statement coverage, branch coverage, path coverage and condition coverage in a paragraph or two?
Which of statement coverage, branch coverage, path coverage and condition coverage can we usually achieve (if we don’t mind the effort)?
Which challenges do we face in white box testing?
Name three consequences of following Myers’ principles
Do Myers principles guarantee that testing will result in fault-free software?
CompSci 230: WB