CompSci 230 Software Construction Lecture Slides #24: White Box testing S1 2016

Agenda Topics: White box testing General considerations Choosing test cases for white box testing Statement coverage Branch coverage Path coverage Condition coverage White box testers Myers’ principles for testing CompSci 230: WB

White Box Testing – general considerations Design tests based on implementation itself (structure of code) When we design tests, we have access to the source code and use it to derive our test design We try to design tests that ensure that the execution always runs correctly, but how do we do that? Usefulness of testing vs. effort is an issue in white box testing, too CompSci 230: WB

Concept: Control flow graphs (CFG) Each statement in the IUT is a node (vertex) If execution can move from node u to node v, we connect u and v via an arc (branch) public void deleteCurrencyByCode(String code) { if (currencies == null) return; for (int i=0; i < currencies.length; i++) { if (currencies[i].getCode().equals(code)) { if (currencies.length == 1) { currencies = null; return; } else { Currency[] tmpCurrencies = new Currency[currencies.length-1]; for (int j=0; j<i; j++) { tmpCurrencies[j] = currencies[j]; for (int j=i; j<tmpCurrencies.length; j++) { tmpCurrencies[j] = currencies[j+1]; currencies = tmpCurrencies; CompSci 230: WB

Statement coverage When a test executes a particular statement (vertex) in the CFG, we say that it covers this statement A test suite whose tests cover a (sub)set of statements in the IUT is said to have (partial/complete) statement coverage Advantage: number of statement in IUT is finite Problem: it may be difficult to design tests that reach every statement (e.g., consider a method that always returns before it reaches the last statements) But then this is a defect and may be detected that way! Can you think of a situation in which a test suite with complete statement coverage might not expose a fault in the IUT? CompSci 230: WB

Branch coverage Another approach is to design a test suite so that for every arc in the CFG, there is at least one test in which the execution follows that branch For this purpose, we may work with a simplified CFG where we remove nodes with one incoming and one outgoing branch only Can you think of a case where branch coverage is not possible? Can you think of a case where a test suite providing branch coverage could fail to expose a fault? CompSci 230: WB

Path coverage Can work from simplified CFG here Each possible execution sequence of nodes in the CFG is a path Idea: Test all possible paths in the IUT Problem: The number of paths may be infinite (e.g., our example, it is) Usually, this is because of loops in the CFG Some paths may infeasible under actual execution E.g., consider two successive if-else statements with the same condition whose variables don’t change. Then the first if-branch followed by the second else-branch is an infeasible path Challenge: find a test set of paths that is most likely to trigger a failure CompSci 230: WB

Condition coverage Consider a fault in an if-statement: Statement should be: if (a > 0 AND b > 0) {… Actual statement: if (a > 0 OR b > 0) {… Testing with (a=0, b=0) and (a=1,b=1) gives statement, branch, and path coverage for the if-statement Fault remains undetected Condition coverage aims to find test cases such that each clause (such as “a > 0” or “b==7”) in an if-statement’s condition will become true and false, in every possible combination. Problem: Potentially large number of combinations, not every combination is feasible Compromise: Try and find test cases where the change in truth value of a clause causes the entire condition to change its truth value E.g., (a > 0 AND (b > 0 OR c < 0)): If b > 0, value of c does not matter, value of a determines the overall value of the condition. Similarly, if c < 0, the value of b does not matter. CompSci 230: WB

White box testing – general considerations Black box testing more or less forces us to look at the specifications: What is that thing supposed to do? White box testing puts the focus on deriving test cases from the code structure of the IUT as it was implemented But what if that structure doesn’t just contain faults, but is in itself faulty? E.g., missing code, such as an if-statement that needs an else branch E.g., developer misunderstood specifications What question does this raise? Need to go back to the specifications to design good test cases! CompSci 230: WB

The ideal white box tester Must-have: Knows the programming language that the IUT is written in Knows and understands the specifications perfectly Was not involved in the thought processes that led to the implementation (carries no preconception as to how the problem ought to be solved, or is biased by how it was solved) So ideally, the tester will not be the developer CompSci 230: WB

Myer’s Principles for Testing A necessary part of a test case is a definition of the expected output or result A programmer should avoid attempting to test his or her own program A programming organization should not test its own programs Thoroughly inspect the results of each test Test cases must be written for input conditions that are invalid and unexpected, as well as for those that are valid and expected Examining a program to see if it does not do what it is supposed to do is only half the battle; the other half is seeing whether the program does what it is not supposed to do Avoid throwaway test cases unless the program is truly a throwaway program Do not plan a testing effort under the tacit assumption that no errors will be found The probability of the existence of more errors in a section of a program is proportional to the number of errors already found in that section Testing is an extremely creative and intellectually challenging task CompSci 230: WB

Example Principle 4: Thoroughly inspect the results of each test Test case input for the method: an identifier for an album in the database Expected return: A list of song titles on the album with their durations in minutes and seconds Get: Sweet love in the rainy sunshine 3:12 Ginger on my mind 4:42 Oh come on (Remix) 3:65 Platitudes for platypus 3:52 Ginger on my mind (Remix) 4:20 Looks like a pass, but is it? Any other principles at stake here? CompSci 230: WB

Review Is it harder to be a black box tester or a white box tester? Can you explain the difference between statement coverage, branch coverage, path coverage and condition coverage in a paragraph or two? Which of statement coverage, branch coverage, path coverage and condition coverage can we usually achieve (if we don’t mind the effort)? Which challenges do we face in white box testing? Name three consequences of following Myers’ principles Do Myers principles guarantee that testing will result in fault-free software? CompSci 230: WB