1. Basic Concepts of Software Testing (2.1 – 2.3) Course Software Testing & Verification 2014/15 Wishnu Prasetya

2. Software error, how costly can it be ? Some errors in the software of UK Inland Revenue is rumored to cause tax-credit over- payment of 3.45 billions USD. (Charette, Why Software Fails. IEEE Spectrum, 2005) 2

3. On the other hand, quality costs effort 3 World Quality Report, 2014/15. 1543 CIOs, 25 countries

4. Challenges in Testing How to make a good test-set ? When have tested enough? Can we automate the execution of the tests? Can we automatically generate tests? If a test fails, how to find the source of the failure? (debugging) Can we automate debugging? 4

5. Testing in large projects 5 is usually organized in stages like this (also called V-model ) : By users/customers (or 3 rd party hired to represent them) By developers Ch. 7 provides you more background on “practical aspect”, e.g. concrete work out of the V- model, outlines of “test plan”  read the chapter yourself!

6. Testing is inherently incomplete Can we at least do it systematically ? – require you to define “test requirements” – come up with a reasonable definition, and preferably also quantifiable (so that you some indication of how far you are from fulfilling the requirements) “Fulfilling TR” does not imply absence of error. Not “fulfilling TR” increases the probability of still having errors. 6

7. Terms used A test-set (Def 1.18): is just a set of test-cases. (Deviate from Def. 1.17) A test-case specifies a series of interactions with the target program and the input needed by those interactions. Some interactions may trigger responses from the program. The test-case also specify what kind of responses it expects from the program. – In the simple case: just one interaction foo(x), and one response, namely return value. Test inputs : the inputs meant above. Test oracle: specify the expectation on responses as meant above. 7

8. Now, how do AO formulate this? (I reformulate it a bit) Def. 1.21. A coverage criterion = a set TR of test requirements to fulfill/cover. Ex. CC: TR: the set of all P’s lines. Def. 1.20. A test requirement specifies a single goal (e.g. a position in your program) that your test-set must cover. 8

9. “line coverage” is too imprecise? Abstractly, a program is a set of branches and branching points. An execution flows through these branches, to form a path through the program. Such an abstraction is called Control Flow Graph (CFG, Legard 1970), such as the one above. AO gives you a set of more precise concept of coverage, based on graph. 9 foo(x,y) { while (x>y) x=x-y ; if (x<y) return -1 else return x }

10. Graph terminologies... (Sec 2.1) A graph is described by (N,N 0,N f,E). E is a subset of N×N. So: – directed, no labels, no multi-edges. A path p : [n 0, n 1,..., n k ] – non-empty! – each consecutive pair is an edge in E length of p = #edges in p = natural length of p - 1 Test-path (Def. 2.31) 10

11. Graph terminologies (OA forget to define this!) A path p is a subpath of q if it is not longer than q and: (  i: i  0 : (  k : 0  k  length p : p k = q i+k )) [1,2] is a subpath of [0,1,2,3] [1,2] is not a subpath of [1,0,2] OA also use the term “q tours p” 11

12. We use TR to specify how to tour Consider a test-requirement that specify a path q to cover. A test-path p tours q if q is a subpath of p. (C2.8) The Coverage Criterion is to tour a given set of paths as TR. Example, TR = { [0,0], [2], [3] } which test-set is good enough to satisfy this? 12 01 3 2

13. OA’s graph coverage criteria (C2.1) Node coverage: TR is the set of paths of length 0 in G. (C2.2) TR is the set of paths of length at most 1 in G. – So, what does this criterion tell? – Why “at most” ? (Def 1.24) A coverage criterion C1 subsumes (stronger than) C2 iff every test-set that satisfies C1 also satisfies C2. 13

14. Does C2.2 subsume C2.1 ? Obviously. How about the other way around? Are we happy now!? How about C2.3 (Edge- pair coverage): TR contains all paths of length at most 2. 14 1 2 0 0 2 1 63 5 4

15. Potential weakness: diamonds row CRMin. size of test-set that will satisfy it Node (C2.1)2 Edge (C2.2)2 Edge-pair (C2.3)4 15 0 2 1 6 3 5 4 9 7 8 A row of k diamond has 2 k different paths. However, suggesting that TR should include all paths in G is not realistic either (think of loops!).

16. AO propose prime paths coverage A simple path p is a path where every node in p appears only once, except the first and the last which are allowed to be the same. Such a path is a prime path if it is not a proper subpath of another simple path. 16 120 3

17. Prime Path Coverage (PPC) (C2.4) TR is the set of all prime paths in G. – Strong in a diamond row (force you to cover all 2 k paths). – Still finite. – But... 17 This can be toured with just one test path. We can fix C2.4 e.g. by requiring minimum length test-paths, but we will not go into this. 2 0 1 TR = { 012, 010, 101 }

18. Calculating PPC’s TR How long can a prime path be? It follows that they can be systematically calculated, e.g. : 1.“invariant” : maintain S to be a set of simple and still extensible (at the end-side) paths, and T of non-extensible (at the end-side) simple paths 2.“extend” S by extending the paths in it; we move non-extensibles to T  keep the invariant maintained. 3.This will terminate!  T contains non-extensible simple paths. 18

19. Example 19 5 0 2 1 34 6 [0] [1] [2] [3] [4] [5] [6] ! [0, 1] [0, 2] [1, 2] [2, 3] [2, 4] [3, 6] ! [4, 6] ! [4, 5] [5, 4] We mark simple paths that cannot be extended. We also mark cycles. !  cannot be extended *  cycle [0, 1, 2] [0, 2, 3] [0, 2, 4] [1, 2, 3] [1, 2, 4] [2, 3, 6] ! [2, 4, 6] ! [2, 4, 5] ! [4, 5, 4] * [5, 4, 6] ! [5, 4, 5] * S :

20. Example 20 5 0 2 1 34 6 [0, 1, 2] [0, 2, 3] [0, 2, 4] [1, 2, 3] [1, 2, 4] [2, 3, 6] ! [2, 4, 6] ! [2, 4, 5] ! [4, 5, 4] * [5, 4, 6] ! [5, 4, 5] * [0, 1, 2, 3] [0, 1, 2, 4] [0, 2, 3, 6] ! [0, 2, 4, 6] ! [0, 2, 4, 5] ! [1, 2, 3, 6] ! [1, 2, 4, 5] ! [1, 2, 4, 6] ! [0, 1, 2, 3, 6] ! [0, 1, 2, 4, 6] ! [0, 1, 2, 4, 5] ! S is now empty; terminate. Prime paths = all the (*) paths, plus the non-cyclic simple paths (!) which are not a subpath of another (!)

21. Unfeasible test requirement If it turns out that there is no real execution that can cover it For example, we cannot test a Kelvin thermometer below 0 o. 21

22. Typical unfeasibility in loops Some loops always iterate at least once. E.g. above if a is never empty  prime path 124 is unfeasible. (Def 2.37) A test-path p tours q with sidetrips if all edges in q appear in p, and they appear in the same order. 22 i = a.length-1 ; while (i  0) { if (a[i]==0) break ; i-- } 2 1 3 4 12324 does not tour 124, but with sidetrips it does.

23. Generalizing Graph-based Coverage Criterion (CC) Obviously, sidetrip is weaker that strict touring. Every CC we have so far (C2.1, 2.2, 2.3, 2.4, 2.7) can be thought to be parameterized by the used concept of “tour”. We can opt to use a weaker “tour”, suggesting this strategy: – First try to meet your CC with the strongest concept of tour. – If you still have some paths uncovered which you suspect to be unfeasible, try again with a weaker touring. 23

24. Oh, another scenario... Often, loops always break in the middle. E.g. above if a always contain a 0. Problem: e.g. 1234 does not tour 124, not even with sidetrip! (Def 2.38) A test-path p tours q with detours if all nodes in q appear in q, and they appear in the same order. Weaker than sidetrip. 24 2 1 3 4

25. Pre-calculating the needed test-paths Given a set of paths as your TR (e.g. set of all prime paths), can we pre-calculate a set T of test-paths that would tour TR ? Let q = [n 0,...,n k ] be the target path to tour: – calculate a path p from N 0 to n 0. – calculate a path r from n k to N f. – The path front(p)++q++tail(r) will tour q. Path from a to b can be calculated using e.g. depth-first algorithm, breath-first algorithm, shortest-path algorithm. 25

26. Breath/depth first algorithm, data structures queue : a list with these operations – isEmpty – add(x), adding x at the end of the list (after its last element) – retrieve() : retrieving the first element of the list. stack: a list with the above operators – but add(x) will now add x at the front of the list. – add is also called push, and retrieve is called pop. We will also maintain a set of already visited nodes. 26

27. Breath/Depth First Algorithm 27 bool reachable(a,b) { visited = { } queue.add(a) while (queue is not empty) { x = queue.retrieve() if (x==b) return true visited.add(x) for each y in { y | (x,y) in E && y not in visited } queue.add(S) } return false } If you replace queue with stack, you get a depth first algorithm. 5 0 2 1 34 6

28. Extending the BFS Extend the breath-first algorithm (BFS) so that it also construct a tree from a to all nodes reachable from a. Once you have this tree, calculating a path from a to b is easy. Data structure: tree, with these operations: – root(a) constructs a tree with a single node a. – add( ) adds an edge into the tree (x must already be in the tree, and y is not already in the tree) 28 5021346

29. Overlying it with tree-construction 29 reachable(a,b) { visited = { } ; tree = root(a) queue.add(a) while (q is not empty) { x = queue.retrieve() if (x==a) return tree visited.add(x) for each y in { y | (x,y) in E && y not in visited } queue.add(S) tree.add( ) } return empty } 5 0 2 1 34 6

30. Reconstructing the path (imperative alg) 30 path(tree,b) { if tree is empty then return Nothing ; x = tree.root ; if x == b then return Just [x] ; for each child u of x in tree do case path(u,b) of Nothing  continue Just s  return Just (x:s) return Nothing } Because the tree was obtained from BFS, the reconstructed path will be the shortest one.

31. Mapping your program to its CFG many ways, depending on purpose See Sec. 2.3.1. AO use left; I usually use middle. (Sec 2.3.1) Define a block a maximum sequence of “instructions” so that if one instruction is executed, all the others are always executed as well. Implies that a block – a block does not contain a jump or branching instruction, except if it is the last one – no instruction except the first one, can be the target of a jump/branching instruction in the program. 31 P1(x) { if (x==0) { x++ ; return 0 } else return x } x++ ; return 0 dummy return x x==0 x!=0 x++ ; return 0 (x==0) return x x++ (x==0) return x return 0

32. Mapping your program to its CFG 32 P2(a) { for (int i=0; i<a.length ; i++) { if (a[i]==0) break ; if (odd a[i]) continue ; a[i] = 0 } Add “end-of-program” as a virtual node. 3 2 4 1 0 5

33. Discussion: exception 33 P3(...) { try { File f = openFile(“input”) x = f.readInt() y = (Double) 1/x } catch (Exception e) { y = -1 } return y } Every instruction can potentially throw an exception. Yet representing each instruction as its own node in our CFG will blow up the size of the graph, and thus also the size of the TR to tour. A minimalistic approach is to ignore the fact different instructions in the same block may throw different exceptions, and thus may be responsible for their own kind of error. Ignoring this, we can pretend that the jump to an exception handler always happen at the end of the block, so giving us the above CFG. But you need to be aware of the above ignorance. 1 10

34. Discussion: exception 34 P3(...) { try { File f = openFile(“input”) x = f.readInt() y = (Double) 1/x } catch (IOException e) { y = -1 } catch (Exception e) { y = -1 } return y } 1 1a 0 1b Had we programmed as above, we can now at least distinguish between an execution that throws an IOException and other kind of exceptions (because they are represented by different nodes in the CFG), but of course we are still blind to the distinction between e.g. an execution that throws a ParseException and that the throws DivisionByZeroException.

35. More discussion What to do with dynamic binding ? What to do with recursion? 35 f(x) { if (x  0) return a else f(....) } register(Course c, Student s) { if (! c.full()) c.add(s) ; } You don’t know ahead which “add” will be called; it depends on the run- time type of c. E.g. it may refuse to add certain type of students. Graph- based coverage is not strong enough to express this aspect.