2. Software Testing and Reliability Southern Methodist University CSE 7314

3. Functional and structural approaches to testing

4. A few words on combinatorics Based on the Cartesian product of sets, we can count the number of possible inputs that a program has, i.e. | I |

5. Example Assume a program has a single input, Customer ID (CID) May be any value in the domain {00000-99999} What is | I | ?

6. Example Now assume we add a second input, the Order ID (OID) This may be any value in the domain {00000-99999} as well Now what is | I |?

7. Example Finally, add a credit card number to the input This is a 12 digit number | I | has now reached 10**22 If we can execute 1 million tests per second, it will take 1016 seconds, or about 300 million years!!

8. Example Since we cannot know what data may exercise a given statement/path in general, we may attempt to resort to exhaustive testing This attempt is doomed to fail due to the combinatorial explosion

9. Engineering the testing process Any engineered product (and most other things) can be tested in one of two ways –Knowing the specified function that a product has been designed to perform, tests can be conducted that demonstrate each function is fully operational while at the same time searching for errors in each function

10. Engineering the testing process –Knowing the internal workings of a product, tests can be conducted to ensure that “all gears mesh” (internal operations performed according to specifications)

11. Structural testing Uses knowledge of the internal workings Also known as Clear box/glass box Code based Can be useful for finding interesting inputs Misses an entire class of faults, missing code

12. Behavioral Uses knowledge of the specific function that is to be performed Based solely on the specification without regard for the internals Also known as Black box More user oriented Misses an entire class of faults, extra code (surprises) except by accident

13. Passing criteria How do we know when 1. a single test has passed 2. when we are done testing

14. Passing criteria A single test passes when its output is correct –This requires a specific definition of correct and ties into the automated oracle problem

15. When are we done? Conway Criteria: No syntactic errors (it compiles) No compile errors or immediate execution failures There exists Some set of data for which the program gives the correct output A typical set of data produces the correct output

16. When are we done? Difficult sets of data produce the correct output. All possible data sets in the problem specification produce the correct output All possible data sets and likely erroneous input succeeds. All inputs produce the correct output

17. Nature of software defects Logic errors and incorrect assumptions are inversely proportional to the probability that a program path will be executed

18. Nature of software defects We often believe that a logical path is not likely to be executed when, in fact, it may be executed on a regular basis Typographical errors are random More of a case for WHITE box testing……

19. Overview of Coverage testing

20. Logical coverage What does statement coverage imply ?

21. Example 1 IF (( X > 1) AND ( Y == 1)) 2 Z = Z / X 3 END 4 IF ((X == 2) OR ( Z >1)) 5 Z = Z + 1 6 END What is required for statement coverage ?

22. Example To achieve statement coverage, one can choose x = __, y = __ Many possible errors are missed: 1 IF ((X > 1) OR (Y ==1)) 1 IF ((X >=1) AND (Y ==1)) 1 IF ((X > 1) AND (Y >=1))

23. Blindness Statement coverage is blind to several classes of error when choosing values to achieve coverage

24. Assignment blindness Due to assignment of a particular value to a variable, the error does not propagate

25. Equality blindness Due to an Equality check of a particular variable, the error does not propagate

26. Self blindness Conditional itself covers up the error

27. Assignment Blindness example 1 IF ( ) 2 X = 1 3 END 4 IF (X + Y > 1) //Should have been (Y > 0) 5.... 6 END

28. Assignment Blindness example In this example, values are chosen to make the conditional true, the statement X=1 is executed and the error in line 4 is not seen

29. Assignment Blindness example If every path forces the same assignment, then the 'error' doesn't really matter, (does it exist??) –For instance, if the conditional in statement 1 always evaluated to true

30. Equality Blindness Example 1 IF (A == 2) 2..... 3 IF (B > 1) // Should have been (A + B > 3)

31. Equality Blindness Example In this example, the value of 2 is chosen for A to force execution of the body The error in statement 3 is missed

32. Self Blindness 1 IF (X < 1) // Should have been (2X < 1) In this example, the value of 0 is chosen for X

33. Observation Statement coverage is the weakest form of coverage Many classes of errors can escape this testing Typical projects have less than 80% statement coverage!!

34. Branch coverage Each branch of a decision must be executed at least once This is stronger that statement coverage, but it is still weak Note: A decision is a logical combination of one or more conditions

35. Branch coverage example 1 IF (( X > 1) AND ( Y == 1)) 2 Z = Z / X 3 END 4 IF ((X == 2) OR ( Z >1)) 5 Z = Z + 1 6 END

36. Branch coverage example To achieve branch coverage, one option is to choose –x = __, y = __

37. Branch coverage example Many possible errors are still missed: 1 IF ((X > 1) OR ( Y ==1) 1 IF ((X >=1) AND (Y ==1)) 1 IF ((X > 1) AND (Y <=1))

38. Blindness in branch coverage Branch coverage is blind to several classes of error when choosing values to achieve coverage –Compound decisions (more than one condition) are weakly tested –Boundaries of conditions (within a decision) are not explicitly tested

39. Condition coverage Each condition within a decision must assume all possible values This is 'potentially' stronger than branch coverage, but not always It may, in fact, be weaker

40. Example of condition coverage 1 IF (( X > 1) AND ( Y == 1)) 2 Z = Z / X 3 END 4 IF ((X == 2) OR ( Z >1)) 5 Z = Z + 1 6 END

41. Example of condition coverage To achieve condition coverage for statement 1, one must choose values such that: X > 1 ~(Y == 1) ~(X > 1) Y == 1

42. Example of condition coverage Both of these vectors miss the execution of statement 2 A better choice may be: ~(X > 1) ~(Y == 1) X > 1 Y == 1

43. Multi-condition coverage Every combination of possible condition values within a decision must be chosen This is strictly stronger than branch or condition coverage –still has weaknesses!

44. Multi-condition coverage example 1 IF (( X > 1) AND ( Y == 1)) 2 Z = Z / X 3 END 4 IF ((X == 2) OR ( Z >1)) 5 Z = Z + 1 6 END

45. Multi-condition coverage example To achieve multi-condition coverage for statement 1, one must choose values such that: 1. X > 1 Y == 1 2. X > 1 ~(Y == 1) 3. ~(X > 1) Y == 1 4. ~(X > 1) ~(Y == 1)

46. Multi-condition coverage example To achieve multi-condition coverage for statement 2, one must choose values such that: 5. X == 2 Z > 1 6. X == 2 ~(Z > 1) 7. ~(X == 2) Z > 1 8. ~(X == 2) ~(Z > 1)

47. Coverage XYZCovers 1011,5 2144,8 1132,6 2013,7

48. Multi-condition coverage example There are 9 paths through this code and this test set only covers 4 of those

49. Multi-condition coverage example Consider this error Line 2 is incorrectly written as Z = Z –X Only the second vector will execute this line but for the values chosen, Z-X = Z/X

50. Multi-condition coverage example Also consider that no vector will execute both line 2 and line 5 In a more complex example, it is easy to imagine variables being set in line 2 that are then incorrectly used in line 5

51. Multi-condition coverage example For instance: 2. Z = Z/X; A = 0.... 5. Z = Z + 1; B = 1/A;

52. Path coverage Every possible path must be chosen Finding vectors to accomplish this is, in general, undecidable, and usually difficult

53. Path coverage This still misses boundary conditions If statement 1 was mis-typed as X >= 1, it is conceivable that the chosen test set does not contain a vector with X=1, missing this error

54. Basis path testing

55. White box technique Test case designer can derive a logical complexity measure of a procedural design Control flow represented by a control graph

56. Flow graph notation sequence if

57. Flow graph notation while

58. Flow graph notation case

59. Flowchart 1 2 3 4 5 6 7 8 9 10 11

60. Flow graph 1 2,3 6 7 8 9 10 11 4,5 R3 R2 R1 R4 region node edge

61. Compound logic Predicate nodes a b yx x IF a OR b then procedure x else procedure y ENDIF

62. Cyclomatic complexity Software metric that provides a quantitative measure of the logical complexity of a program Value computed defined the number of independent paths in the basis set of a program Upper bound for the number of tests that must be conducted to ensure all statements have been executed at least once

63. Independent path Any path through the program that introduces at least one new set of processing statements or a new condition Must move along at least one edge that has not been traversed before the path is defined

64. Example Path 1; 1-11 Path 2; 1-2-3-4-5-10-1-11 Path 3; 1-2-3-6-8-9-10-1-11 Path 4; 1-2-3-6-7-9-10-1-11 1-2-3-4-5-10-1-2-3-6-8-9-10-1-11 is not independent because no new edges traversed

65. Basis set Paths 1,2,3, and 4 constitute the basis set If tests can be designed to force execution of these paths, every statement in the program will have been guaranteed to be executed at least one time and every condition will have been executed on its true and false sides

66. Basis set Basis set is not unique; a number of different basis sets can be derived for a given procedural design

67. How do we know how many paths to look for? Compute the cyclomatic complexity A graph theoretic computation which yields a good software metric

68. Three ways of computing 1. The number of regions of the flow graph correspond to the cyclomatic complexity 2. V(G) = E – N + 2 –E = number of flow graph edges –N = number of flow graph nodes 3. V(G) = P + 1 –P = number of predicate nodes in G

69. V(G) for Flow graph 1 2,3 6 7 8 9 10 11 4,5 R3 R2 R1 R4 region node edge

70. Example 1. Flow graph has 4 regions 2. V(G) = 11 edges – 9 nodes + 2 = 4 3. V(G) = 3 predicate nodes + 1 = 4 V(G) provides upper bound on number of tests that must be designed and executed to guarantee coverage of all program statements

71. Deriving test cases Basis path testing can be applied to a procedural design or to source code

72. PDL for test case design PROCEDURE average; * This procedure computes the average of 100 or fewer numbers that lie between bounding values; it also computes the sum and the total number valid. INTERFACE RETURNS average, total.input, total.valid; INTERFACE ACCEPTS value, minimum, maximum; TYPE value[1:100] IS SCALAR ARRAY; TYPE average, total.input, total.valid; minimum, maximum, sum IS SCALAR; TYPE i IS INTEGER

73. i = 1; total.input = total.valid = 0; sum =0; DO WHILE value[i] <> -999 AND total.input < 100; increment total.input by 1; IF value[i] >= minimum AND value[i] <= maximum THEN increment total.valid by 1; sum = sum + value[i]; ELSE skip ENDIF increment i by 1; ENDDO IF total.valid > 0 THEN average = sum / total.valid; ELSE average = -999 ENDIF END average

74. Deriving test cases for “average” 1. Using the design or code as a foundation, draw a corresponding flow graph

75. i = 1; total.input = total.valid = 0; sum =0; DO WHILE value[i] <> -999 AND total.input < 100; increment total.input by 1; IF value[i] >= minimum AND value[i] <= maximum THEN increment total.valid by 1; sum = sum + value[i]; ELSE skip ENDIF increment i by 1; ENDDO IF total.valid > 0 THEN average = sum / total.valid; ELSE average = -999 ENDIF END average Identify the nodes in this PDL!!!

76. Draw the Flow graph

77. Deriving test cases for “average” 2. Determine the cyclomatic complexity of the resultant flow graph –V(G) can be determined without developing a flow graph by counting all conditional statements in the PDL (compound conditions count as two) and then adding 1

78. Computer V(G) using the three methods 2. (continued) –V(G) =

79. Deriving test cases for “average” 3. Determine a basis set of linearly independent paths –Value of V(G) provides the number of linearly independent paths Path 1; Path 2; Path 3; Path 4; Path 5; Path 6; List the basis set paths!!

80. Deriving test cases for “average” 4. Prepare test cases that will force execution of each path in the basis set –Data should be chosen to that conditions at the predicate nodes are appropriately set as each path is tested. Test cases that satisfy the basis set just described are;

81. Prepare a set of test cases

82. Summary Each test case is executed and compared to expected results Ensures all statements executed at least once Some independent paths cannot be tested in stand-alone fashion –Tested as part of another path test

83. Graph matrices

84. Testing basis paths can be mechanized by using graph matrices Size based on number of nodes in the graph Tabular representation of a flow graph

85. Graph matrix 1 3 4 2 5 a b c d e f g

86. a db cf ge 1 2 3 4 5 1234512345 node Connected to node

87. Link weight Provides additional information about control flow Connection either exists (1) or does not exist (0) Can also represent –Probabilities of execution –Processing time of a link –Memory or resources required

88. Connection matrix 1 11 11 11 1 2 3 4 5 1234512345 node Connected to node connections 1 – 1 = 0 2 – 1 = 1 3 + 1 = 4 Cyclomatic complexity

89. Data flow testing

90. This approach selects test paths according to the locations of definitions and uses of variables in the program Definition Use (DU) testing –Does not guarantee coverage of all branches of a program but does pretty good

91. Example proc x B1; do while C1 if C2 then if C4 then B4; else B5; endif; else if C3 then B2; else B3; endif; enddo; B6; end proc;

92. Now apply the DU testing strategy We must first know the definitions and uses of variables in each condition or block in the PDL Assume variable x is defined in the last statement of blocks B1, B2, B3, B4, and B5 and used in the first statement of blocks B2, B3, B4, B5, and B6

93. Now apply the DU testing strategy The DU testing strategy requires an execution of the shortest path from each of B i, 0<i<5, to each of B j, 1<j≤6 This testing also covers any use of variable x in conditions C1, C2, C3, and C4

94. Now apply the DU testing strategy There are 25 DU chains of variable x, but we need only five paths to cover these DU chains 5 paths are needed to cover the DU chain of x from B j, 1<j≤5, and other BU chains can be covered by making these five paths contain iterations of the loop

95. Loop testing

96. Loops are the cornerstone of many algorithms in software White box testing technique Four different classes of loops –Simple loops –Concatenated loops –Nested loops –Unstructured loops

97. Simple loops

98. Testing Simple loops Skip the loop entirely Only one pass through the loop Two passes through the loop m passes through the loop where m < n (n is maximum allowable passes through the loop) n-1, n, n+1 passes through the loop

99. Nested loops

100. Cannot simply extend the approach from simple loops –Can grow geometrically 1. Start at the innermost loop. Set all other loops to minimum values

101. Nested loops 2. Conduct simple loop tests for the innermost loop while holding the outer loops at their minimum iteration parameter (e.g. loop counter) values. Add other tests for out of range or excluded values

102. Nested loops 3. work outward, conducting tests for the next loop, but keeping all other outer loops at minimum values and other nested loops to “typical” values 4. continue until all loops have been tested

103. Concatenated loops

104. These loops can be tested using the approach defined for simple loops, if each of the loops is independent of the other However, if two loops are concatenated and the loop counter for loop 1 us used as the initial value for loop 2, then the loops are not independent. When this happens, use the nesting loops approach

105. Unstructured loops

106. Unstructured loop Redesign !!! Use structured programming constructs http://acweb.colum.edu/users/rc ourington/Progclass/struprog.ht ml

107. End of lecture