2. Lecture 2: testing Book: Chapter 9

3. What is testing? Testing is not showing that there are no errors in the program. Testing cannot show that the program performs its intended goal correctly. So, what is testing? Testing is the process of executing the program in order to find errors. A successful test is one that finds an error.

4. Some drawbacks of testing There are never sufficiently many test cases. Testing does not find all the errors. Testing is hard and takes a lot of time. Testing is still a largely informal task.

5. Black-Box (data-driven, input-output) testing The testing is not based on the structure of the program (which is unknown). In order to ensure correctness, every possible input needs to be tested - this is impossible! The goal: to maximize the number of errors found.

6. White-Box testing Is based on the internal structure of the program. There are several alternative criterions for checking enough paths in the program. Even checking all paths (highly impractical) does not guarantee finding all errors (e.g., missing paths!)

7. Some testing principles A programmer should not test his/her own program. One should test not only that the program does what it is supposed to do, but that it does not do what it is not supposed to. The goal of testing is to find errors, not to show that the program is errorless. No amount of testing can guarantee error-free program. Parts of programs where a lot of errors have already been found are a good place to look for more errors. The goal is not to humiliate the programmer!

8. Inspections and Walkthroughs Manual testing methods. Done by a team of people. Performed at a meeting (brainstorming). Takes 90-120 minutes. Can find 30%-70% of errors.

9. Code Inspection Team of 3-5 people. One is the moderator. He distributes materials and records the errors. The programmer explains the program line by line. Questions are raised. The program is analyzed w.r.t. a checklist of errors.

10. An example: Sieve of Erathosthenes 235 6 - is dropped because it is divisible by 2. 7 - is added to the next additional slot. 8 - dropped (divisible by 2). 9 - dropped (divisible by 3)....

11. The Sieve program int P[101]; int V=1; int N=1; int I; main() { while (N<101) { for (I=1;I<=N;I++) if (P[I]=0) { P[I]=V++ ; N++ ; } else if (V%P[I]==0) I=N+1; else I++} } Is this program correct?

12. Checklist for inspections Data declaration All variables declared? Default values understood? Arrays and strings initialized? Variables with similar names? Correct initialization? Control flow Each loop terminates? DO/END statements match? Input/output OPEN statements correct? Format specification correct? End-of-file case handled?

13. Walkthrough Team of 3-5 people. Moderator, as before. Secretary, records errors. Tester, play the role of a computer on some test suits on paper and board.

14. Selection of test cases (for white-box testing) The main problem is to select a good coverage criterion. Some options are: Cover all paths of the program. Execute every statement at least once. Each decision has a true or false value at least once. Each condition is taking each truth value at least once. Check all possible combinations of conditions in each decision.

15. Cover all the paths of the program Infeasible. Consider the flow diagram on the left. It corresponds to a loop. The loop body has 5 paths. If the loops executes 20 times there are 5^20 different paths! May also be unbounded!

16. How to cover the executions? IF (A>1)&(B=0) THEN X=X/A; END; IF (A=2)|(X>1) THEN X=X+1; END; Choose values for A,B,X. Value of X may change, depending on A,B. What do we want to cover? Paths? Statements? Conditions?

17. Edge coverage Execute every statement at least once By choosing A=2,B=0,X=3 each statement will be chosen. The case that the tests fail is not checked! IF (A>1)&(B=0) THEN X=X/A; END; IF (A=2)|(X>1) THEN X=X+1; END;

18. Decision coverage Each decision has a true and false outcome at least once. Can be achieved using A=3,B=0,X=3 A=2,B=1,X=1 Problem: Does not test individual conditions. E.g., when X>1 is erroneous in second decision. IF (A>1)&(B=0) THEN X=X/A; END; IF (A=2)|(X>1) THEN X=X+1; END;

19. Condition coverage Each condition has a true and false value at least once. For example: A=1,B=0,X=3 A=2,B=1,X=0 lets each condition be true and false once. Problem:covers only the path where the first test fails and the second succeeds. (A>1) IF (A>1)&(B=0) THEN X=X/A; END; IF (A=2)|(X>1) THEN X=X+1; END;

20. Multiple Condition Coverage Test all combinations of all conditions in each test. A>1,B=0 A>1,B0 A<=1,B=0 A<=1,B0 A=2,X>1 A=2,X<=1 A2,X>1 A2,X<=1 IF (A>1)&(B=0) THEN X=X/A; END; IF (A=2)|(X>1) THEN X=X+1; END;

21. A smaller number of cases: A=2,B=0,X=4 A=2,B=1,X=1 A=1,B=0,X=2 A=1,B=1,X=1 Note the X=4 in the first case: it is due to the fact that X changes before being used! IF (A>1)&(B=0) THEN X=X/A; END; IF (A=2)|(X>1) THEN X=X+1; END;

22. How to find values for coverage? A>1 & B=0 A=2 | X>1 X=X+1 X=X/A no yes true A=2 | X>1 (A=2 | X>1) & ¬(A>1 & B=0) Put true at end of path. Propagate path backwards. On assignment, relativize expression. On yes edge of decision, add decision as conjunction. On no edge, add negation of decision as conjunction. May need to give more specific choice of covering condition, which ones are true or false.

23. How to find values for coverage? A>1 & B=0 A=2 | X>1 X=X+1 X=X/A no yes true A2 & X>1 (A2 & X/A>1) & (A>1 & B=0) A2 & X/A>1 Need to find a satisfying assignment: A=3, X=6, B=0

24. Test case design for black box testing Equivalence partition Boundary value analysis Cause-effect graphs

25. Test cases based on data-flow analysis Partition the program into pieces of code with a single entry/exit point. For each piece find which variables are set/used/tested. Various covering criteria: from each set to each use/test From each set to some use/test.

26. Equivalence partition Goals: Find a small number of test cases. Cover as much possibilities as you can. Try to group together inputs for which the program would likely to behave the same. Specification condition Valid equivalence class Invalid equivalence class

27. Example: A legal variable Begins with A-Z Contains [A-Z0-9] Has 1-6 characters. Specification condition Valid equivalence class Invalid equivalence class Starting char Chars Length Starts A-ZStarts other [A-Z0-9]Has others 1-6 chars0 chars, >6 chars 12 34 5 67

28. Equivalence partition (cont.) Add a new test case until all valid equivalence classes have been covered. A test case can cover multiple such classes. Add a new test case until all invalid equivalence class have been covered. Each test case can cover only one such class. Specification condition Valid equivalence class Invalid equivalence class

29. AB36P (1,3,5) 1XY12 (2) A17#%X (4) Specification condition Valid equivalence class Invalid equivalence class Starting char Chars Length Starts A-ZStarts other [A-Z0-9]Has others 1-6 chars0 chars, >6 chars 12 34 5 67 (6) VERYLONG (7)

30. Boundary value analysis In every element class, select values that are closed to the boundary. If input is within range -1.0 -- +1.0, select values -1.001, -1.0, -0.999, 0.999, 1.0, 1.001. If needs to read N data elements, check with N-1, N, N+1. Also, check with N=0.