2. Formal Methods and Testing

3. Goal: software reliability Use software engineering methodologies to develop the code. Use formal methods during code development

4. What are formal methods? Techniques for analyzing systems, based on some mathematics. This does not mean that the user must be a mathematician. Some of the work is done in an informal way, due to complexity.

5. Examples for FM Deductive verification: Using some logical formalism, prove formally that the software satisfies its specification. Model checking: Use some software to automatically check that the software satisfies its specification. Testing: Check executions of the software according to some coverage scheme.

6. Typical situation: Boss: Mark, I want that the new internet marketing software will be flawless. OK? Mark: Hmmm. Well,..., Aham, Oh! Ah??? Where do I start? Bob: I have just the solution for you. It would solve everything.

7. Some concerns Which technique? Which tool? Which experts? What limitations? What methodology? At which points? How expensive? How many people? Needed expertise. Kind of training. Size limitations. Exhaustiveness. Reliability. Expressiveness. Support.

8. Badmouth Formal methods can only be used by mathematicians. The verification process is itself prone to errors, so why bother? Using formal methods will slow down the project.

9. Some answers... Formal methods can only be used by mathematicians. Wrong. They are based on some math but the user should not care. The verification process is itself prone to errors, so why bother? We opt to reduce the errors, not eliminate them. Using formal methods will slow down the project. Maybe it will speed it up, once errors are found earlier.

10. Some exaggerations Automatic verification can always find errors. Deductive verification can show that the software is completely safe. Testing is the only industrial practical method.

11. Our approach Learn several methods (deductive verification, model checking, testing process algebra). Learn advantages and limitations, in order to choose the right methods and tools. Learn how to combine existing methods.

12. Emphasis The process: Selecting the tools, Modeling, Verification, Locating errors. Use of tools: Hands on. PVS, SPIN (but not in this course). Visual notation: Statecharts, MSCs, UML.

13. Some emphasis The process of selecting and using formal methods. The appropriate notation. In particular, visual notation. Hands-on experience with tools.

14. Where do we start? Boss: Mark, can you verify this for me? Mark: OK, first I have to...

15. Things to do Check the kind of software to analyze. Choose methods and tools. Express system properties. Model the software. Apply methods. Obtain verification results. Analyze results. Identify errors. Suggest correction.

16. Different types of software Sequential. Concurrent. Distributed. Reactive. Protocols. Abstract algorithms. Finite state.

17. Specification: Informal, textual, visual The value of x will be between 1 and 5, until some point where it will become 7. In any case it will never be negative. (1 =0 1<=x<=5 X=7 X>=0

18. Verification methods Finite state machines. Apply model checking. Apply deductive verification (theorem proving). Program too big, too complicated. Apply testing techniques. Apply a combination of the above!

19. Modeling Use the program text. Translate to a programming language embedded in some proof system. Translate to some notation (transition system). Translate to finite automata. Use visual notation. Special case: black box system.

20. What is testing? Testing is not showing that there are no errors in the program. Testing cannot show that the program performs its intended goal correctly. So, what is testing? Testing is the process of executing the program in order to find errors. A successful test is one that finds an error.

21. Some drawbacks of testing There are never sufficiently many test cases. Testing does not find all the errors. Testing is hard and takes a lot of time. Testing is still a largely informal task.

22. Black-Box (data-driven, input-output) testing The testing is not based on the structure of the program (which is unknown). In order to ensure correctness, every possible input needs to be tested - this is impossible! The goal: to maximize the number of errors found.

23. White-Box testing Is based on the internal structure of the program. There are several alternative criterions for checking enough paths in the program. Even checking all paths (highly impractical) does not guarantee finding all errors (e.g., missing paths!)

24. Some testing principles A programmer should not test his/her own program. One should test not only that the program does what it is supposed to do, but that it does not do what it is not supposed to. The goal of testing is to find errors, not to show that the program is errorless. No amount of testing can guarantee error-free program. Parts of programs where a lot of errors have already been found are a good place to look for more errors. The goal is not to humiliate the programmer!

25. Inspections and Walkthroughs Manual testing methods. Done by a team of people. Performed at a meeting (brainstorming). Takes 90-120 minutes. Can find 30%-70% of errors.

26. Code Inspection Team of 3-5 people. One is the moderator. He distributes materials and records the errors. The programmer explains the program line by line. Questions are raised. The program is analyzed w.r.t. a checklist of errors.

27. Checklist for inspections Data declaration All variables declared? Default values understood? Arrays and strings initialized? Variables with similar names? Correct initialization? Control flow Each loop terminates? DO/END statements match? Input/output OPEN statements correct? Format specification correct? End-of-file case handled?

28. Walkthrough Team of 3-5 people. Moderator, as before. Secretary, records errors. Tester, play the role of a computer on some test suits on paper and board.

29. Selection of test cases (for white-box testing) The main problem is to select a good coverage criterion. Some options are: Cover all paths of the program. Execute every statement at least once. Each decision has a true or false value at least once. Each condition is taking each truth value at least once. Check all possible combinations of conditions in each decision.

30. Cover all the paths of the program Infeasible. Consider the flow diagram on the left. It corresponds to a loop. The loop body has 5 paths. If the loops executes 20 times there are 5^20 different paths! May also be unbounded!

31. Statement coverage Execute every statement at least once By choosing A=2,B=0,X=3 each statement will be chosen. The case where the tests fail is not checked! IF (A>1)&(B=0) THEN X=X/A END; IF (A=2)|(X>1) THEN X=X+1 END; Now x=1.5

32. Decision coverage Each decision has a true and false outcome at least once. Can be achieved using A=3,B=0,X=3 A=2,B=1,X=1 Problem: Does not test individual conditions. E.g., when X>1 is erroneous in second decision. IF (A>1)&(B=0) THEN X=X/A; END; IF (A=2)|(X>1) THEN X=X+1; END;

33. Decision coverage A=3,B=0,X=3 IF (A>1)&(B=0) THEN X=X/A; END; IF (A=2)|(X>1) THEN X=X+1; END; Now x=1

34. Condition coverage Each condition has a true and false value at least once. For example: A=1,B=0,X=3 A=2,B=1,X=0 lets each condition be true and false once. Problem:covers only the path where the first test fails and the second succeeds. (A>1) IF (A>1)&(B=0) THEN X=X/A END; IF (A=2)|(X>1) THEN X=X+1 END;

35. Condition coverage A=1,B=0,X=3 (A>1) IF (A>1) & (B=0) THEN X=X/A; END; IF (A=2) | (X>1) THEN X=X+1; END;

36. Condition coverage A=2,B=1,X=0 Did not check the first THEN part at all!!! Can use condition+decision coverage. (A>1) IF (A>1)&(B=0) THEN X=X/A; END; IF (A=2)|(X>1) THEN X=X+1; END;

37. Multiple Condition Coverage Test all combinations of all conditions in each test. A>1,B=0 A>1,B 0 A 1,B=0 A 1,B 0 For second IF A=2,X>1 A=2,X 1 A 2,X>1 A 2,X 1 IF (A>1)&(B=0) THEN X=X/A END; IF (A=2)|(X>1) THEN X=X+1 END;

38. A smaller number of states that cover all cases: A=2,B=0,X=4 A=2,B=1,X=1 A=1,B=0,X=2 A=1,B=1,X=1 Note the X=4 in the first case: it is due to the fact that X changes before being used! IF (A>1)&(B=0) THEN X=X/A; END; IF (A=2)|(X>1) THEN X=X+1; END; Further optimization: not all combinations. For C /\ D, check (C, D), ( C, D), (C, D). For C \/ D, check ( C, D), ( C, D), (C, D).

39. Test cases based on data-flow analysis Partition the program into pieces of code with a single entry/exit point. For each piece find which variables are set/used/tested. Various covering criteria: from each set to each use/test From each set to some use/test. x:=3 x>2? y:=x+1 Variable set Variable use Variable test

40. Test case design for black box testing Equivalence partition Boundary value analysis Cause-effect graphs

41. Equivalence partition Goals: Find a small number of test cases. Cover as much possibilities as you can. Try to group together inputs for which the program would likely to behave the same. Specification condition Valid equivalence class Invalid equivalence class

42. Example: A legal variable Begins with A-Z Contains [A-Z0-9] Has 1-6 characters. Specification condition Valid equivalence class Invalid equivalence class Starting char Chars Length Starts A-ZStarts other [A-Z0-9]Has others 1-6 chars0 chars, >6 chars 12 34 5 67

43. Equivalence partition (cont.) Add a new test case until all valid equivalence classes have been covered. A test case can cover multiple such classes. Add a new test case until all invalid equivalence class have been covered. Each test case can cover only one such class. Specification condition Valid equivalence class Invalid equivalence class

44. AB36P (1,3,5) 1XY12 (2) A17#%X (4) Specification condition Valid equivalence class Invalid equivalence class Starting char Chars Length Starts A-ZStarts other [A-Z0-9]Has others 1-6 chars0 chars, >6 chars 12 34 5 67 (6) VERYLONG (7)

45. Boundary value analysis In every element class, select values that are closed to the boundary. If input is within range -1.0 -- +1.0, select values -1.001, -1.0, -0.999, 0.999, 1.0, 1.001. If needs to read N data elements, check with N-1, N, N+1. Also, check with N=0.