Presentation is loading. Please wait.

Presentation is loading. Please wait.

Softrare Reliability Methods Prof. Doron A. Peled Bar Ilan University, Israel And Univeristy of warwick, UK Version 2008.

Similar presentations


Presentation on theme: "Softrare Reliability Methods Prof. Doron A. Peled Bar Ilan University, Israel And Univeristy of warwick, UK Version 2008."— Presentation transcript:

1 Softrare Reliability Methods Prof. Doron A. Peled Bar Ilan University, Israel And Univeristy of warwick, UK Version 2008

2 Some related books: Also:Mainly:

3 Goal: software reliability Use software engineering methodologies to develop the code. Use formal methods during code development

4 What are formal methods? Techniques for analyzing systems, based on some mathematics. This does not mean that the user must be a mathematician. Some of the work is done in an informal way, due to complexity.

5 Examples for FM Deductive verification: Using some logical formalism, prove formally that the software satisfies its specification. Model checking: Use some software to automatically check that the software satisfies its specification. Testing: Check executions of the software according to some coverage scheme.

6 Typical situation: Boss: Mark, I want that the new internet marketing software will be flawless. OK? Mark: Hmmm. Well,..., Aham, Oh! Ah??? Where do I start? Bob: I have just the solution for you. It would solve everything.

7 Some concerns Which technique? Which tool? Which experts? What limitations? What methodology? At which points? How expensive? How many people? Needed expertise. Kind of training. Size limitations. Exhaustiveness. Reliability. Expressiveness. Support.

8 Badmouth Formal methods can only be used by mathematicians. The verification process is itself prone to errors, so why bother? Using formal methods will slow down the project.

9 Some answers... Formal methods can only be used by mathematicians. Wrong. They are based on some math but the user should not care. The verification process is itself prone to errors, so why bother? We opt to reduce the errors, not eliminate them. Using formal methods will slow down the project. Maybe it will speed it up, once errors are found earlier.

10 Some exaggerations Automatic verification can always find errors. Deductive verification can show that the software is completely safe. Testing is the only industrial practical method.

11 Our approach Learn several methods (deductive verification, model checking, testing process algebra). Learn advantages and limitations, in order to choose the right methods and tools. Learn how to combine existing methods.

12 Emphasis The process: Selecting the tools, Modeling, Verification, Locating errors. Use of tools: Hands on. PVS, SPIN Visual notation: Statecharts, MSCs, UML.

13 Some emphasis The process of selecting and using formal methods. The appropriate notation. In particular, visual notation. Hands-on experience with tools.

14 The unbearable easiness of grading During class, choose some small project in groups, e.g., Explore some examples using tools. Implementing a simple algorithm. Dealing with new material. or covering advanced subject. - Office presentation (1 hour). - Written description (2-3 pages + computer output or 6-10 pages). - Class presentation (0.5-1.5 hours per group).

15 Example topics Project: Verify some example using some tools. Communication protocols. Mutual exclusion. Advanced topics: Abstractions. Reductions. Partitions. Static analysis. Verifying pushdown automata. Verifying security protocols.

16 Where do we start? Boss: Mark, can you verify this for me? Mark: OK, first I have to...

17 Things to do Check the kind of software to analyze. Choose methods and tools. Express system properties. Model the software. Apply methods. Obtain verification results. Analyze results. Identify errors. Suggest correction.

18 Different types of software Sequential. Concurrent. Distributed. Reactive. Protocols. Abstract algorithms. Finite state.

19 Specification: Informal, textual, visual The value of x will be between 1 and 5, until some point where it will become 7. In any case it will never be negative. (1 =0)) 1<=x<=5 X=7 X>=0

20 Verification methods Finite state machines. Apply model checking. Apply deductive verification (theorem proving). Program too big, too complicated. Apply testing techniques. Apply a combination of the above!

21 Modeling Use the program text. Translate to a programming language embedded in some proof system. Translate to some notation (transition system). Translate to finite automata. Use visual notation. Special case: black box system.

22 Modeling Software Systems for Analysis (Book: Chapter 4)

23 Modelling and specification for verification and validation How to specify what the software is supposed to do? Can we use the UML model or parts of it? How to model it in a way that allows us to check it?

24 Systems of interest Sequential systems. Concurrent systems (multi-threaded). 1. Distributive systems. 2. Reactive systems. 3. Embedded systems (software + hardware).

25 Sequential systems. Perform some computational task. Have some initial condition, e.g., 0 i n A[i] integer. Have some final assertion, e.g., 0 i n-1 A[i] A[i+1]. (What is the problem with this spec?) Are supposed to terminate.

26 Concurrent Systems Involve several computation agents. Termination may indicate an abnormal event (interrupt, strike). May exploit diverse computational power. May involve remote components. May interact with users (Reactive). May involve hardware components (Embedded).

27 Problems in modeling systems Representing concurrency: - Allow one transition at a time, or - Allow coinciding transitions. Granularity of transitions. Assignments and checks? Application of methods? Global (all the system) or local (one thread at a time) states.

28 Modeling. The states based model. V={v 0,v 1,v 2, …} - a set of variables, over some domain. p(v 0, v 1, …, v n ) - a parametrized assertion, e.g., v 0 =v 1 +v 2 /\ v 3 >v 4. A state is an assignment of values to the program variables. For example: s= For predicate (first order assertion) p: p(s) is p under the assignment s. Example: p is x>y /\ y>z. s=. Then we have 4>3 /\ 3>5, which is false.

29 State space The state space of a program is the set of all possible states for it. For example, if V={a, b, c} and the variables are over the naturals, then the state space includes:,,, …

30 Atomic Transitions Each atomic transition represents a small piece of code such that no smaller piece of code is observable. Is a:=a+1 atomic? In some systems, e.g., when a is a register and the transition is executed using an inc command.

31 Non atomicity Execute the following when a=0 in two concurrent processes: P 1 :a=a+1 P 2 :a=a+1 Result: a=2. Is this always the case? Consider the actual translation: P 1 :load R1,a inc R1 store R1,a P 2 :load R2,a inc R2 store R2,a a may be also 1.

32 Scenario P 1 :load R1,a inc R1 store R1,a P 2 :load R2,a inc R2 store R2,a a=0 R1=0 R2=0 R1=1 R2=1 a=1

33 Representing transitions Each transition has two parts: The enabling condition: a predicate. The transformation: a multiple assignment. For example: a>b (c,d ):=(d,c ) This transition can be executed in states where a>b. The result of executing it is switching the value of c with d.

34 Initial condition A predicate I. The program can start from states s such that I (s) holds. For example: I (s)=a >b /\ b >c.

35 A transition system A (finite) set of variables V over some domain. A set of states. A (finite) set of transitions T, each transition e t has an enabling condition e, and a transformation t. An initial condition I.

36 Example V={a, b, c, d, e}. : all assignments of natural numbers for variables in V. T={c >0 (c,e):=(c -1,e +1), d >0 (d,e):=(d -1,e +1)} I: c =a /\ d =b /\ e =0 What does this transition system do?

37 The interleaving model An execution is a maximal finite or infinite sequence of states s 0, s 1, s 2, … That is: finite if nothing is enabled from the last state. The first state s 0 satisfies the initial condition, I.e., I (s 0 ). Moving from one state s i to its successor s i+1 is by executing a transition e t: e (s i ), i.e., s i satisfies e. s i+1 is obtained by applying t to s i.

38 Example: s0= s1= s2= s3= T={c>0 (c,e):=(c -1,e +1), d>0 (d,e):=(d-1,e+1)} I: c=a /\ d=b /\ e=0

39 L0:While True do NC0:wait(Turn=0); CR0:Turn=1 endwhile || L1:While True do NC1:wait(Turn=1); CR1:Turn=0 endwhile T0:PC0=L0 PC0:=NC0 T1:PC0=NC0/\Turn=0 PC0:=CR0 T2:PC0=CR0 (PC0,Turn):=(L0,1) T3:PC1=L1 PC1=NC1 T4:PC1=NC1/\Turn=1 PC1:=CR1 T5:PC1=CR1 (PC1,Turn):=(L1,0) Initially: PC0=L0/\PC1=L1 The transitions Is this the only reasonable way to model this program?

40 The state graph:Successor relation between reachable states. Turn=0 L0,L1 Turn=0 L0,NC1 Turn=0 NC0,L1 Turn=0 CR0,NC1 Turn=0 NC0,NC1 Turn=0 CR0,L1 Turn=1 L0,CR1 Turn=1 NC0,CR1 Turn=1 L0,NC1 Turn=1 NC0,NC1 Turn=1 NC0,L1 Turn=1 L0,L1 T0 T3 T1 T4 T3 T0T3 T0 T4 T1T3 T2 T5

41 Some important points Reachable states: obtained from an initial state through a sequence of enabled transitions. Executions: the set of maximal paths (finite or terminating in a node where nothing is enabled). Nondeterministic choice: when more than a single transition is enabled at a given state. We have a nondeterministic choice when at least one node at the state graph has more than one successor.

42 Always ¬(PC0=CR0/\PC1=CR1) (Mutual exclusion) Turn=0 L0,L1 Turn=0 L0,NC1 Turn=0 NC0,L1 Turn=0 CR0,NC1 Turn=0 NC0,NC1 Turn=0 CR0,L1 Turn=1 L0,CR1 Turn=1 NC0,CR1 Turn=1 L0,NC1 Turn=1 NC0,NC1 Turn=1 NC0,L1 Turn=1 L0,L1

43 Always if Turn=0 then at some point Turn=1 Turn=0 L0,L1 Turn=0 L0,NC1 Turn=0 NC0,L1 Turn=0 CR0,NC1 Turn=0 NC0,NC1 Turn=0 CR0,L1 Turn=1 L0,CR1 Turn=1 NC0,CR1 Turn=1 L0,NC1 Turn=1 NC0,NC1 Turn=1 NC0,L1 Turn=1 L0,L1

44 Always if Turn=0 then at some point Turn=1 Turn=0 L0,L1 Turn=0 L0,NC1 Turn=0 NC0,L1 Turn=0 CR0,NC1 Turn=0 NC0,NC1 Turn=0 CR0,L1 Turn=1 L0,CR1 Turn=1 NC0,CR1 Turn=1 L0,NC1 Turn=1 NC0,NC1 Turn=1 NC0,L1 Turn=1 L0,L1

45 Interleaving semantics: Execute one transition at a time. Turn=0 L0,L1 Turn=0 L0,NC1 Turn=0 CR0,NC1 Turn=0 NC0,NC1 Turn=1 L0,CR1 Turn=1 L0,NC1 Need to check the property for every possible interleaving!

46 Interleaving semantics Turn=0 L0,L1 Turn=0 L0,NC1 Turn=0 CR0,NC1 Turn=0 NC0,NC1 Turn=1 L0,CR1 Turn=1 L0,NC1 Turn=0 L0,L1 Turn=0 L0,NC1

47 L0:While True do NC0:wait(Turn=0); CR0:Turn=1 endwhile || L1:While True do NC1:wait(Turn=1); CR1:Turn=0 endwhile T0:PC0=L0 PC0:=NC0 T1:PC0=NC0/\Turn=0 PC0:=CR0 T1:PC0=NC0/\Turn=1 PC0:=NC0 T2:PC0=CR0 (PC0,Turn):=(L0,1) T3:PC1==L1 PC1=NC1 T4:PC1=NC1/\Turn=1 PC1:=CR1 T4:PC1=NC1/\Turn=0 PC1:=NC1 T5:PC1=CR1 (PC1,Turn):=(L1,0) Initially: PC0=L0/\PC1=L1 Busy waiting

48 Always when Turn=0 then at some point Turn=1 Now it does not hold! (Red subgraph generates a counterexample execution.) Turn=0 L0,L1 Turn=0 L0,NC1 Turn=0 NC0,L1 Turn=0 CR0,NC1 Turn=0 NC0,NC1 Turn=0 CR0,L1 Turn=1 L0,CR1 Turn=1 NC0,CR1 Turn=1 L0,NC1 Turn=1 NC0,NC1 Turn=1 NC0,L1 Turn=1 L0,L1 T4 T1

49 Combinatorial explosion V 1 :=1 V 1 :=3 V 1 :=2 V n :=1 V n :=3 V n :=2 … How many states?

50 Global states 3 n states v 1 =1,v 2 =1…v n =1 v 1 =2,v 2 =1…v n =1v 1 =1,v 2 =1…v n =2 … v 1 =3,v 2 =1…v n =1 … … v 1 =1,v 2 =1…v n =3

51 Program verification: flowchart programs (Book: chapter 7)

52 History Verification of flowchart programs: Floyd, 1967 Hoares logic: Hoare, 1969 Linear Temporal Logic: Pnueli, Krueger, 1977 Model Checking: Clarke & Emerson, 1981

53 Program Verification Predicate (first order) logic. Partial correctness, Total correctness Flowchart programs Invariants, annotated programs Well founded ordering (for termination) Hoares logic

54 Predicate (first order logic) Variables, functions, predicates Terms Formulas (assertions)

55 Signature Variables: v1, x, y18 Each variable represents a value of some given domain (int, real, string, …). Function symbols: f(_,_), g2(_), h(_,_,_). Each function has an arity (number of paramenters), a domain for each parameter, and a range. f:int*int->int (e.g., addition), g:real->real (e.g., square root) A constant is a predicate with arity 0. Relation symbols: R(_,_), Q(_). Each relation has an arity, and a domain for each parameter. R : real*real (e.g., greater than). Q : int (e.g., is a prime).

56 Terms Terms are objects that have values. Each variable is a term. Applying a function with arity n to n terms results in a new term. Examples: v1, 5.0, f(v1,5.0), g2(f(v1,5.0)) More familiar notation: sqr(v1+5.0)

57 Formulas Applying predicates to terms results in a formula. R(v1,5.0), Q(x) More familiar notation: v1>5.0 One can combine formulas with the boolean operators (and, or, not, implies). R(v1,5.0)->Q(x) x>1 -> x*x>x One can apply existentail and universal quantification to formulas. x Q(X) x1 R(x1,5.0) x y R(x,y)

58 A model, A proofs A model gives a meaning (semantics) to a first order formula: A relation for each relation symbol. A function for each function symbol. A value for each variable. An important concept in first order logic is that of a proof. We assume the ability to prove that a formula holds for a given model. Example proof rule (MP) :

59 Flowchart programs Input variables: X=x1,x2,…,xl Program variables: Y=y1,y2,…,ym Output variables: Z=z1,z2,…,zn start halt Y=f(X) Z=h(X,Y)

60 Assignments and tests Y=g(X,Y)t(X,Y) FT

61 start halt (y1,y2)=(0,x1) y2>=x2 (y1,y2)=(y1+1,y2-x2)(z1,z2)=(y1,y2) Initial condition Initial condition: the values for the input variables for which the program must work. x1>=0 /\ x2>0 F T

62 start halt (y1,y2)=(0,x1) y2>=x2 (y1,y2)=(y1+1,y2-x2)(z1,z2)=(y1,y2) The input-output claim The relation between the values of the input and the output variables at termination. x1=z1*x2+z2 /\ 0<=z2 { "@context": "http://schema.org", "@type": "ImageObject", "contentUrl": "http://images.slideplayer.com/799594/3/slides/slide_61.jpg", "name": "start halt (y1,y2)=(0,x1) y2>=x2 (y1,y2)=(y1+1,y2-x2)(z1,z2)=(y1,y2) The input-output claim The relation between the values of the input and the output variables at termination.", "description": "x1=z1*x2+z2 /\ 0<=z2

63 Partial correctness, Termination, Total correctness Partial correctness: if the initial condition holds and the program terminates then the input-output claim holds. Termination: if the initial condition holds, the program terminates. Total correctness: if the initial condition holds, the program terminates and the input-output claim holds.

64 Subtle point: The program is partially correct with respect to x1>=0/\x2>=0 and totally correct with respect to x1>=0/\x2>0 start halt (y1,y2)=(0,x1) y2>=x2 (y1,y2)=(y1+1,y2-x2)(z1,z2)=(y1,y2) T F

65 start halt (y1,y2)=(0,x1) y2>=x2 (y1,y2)=(y1+1,y2-x2)(z1,z2)=(y1,y2) Annotating a scheme Assign an assertion for each pair of nodes. The assertion expresses the relation between the variable when the program counter is located between these nodes. A B CD E FT

66 Invariants Invariants are assertions that hold at each state throughout the execution of the program. One can attach an assertion to a particular location in the code: e.g., at(B) (B). This is also an invariant; in other locations, at(B) does not hold hence the implication holds. If there is an assertion attached to each location, (A), (B), (C), (D), (E), then their disjunction is also an invariant: (A)\/ (B)\/ (C)\/ (D)\/ (E) (since location is always at one of these locations).

67 Annotating a scheme with invariants A): x1>=0 /\ x2>=0 B): x1=y1*x2+y2 /\ y2>=0 C): x1=y1*x2+y2 /\ y2>=0 /\ y2>=x2 D):x1=y1*x2+y2 /\ y2>=0 /\ y2=x2 (y1,y2)=(y1+1,y2-x2)(z1,z2)=(y1,y2) A B CD E F T A) Is the precondition of (y1,y2)=(0,x1) and B) is its postcondition

68 Preliminary: Relativizing assertions (B) : x1= y1 * x2 + y2 /\ y2 >= 0 Relativize B) w.r.t. the assignment, obtaining B) [Y\g(X,Y)] e (B) expressed w.r.t. variables at A.) (B) A = x1=0 * x2 + x1 /\ x1>=0 Think about two sets of variables, before={x, y, z, …} after={x,y,z…}. Rewrite (B) using after, and the assignment as a relation between the set of variables. Then eliminate after by substitution. Here: x1=y1 * x2 + y2 /\ y2>=0 /\ x1=x1 /\ x2=x2 /\ y1=0 /\ y2=x1 now eliminate x1, x2, y1, y2. (y1,y2)=(0,x1) A B A B Y=g(X,Y)

69 Preliminary: Relativizing assertions (y1,y2)=(0,x1) A B A B A): (B) A (B) Y=g(X,Y)

70 Verification conditions: assignment A) B) A where B) A = B)[Y\g(X,Y)] A): x1>=0 /\ x2>=0 B): x1=y1*x2+y2 /\ y2>=0 B) A = x1=0*x2+x1 /\ x1>=0 (y1,y2)=(0,x1) A B A B Y=g(X,Y)

71 (y1,y2)=(y1+1,y2-x2) Second assignment C): x1=y1*x2+y2 /\ y2>=0 /\ y2>=x2 B): x1=y1*x2+y2 /\ y2>=0 B) C : x1=(y1+1)*x2+y2- x2 /\ y2-x2>=0 C B

72 (z1,z2)=(y1,y2) Third assignment D):x1=y1*x2+y2 /\ y2>=0 /\ y2 { "@context": "http://schema.org", "@type": "ImageObject", "contentUrl": "http://images.slideplayer.com/799594/3/slides/slide_71.jpg", "name": "(z1,z2)=(y1,y2) Third assignment D):x1=y1*x2+y2 /\ y2>=0 /\ y2=0 /\ y2

73 Verification conditions: tests B) /\ t(X,Y) C) B) /\¬t(X,Y) D) B): x1=y1*x2+y2 /\y2>=0 C): x1=y1*x2+y2 /\ y2>=0 /\ y2>=x2 D):x1=y1*x2+y2 /\ y2>=0 /\ y2=x2 B C D B C D t(X,Y) F T FT

74 Verification conditions: tests y2>=x2 B C D B C D t(X,Y) F T FT ¬t(X,Y) B) C)

75 Partial correctness proof: An induction on length of execution B) D) C) Initially, states satisfy the initial conditions. Then, passing from one set of states to another, we preserve the invariants at the appropriate location. We prove: starting with a state satisfying the initial conditions, if are at a point in the execution, the invariant there holds. Not a proof of termination! start halt (y1,y2)=(0,x1) y2>=x2 (y1,y2)=(y1+1,y2-x2)(z1,z2)=(y1,y2) A B CD E A) no yes TF

76 Exercise: prove partial correctness Initial condition: x>=0 Input-output claim: z=x! start halt (y1,y2)=(0,1) y1=x (y1,y2)=(y1+1,(y1+1)*y2)z=y2 TF

77 What have we achieved? For each statement S that appears between points X and Y we showed that if the control is in X when (X) holds (the precondition of S) and S is executed, then (Y) (the postcondition of S) holds. Initially, we know that (A) holds. The above two conditions can be combined into an induction on the number of statements that were executed: If after n steps we are at point X, then (X) holds.

78 Another example (A) : x>=0 (F) : z^2<=x<(z+1)^2 z is the biggest number that is not greater than sqrt x. start (y1,y2,y3)=(0,0,1) A halt y2>x (y1,y3)=(y1+1,y3+2)z=y1 B C D F truefalse E y2=y2+y3

79 Some insight 1+3+5+…+(2n+1)=(n+1)^2 y2 accumulates the above sum, until it is bigger than x. y3 ranges over odd numbers 1,3,5,… y1 is n-1. start (y1,y2,y3)=(0,0,1) A halt y2>x (y1,y3)=(y1+1,y3+2)z=y1 B C D F truefalse E y2=y2+y3

80 Invariants It is sufficient to have one invariant for every loop (cycle in the programs graph). We will have (C)=y1^2<=x /\ y2=(y1+1)^2 /\ y3=2*y1+1 start (y1,y2,y3)=(0,0,1) A halt y2>x (y1,y3)=(y1+1,y3+2)z=y1 B C D F truefalse E y2=y2+y3

81 Obtaining (B) By backwards substitution in (C). (C)=y1^2<=x /\ y2=(y1+1)^2 /\ y3=2*y1+1 (B)=y1^2<=x /\ y2+y3=(y1+1)^2 /\ y3=2*y1+1 start (y1,y2,y3)=(0,0,1) A halt y2>x (y1,y3)=(y1+1,y3+2)z=y1 B C D F truefalse E y2=y2+y3

82 Check assignment condition (A)=x>=0 (B)=y1^2<=x /\ y2+y3=(y1+1)^2 /\ y3=2*y1+1 (B) relativized is 0^2<=x /\ 0+1=(0+1)^2 /\ 1=2*0+1 Simplified: x>=0 start (y1,y2,y3)=(0,0,1) A halt y2>x (y1,y3)=(y1+1,y3+2)z=y1 B C D F truefalse E y2=y2+y3

83 Obtaining (D) By backwards substitution in (B). (B)=y1^2<=x /\ y2+y3=(y1+1)^2 /\ y3=2*y1+1 (D)=(y1+1)^2<=x /\ y2+y3+2=(y1+2)^2 /\ y3+2=2*(y1+1)+1 start (y1,y2,y3)=(0,0,1) A halt y2>x (y1,y3)=(y1+1,y3+2)z=y1 B C D F truefalse E y2=y2+y3

84 Checking (C)=y1^2<=x /\ y2=(y1+1)^2 /\ y3=2*y1+1 (C)/\y2<=x) (D) (D)=(y1+1)^2<=x /\ y2+y3+2=(y1+2)^2 /\ y3+2=2*(y1+1)+1 start (y1,y2,y3)=(0,0,1) A halt y2>x (y1,y3)=(y1+1,y3+2)z=y1 B C D F truefalse E y2=y2+y3

85 y1^2<=x /\ y2=(y1+1)^2 /\ y3=2*y1+1 /\y2<=x (y1+1)^2<=x /\ y2+y3+2=(y1+2)^2 /\ y3+2=2*(y1+1)+1 y1^2<=x /\ y2=(y1+1)^2 /\ y3=2*y1+1 /\y2<=x (y1+1)^2<=x /\ y2+y3+2=(y1+2)^2 /\ y3+2=2*(y1+1)+1 y1^2<=x /\ y2=(y1+1)^2 /\ y3=2*y1+1 /\y2<=x (y1+1)^2<=x /\ y2+y3+2=(y1+2)^2 /\ y3+2=2*(y1+1)+1

86 Not finished! Still needs to: Calculate (E) by substituting backwards from (F). Check that (C)/\y2>x (E) start (y1,y2,y3)=(0,0,1) A halt y2>x (y1,y3)=(y1+1,y3+2)z=y1 B C D F truefalse E y2=y2+y3

87 Exercise: prove partial correctness. Initially: x1>0/\x2>0. At termination: z1=gcd(x1,x2). halt start (y1,y2)=(x1,x2) z1=y1 y1=y2 FT y1>y2 y2=y2-y1 y1=y1-y2 TF

88 Annotation of program with invariants halt start (y1,y2)=(x1,x2) z1=y1 y1=y2 F T y1>y2 y2=y2-y1 y1=y1-y2 TF z1=gcd(x1,x2) x1>0 /\ x2>0 gcd(y1,y2)=gcd(x1,x2) /\y1>0/\y2>0 gcd(y1,y2)=gcd(x1,x2) /\y1>0/\y2>0/\y1 y2 gcd(y1,y2)=gcd(x1,x2)/\ y1>0/\y2>0/\y10/\y2>0/\y1>y2 y1=gcd(x1,x2) A B C D E F G H

89 Part 1 halt start (y1,y2)=(x1,x2) z1=y1 y1=y2 F T y1>y2 y2=y2-y1 y1=y1-y2 TF (A)= x1>0 /\ x2>0 (B)=gcd(y1,y2)=gcd(x1,x2) /\y1>0/\y2>0 A B C D E F G H (B)rel= gcd(x1,x2)=gcd(x1,x2)/\ x1>0/\x2>0 (A) (B)rel

90 Part 2a halt start (y1,y2)=(x1,x2) z1=y1 y1=y2 F T y1>y2 y2=y2-y1 y1=y1-y2 TF (B)= gcd(y1,y2)=gcd(x1,x2) /\y1>0/\y2>0 (C)=gcd(y1,y2)=gcd(x1,x2) /\y1>0/\y2>0/\y1 y2 A B C D E F G H (B)/\ ¬(y1=y2) (C)

91 Part 2b halt start (y1,y2)=(x1,x2) z1=y1 y1=y2 F T y1>y2 y2=y2-y1 y1=y1-y2 TF (G)= y1=gcd(x1,x2) A B C D E F G H (B)= gcd(y1,y2)=gcd(x1,x2) /\y1>0/\y2>0 (B)/\ (y1=y2) (G)

92 Part 3 halt start (y1,y2)=(x1,x2) z1=y1 y1=y2 F T y1>y2 y2=y2-y1 y1=y1-y2 TF (D)= gcd(y1,y2)=gcd(x1,x2) /\y1>0/\y2>0/\y1 y2 (E)=gcd(y1,y2)=gcd(x1,x2) /\y1>0/\y2>0/\y10/\y2>0/\y1>y2 A B C D E F G H (D)/\ (y1>y2) (F) (D)/\ ¬(y1>y2) (E)

93 Part 4 halt start (y1,y2)=(x1,x2) z1=y1 y1=y2 F T y1>y2 y2=y2-y1 y1=y1-y2 TF x1>0 /\ x2>0 (B)= gcd(y1,y2)=gcd(x1,x2) /\y1>0/\y2>0 (E)= gcd(y1,y2)=gcd(x1,x2)/\ y1>0/\y2>0/\y10/\y2>0/\y1>y2 A B C D E F G H (B)rel1= gcd(y1,y2-y1)=gcd(x1,x2) /\y1>0/\y2-y1>0 (B)rel2= gcd(y1-y2,y2)=gcd(x1,x2) /\y1-y2>0/\y2>0 (E) (B)rel1 (F) (B)rel2

94 Part 5 halt start (y1,y2)=(x1,x2) z1=y1 y1=y2 F T y1>y2 y2=y2-y1 y1=y1-y2 TF (H)= z1=gcd(x1,x2) (G)= y1=gcd(x1,x2) A B C D E F G H (H)rel= y1=gcd(x1,x2) (G) (H)rel2

95 Proving termination

96 Well-founded sets Partially ordered set (W,<): If aa2>a3>…

97 Examples for well founded sets Natural numbers with the bigger than relation. Finite sets with the set inclusion relation. Strings with the substring relation. Tuples with alphabetic order: (a1,b1)>(a2,b2) iff a1>a2 or [a1=a2 and b1>b2]. (a1,b1,c1)>(a2,b2,c2) iff a1>a2 or [a1=a2 and b1>b2] or [a1=a2 and b1=b2 and c1>c2].

98 Why does the program terminate y2 starts as x1. Each time the loop is executed, y2 is decremented. y2 is natural number The loop cannot be entered again when y2=x2 C true

99 Proving termination Choose a well-founded set (W,<). Attach a function u(N) to each point N. Annotate the flowchart with invariants, and prove their consistency conditions. Prove that (N) (u(N) in W).

100 How not to stay in a loop? Show that u(M)>=u(N)rel. At least once in each loop, show that u(M)>u(N). S M N T N M

101 How not to stay in a loop? For stmt: (M) (u(M)>=u(N)rel) Relativize since we need to compare values not syntactic expressions. For test (true side): ( (M)/\test) (u(M)>=u(N)) For test (false side): ( (M)/\¬test) (u(M)>=u(L)) stmt M N test N M true L false

102 What did we achieve? There are finitely many control points. The value of the function u cannot increase. If we return to the same control point, the value of u must decrease (its a loop!). The value of u can decrease only a finite number of times.

103 Why does the program terminate u(A)=x1 u(B)=y2 u(C)=y2 u(D)=y2 u(E)=z2 W: naturals > : greater than start halt (y1,y2)=(y1+1,y2-x2)(z1,z2)=(y1,y2) (y1,y2)=(0,x1) A B D E false y2>=x2 C true

104 Recall partial correctness annotation A): x1>=0 /\ x2>=0 B): x1=y1*x2+y2 /\ y2>=0 C): x1=y1*x2+y2 /\ y2>=0 /\ y2>=x2 D):x1=y1*x2+y2 /\ y2>=0 /\ y2=x2 (y1,y2)=(y1+1,y2-x2)(z1,z2)=(y1,y2) A B CD E false true

105 Strengthen for termination A): x1>=0 /\ x2>0 B): x1=y1*x2+y2 /\ y2>=0/\x2>0 C): x1=y1*x2+y2 /\ y2>=0/\y2>=x2/\x2>0 D):x1=y1*x2+y2 /\ y2>=0 /\ y2 0 E):x1=z1*x2+z2 /\ 0<=z2=x2 (y1,y2)=(y1+1,y2-x2)(z1,z2)=(y1,y2) A B CD E falsetrue

106 Strengthen for termination A): x1>=0 /\ x2>0 u(A)>=0 B): x1=y1*x2+y2 /\ y2>=0/\x2>0 u(B)>=0 C): x1=y1*x2+y2 /\y2>=0 /\y2>=x2/\x2>0 u(c)>=0 D):x1=y1*x2+y2 /\ y2>=0 /\ y2 0 u(D)>=0 E):x1=z1*x2+z2 /\ 0 =0 This proves that u(M) is natural for each point M. u(A)=x1 u(B)=y2 u(C)=y2 u(D)=y2 u(E)=z2

107 We shall show: u(A)=x1 u(B)=y2 u(C)=y2 u(D)=y2 u(E)=z2 A) u(A)>=u(B)rel B) u(B)>=u(C) C) u(C)>u(B)rel B) u(B)>=u(D) D) u(D)>=u(E)rel start halt (y1,y2)=(y1+1,y2-x2)(z1,z2)=(y1,y2) (y1,y2)=(0,x1) A B D E false y2>=x2 C true

108 Proving decrement C): x1=y1*x2+y2 /\ y2>=0 /\ y2>=x2/\x2>0 u(C)=y2 u(B)=y2 u(B)rel=y2-x2 C) y2>y2-x2 (notice that C) x2>0) start halt (y1,y2)=(y1+1,y2-x2)(z1,z2)=(y1,y2) (y1,y2)=(0,x1) A B D E false y2>=x2 C true

109 Integer square prog. (C)=y1^2<=x /\ y2=(y1+1)^2 /\ y3=2*y1+1 (B)=y1^2<=x /\ y2+y3=(y1+1)^2 /\y3=2*y1+1 start (y1,y2,y3)=(0,0,1) A halt y2>x (y1,y3)=(y1+1,y3+2) z=y1 B C D F truefalse E y2=y2+y3

110 u(A)=x+1 u(B)=x-y2+1 u(C)=max(0,x-y2+1) u(D)=x-y2+1 u(E)=u(F)=0 u(A)>=u(B)rel u(B)>u(C)rel u(C)>=u(D) u(C)>=u(E) u(D)>=u(B)rel Need some invariants, i.e., y2 0 at points B and D, and y3>0 at point C. start (y1,y2,y3)=(0,0,1) A halt y2>x (y1,y3)=(y1+1,y3+2) z=y1 B C D F truefalse E y2=y2+y3

111

112 Why testing? Reduce design/programming errors. Can be done during development, before production/marketing. Practical, simple to do. Check the real thing, not a model. Scales up reasonably. Being state of the practice for decades.

113 Part 1: Testing of black box finite state machine Know: Transition relation Size or bound on size Wants to know: In what state we started? In what state we are? Transition relation Conformance Satisfaction of a temporal property

114 Finite automata (Mealy machines) S - finite set of states. (size n) - set of inputs. (size d) O - set of outputs, for each transition. (s 0 S - initial state). δ S S - transition relation (deterministic but sometimes not defined for each input per each state). S O - output on edges. Notation: δ(s,a 1 a 2..a n )= δ(… (δ(δ(s,a 1 ),a 2 ) … ),a n ) (s,a 1 a 2..a n )= (s,a 1 ) (δ(s,a 1 ),a 2 )… (δ(… δ(δ(s,a 1 ),a 2 ) … ),a n )

115 Finite automata (Mealy machines) S - finite set of states. (size n) – set of inputs. (size d) O – set of outputs, for each transition. (s 0 S - initial state). δ S S - transition relation. S O – output on edge. S={s 1, s 2, s 3 }, {a, b }, O={0,1 }. δ(s 1,a)=s 3 (also s 1 =a=>s 3 ), δ(s 1,b)=s 2,(also s 1 =b=>s 2 )… (s 1,a)=0, (s 1,b)=1,… δ(s 1,ab)=s 1, (s 1,ab)=01 s1s1 s3s3 s2s2 a/0 b/1 b/0 b/1 a/0

116 Why deterministic machines? Otherwise no amount of experiments would guarantee anything. If dependent on some parameter (e.g., temperature), we can determinize, by taking parameter as additional input. We still can model concurrent system. It means just that the transitions are deterministic. All kinds of equivalences are unified into language equivalence. Also: connected machine (otherwise we may never get to the completely separate parts).

117 Determinism When the black box is nondeterministic, we might never test some choices. b/1 a/1

118 Preliminaries: separating sequences s1s1 s3s3 s2s2 a/0 b/1 b/0 b/1 a/0 Start with one block containing all states {s 1, s 2, s 3 }. A set of sequences X such that if we execute them from different states, at least one of them will give a different output sequence. st µ X (s, µ ) (s, µ )

119 A: separate to blocks of states with different output. s1s1 s3s3 s2s2 a/0 b/1 b/0 b/1 a/0 The states are separated into {s 1, s 3 }, {s 2 } using the string b. But s 1 and s 3 have the same outputs to same inputs.

120 Repeat B : Separate blocks based on input that cause moving to different blocks. s1s1 s3s3 s2s2 a/0 b/1 b/0 b/1 a/0 Separate first block using b to three singleton blocks obtaining separation sequence bb. Separating sequences: b, bb. Max rounds: n-1, sequences: n-1, length: n-1. If string X separated blocks B 1, B 2, and letter a splits part of block C to move to B 1 and part to B 2, then aX separates C into C 1, C 2, accordingly.

121 Want to know the state of the machine (at end): Homing sequence. Know: transition relation. Dont know: which state we start. Want to know: which state we end up. After firing homing sequence, we know in what state we are by observing the outputs. Find a sequence µ such that δ(s, µ )δ(t, µ ) (s, µ ) (t, µ ) So, given an input µ that is executed from state s, we look at a table of outputs and according to a table, know in which state r we are.

122 Homing sequence Algorithm: Put all the states in one block (initially we do not know what is the current state). Then repeatedly separate blocks, as long as they are not singletons, as follows: Take a non singleton block, append a distinguishing sequence µ that separates at least two states in that block. Update each block to the states after executing µ. (Blocks may not be disjoint, its not a partition!) Note: blocks do not have to be disjoint! Max length of sequence: (n-1) 2 (Lower bound: n(n-1)/2.)

123 Example (homing sequence) s1s1 s3s3 s2s2 a/0 b/1 b/0 b/1 a/0 {s 1, s 2, s 3 } {s 1, s 2 } {s 3 } {s 1 } {s 2 } {s 3 } b b 1 0 011 1 On input b and output 1, we still dont know if we were in s 1 or s 3, i.e., if we are currently in s 2 or s 1. So separate these cases with another b.

124 Synchronizing sequence One sequence takes the machine to the same final state, regardless of the initial state or the outputs. That is: find a sequence µ such that for each states s, t, δ(s, µ )=δ(t, µ ) Not every machine has a synchronizing sequence. Can be checked whether exists and if so, can be found in polynomial time. a/1 b/1 a/0 b/0b/1 a/0

125 Algorithm for synchronizing sequeneces s1s1 s2s2 s3s3 a/1 b/1 a/0 b/0b/1 a/0 Construct a graph with ordered pairs of nodes (s,t) such that (s,t )=a=>(s,t ) when s=a=>s, t=a=>t. (Ignore self loops, e.g., on (s 2,s 2 ).) s 1,s 1 s 2,s 2 s 3,s 3 s 1,s 2 s 2,s 3 s 1,s 3 bb b b b b a a a

126 Algorithm continues (2) There is an input sequence from both st to some r iff there is a path in this graph from (s,t ) to (r,r ). There is a synchronization sequence iff some twins node (r,r ) is reachable from every pair of distinct nodes. In this case it is (s 2,s 2 ). s 1,s 1 s 2,s 2 s 3,s 3 s 1,s 2 s 2,s 3 s 1,s 3 b b b b b b a a a If original automaton is not strongly connected, there is no synchronizing sequence. Otherwise, all twins are reachable from each other in product graph. Thus, in this case, if there is not twins node (r,r) reachable from all pairs, there is some pair (S i,S j ) from which no twins node is reachable, thus no sequence will synchronize S i and S j.

127 Algorithm continues (3) Notation:δ(S,x)= {t | s S,δ(s,x)=t } 1. i=1; S 1 =S 2. Take some nodes st S i, and find a shortest path labeled x i to some twins (r,r ). 3. i=i+1; S i :=δ(S i-1,x ). If |S i |>1,goto 2., else goto 4. 4. Concatenate x 1 x 2 …x k. s 1,s 1 s 2,s 2 s 3,s 3 s 1,s 2 s 2,s 3 s 1,s 3 b b b b b b a a a Number of sequences n-1. Each of length n(n-1)/2. Overall O(n(n-1) 2 /2).

128 Example: (s 2,s 3 )=a=>(s 2,s 2 ) x 1 :=a δ({s 1,s 2,s 3 },a)={s 1,s 2 } (s 1,s 2 )=ba=>(s 2,s 2 ) x 2 :=ba δ({s 1,s 2 },ba)={s 2 } So x 1 x 2 =aba is a synchronization sequence, bringing every state into state s 2. s 1,s 1 s 2,s 2 s 3,s 3 s 1,s 2 s 2,s 3 s 1,s 3 b b b b b b a a a

129 State identification: Want to know in which state the system has started (was reset to). Can be a preset distinguishing sequence (fixed), or a tree (adaptive). May not exist (PSPACE complete to check if preset exists, polynomial for adaptive). Best known algorithm: exponential length for preset, polynomial for adaptive [LY].

130 Sometimes cannot identify initial state b/1 a/1 s1s1 s3s3 s2s2 b/0 b/1 a/1 Start with a: in case of being in s 1 or s 3 well move to s 1 and cannot distinguish. Start with b: In case of being in s 1 or s 2 well move to s 2 and cannot distinguish. The kind of experiment we do affects what we can distinguish. Much like the Heisenberg principle in Physics.

131 So… Well assume resets from now on!

132 Conformance testing Unknown deterministic finite state system B. Known: n states and alphabet. An abstract model C of B. C satisfies all the properties we want from B. C has m states. Check conformance of B and C. Another version: only a bound n on the number of states l is known.

133 Check conformance with a given state machine Black box machine has no more states than specification machine (errors are mistakes in outputs, mistargeted edges). Specification machine is reduced, connected, deterministic. Machine resets reliably to a single initial state (or use homing sequence). s1s1 s3s3 s2s2 a/1 b/0 b/1 a/1 ?=?= b/1

134 Conformance testing [Ch,V] a/1 b/1 Cannot distinguish if reduced or not. a/1 b/1 a/1 b/1 a/1 b/1 a/1 b/1

135 Conformance testing (cont.) a bb a a a a b b b a Need: bound on number of states of B. a

136 Preparation: Construct a spanning tree, and separating sequences b/1 a/1 s1s1 s3s3 s2s2 b/0 b/1 a/1 s1s1 s2s2 s3s3 b/1a/1 Given an initial state, we can reach any state of the automaton.

137 How the algorithm works? According to the spanning tree, force a sequence of inputs to go to each state. 1. From each state, perform the distinguishing sequences. 2. From each state, make a single transition, check output, and use distinguishing sequences to check that in correct target state. s1s1 s2s2 s3s3 b/1a/1 Reset Distinguishing sequences

138 Comments 1. Checking the different distinguishing sequences (m-1 of them) means each time resetting and returning to the state under experiment. 2. A reset can be performed to a distinguished state through a homing sequence. Then we can perform a sequence that brings us to the distinguished initial state. 3. Since there are no more than m states, and according to the experiment, no less than m states, there are m states exactly. 4. Isomorphism between the transition relation is found, hence from minimality the two automata recognize the same languages.

139 Combination lock automaton Assume accepting states. Accepts only words with a specific suffix (cdab in the example). s1s1 s2s2 s3s3 s4s4 s5s5 bdca Any other input

140 When only a bound on size of black box is known… Black box can pretend to behave as a specification automaton for a long time, then upon using the right combination, make a mistake. b/1 a/1 s1s1 s3s3 s2s2 b/0 b/1 a/1 b/1 Pretends to be s 3 Pretends to be s 1 a/1

141 Conformance testing algorithm [VC] The worst that can happen is a combination lock automaton that behaves differently only in the last state. The length of it is the difference between the size n of the black box and the specification m. Reach every state on the spanning tree and check every word of length n-m+1 or less. Check that after the combination we are at the state we are supposed to be, using the distinguishing sequences. No need to check transitions: already included in above check. Complexity: m 2 n d n-m+1 Probabilistic complexity: Polynomial. Distinguishing sequences s1s1 s2s2 s3s3 b/1a/1 Words of length n-m+1 Reset

142 Part 2: Software testing (Book: chapter 9) Testing is not about showing that there are no errors in the program. Testing cannot show that the program performs its intended goal correctly. So, what is software testing? Testing is the process of executing the program in order to find errors. A successful test is one that finds an error.

143 Some software testing stages Unit testing – the lowest level, testing some procedures. Integration testing – different pieces of code. System testing – testing a system as a whole. Acceptance testing – performed by the customer. Regression testing – performed after updates. Stress testing – checking the code under extreme conditions. Mutation testing – testing the quality of the test suite.

144 Some drawbacks of testing There are never sufficiently many test cases. Testing does not find all the errors. Testing is not trivial and requires considerable time and effort. Testing is still a largely informal task.

145 Black-Box (data-driven, input-output) testing The testing is not based on the structure of the program (which is unknown). In order to ensure correctness, every possible input needs to be tested - this is impossible! The goal: to maximize the number of errors found.

146 testing Is based on the internal structure of the program. There are several alternative criterions for checking enough paths in the program. Even checking all paths (highly impractical) does not guarantee finding all errors (e.g., missing paths!)

147 Some testing principles A programmer should not test his/her own program. One should test not only that the program does what it is supposed to do, but that it does not do what it is not supposed to. The goal of testing is to find errors, not to show that the program is errorless. No amount of testing can guarantee error-free program. Parts of programs where a lot of errors have already been found are a good place to look for more errors. The goal is not to humiliate the programmer!

148 Inspections and Walkthroughs Manual testing methods. Done by a team of people. Performed at a meeting (brainstorming). Takes 90-120 minutes. Can find 30%-70% of errors.

149 Code Inspection Team of 3-5 people. One is the moderator. He distributes materials and records the errors. The programmer explains the program line by line. Questions are raised. The program is analyzed w.r.t. a checklist of errors.

150 Checklist for inspections Data declaration All variables declared? Default values understood? Arrays and strings initialized? Variables with similar names? Correct initialization? Control flow Each loop terminates? DO/END statements match? Input/output OPEN statements correct? Format specification correct? End-of-file case handled?

151 Walkthrough Team of 3-5 people. Moderator, as before. Secretary, records errors. Tester, play the role of a computer on some test suits on paper and board.

152 Selection of test cases (for white-box testing) The main problem is to select a good coverage criterion. Some options are: Cover all paths of the program. Execute every statement at least once. Each decision has a true or false value at least once. Each condition is taking each truth value at least once. Check all possible combinations of conditions in each decision.

153 Cover all the paths of the program Infeasible. Consider the flow diagram on the left. It corresponds to a loop. The loop body has 5 paths. If the loops executes 20 times there are 5^20 different paths! May also be unbounded!

154 How to cover the executions? IF (A>1) & (B=0) THEN X=X/A; END; IF (A=2) | (X>1) THEN X=X+1; END; Choose values for A,B,X. Value of X may change, depending on A,B. What do we want to cover? Paths? Statements? Conditions?

155 Statement coverage Execute every statement at least once By choosing A=2,B=0,X=3 each statement will be chosen. The case where the tests fail is not checked! IF (A>1) & (B=0) THEN X=X/A; END; IF (A=2) | (X>1) THEN X=X+1; END; Now x=1.5

156 Decision coverage (if, while checks) Each decision has a true and false outcome at least once. Can be achieved using A=3,B=0,X=3 A=2,B=1,X=1 Problem: Does not test individual conditions. E.g., when X>1 is erroneous in second decision. IF (A>1) & (B=0) THEN X=X/A; END; IF (A=2) | (X>1) THEN X=X+1; END;

157 Decision coverage A=3,B=0,X=3 IF (A>1) & (B=0) THEN X=X/A; END; IF (A=2) | (X>1) THEN X=X+1; END; Now x=1

158 Decision coverage A=2,B=1,X=1 The case where A 1 and the case where x>1 where not checked! IF (A>1) & (B=0) THEN X=X/A; END; IF (A=2)|(X>1) THEN X=X+1; END;

159 Condition coverage Each condition has a true and false value at least once. For example: A=1,B=0,X=3 A=2,B=1,X=0 lets each condition be true and false once. Problem:covers only the path where the first test fails and the second succeeds. (A>1) IF (A>1) & (B=0) THEN X=X/A; END; IF (A=2) | (X>1) THEN X=X+1; END;

160 Condition coverage A=1,B=0,X=3 (A>1) IF (A>1) & (B=0) THEN X=X/A; END; IF (A=2) | (X>1) THEN X=X+1; END;

161 Condition coverage A=2,B=1,X=0 Did not check the first THEN part at all!!! Can use condition+decision coverage. (A>1) IF (A>1) & (B=0) THEN X=X/A; END; IF (A=2) | (X>1) THEN X=X+1; END;

162 Multiple Condition Coverage Test all combinations of all conditions in each test. A>1,B=0 A>1,B0 A 1,B=0 A 1,B0 A=2,X>1 A=2,X 1 A2,X>1 A2,X 1 IF (A>1) & (B=0) THEN X=X/A; END; IF (A=2) | (X>1) THEN X=X+1; END;

163 A smaller number of cases: A=2,B=0,X=4 A=2,B=1,X=1 A=1,B=0,X=2 A=1,B=1,X=1 Note the X=4 in the first case: it is due to the fact that X changes before being used! IF (A>1) & (B=0) THEN X=X/A; END; IF (A=2) | (X>1) THEN X=X+1; END; Further optimization: not all combinations. For C /\ D, check (C, D), ( C, D), (C, D). For C \/ D, check ( C, D), ( C, D), (C, D).

164 Preliminary:Relativizing assertions (Book: Chapter 7) (B) : x1= y1 * x2 + y2 /\ y2 >= 0 Relativize B) w.r.t. the assignment becomes B) [Y\g(X,Y)] e ( B) expressed w.r.t. variables at A.) (B) A = x1=0 * x2 + x1 /\ x1>=0 Think about two sets of variables, before={x, y, z, …} after={x,y,z…}. Rewrite (B) using after, and the assignment as a relation between the set of variables. Then eliminate after. Here: x1=y1 * x2 + y2 /\ y2>=0 /\ x1=x1 /\ x2=x2 /\ y1=0 /\ y2=x1 now eliminate x1, x2, y1, y2. (y1,y2)=(0,x1) A B A B Y=g(X,Y)

165 Verification conditions: tests C) B)= t(X,Y) /\ C) D) B)= t(X,Y) /\ D) B)= D) /\ y2 x2 y2>=x2 B C D B C D t(X,Y) F T FT

166 How to find values for coverage? Put true at end of path. Propagate path backwards. On assignment, relativize expression. On yes edge of decision, add decision as conjunction. On no edge, add negation of decision as conjunction. Can be more specific when calculating condition with multiple condition coverage. A>1 & B=0 A=2 | X>1 X=X+1 X=X/A no yes true

167 How to find values for coverage? A>1 & B=0 A=2 | X>1 X=X+1 X=X/A no yes true A2 /\ X>1 (A2 /\ X/A>1) /\ (A>1 & B=0) A2 /\ X/A>1 Need to find a satisfying assignment: A=3, X=6, B=0 Can also calculate path condition forwards.

168 How to cover a flow chart? Cover all nodes, e.g., using search strategies: DFS, BFS. Cover all paths (usually impractical). Cover each adjacent sequence of N nodes. Probabilistic testing. Using random number generator simulation. Based on typical use. Chinese Postman: minimize edge traversal Find minimal number of times time to travel each edge using linear programming or dataflow algorithms.

169 Test cases based on data-flow analysis Partition the program into pieces of code with a single entry/exit point. For each piece find which variables are set/used/tested. Various covering criteria: from each set to each use/test From each set to some use/test. X:=3 z:=z+x x>y t>y

170 Test case design for black box testing Equivalence partition Boundary value analysis Cause-effect graphs

171 Equivalence partition Goals: Find a small number of test cases. Cover as much possibilities as you can. Try to group together inputs for which the program is likely to behave the same. Specification condition Valid equivalence class Invalid equivalence class

172 Example: A legal variable Begins with A-Z Contains [A-Z0-9] Has 1-6 characters. Specification condition Valid equivalence class Invalid equivalence class Starting char Chars Length Starts A-ZStarts other [A-Z0-9]Has others 1-6 chars0 chars, >6 chars 12 34 5 67

173 Equivalence partition (cont.) Add a new test case until all valid equivalence classes have been covered. A test case can cover multiple such classes. Add a new test case until all invalid equivalence class have been covered. Each test case can cover only one such class. Specification condition Valid equivalence class Invalid equivalence class

174 Example AB36P (1,3,5) 1XY12 (2) A17#%X (4) Specification condition Valid equivalence class Invalid equivalence class Starting char Chars Length Starts A-ZStarts other [A-Z0-9]Has others 1-6 chars0 chars, >6 chars 12 34 5 67 (6) VERYLONG (7)

175 Boundary value analysis In every element class, select values that are closed to the boundary. If input is within range -1.0 to +1.0, select values -1.001, -1.0, -0.999, 0.999, 1.0, 1.001. If needs to read N data elements, check with N-1, N, N+1. Also, check with N=0.

176 For Unit Testing Write drivers, which replaces procedures calling the code. Write stubs, which replaces procedures called by the code. Tested unit Driver Stub Driver

177 package bucket; import java.util.*; import java.math.*; public class BugBucketSort { public static void main(String[] args) { Random r = new Random(100); r.nextInt(100); LinkedList ls = new LinkedList (); for (int j = 0; j < 10; j++) ls.push(r.nextInt(100)); System.out.println(ls); bucketSort(ls); } public static void bucketSort(LinkedList ls) { LinkedList > la = new LinkedList >(); for (int i = 0 ; i<11; i++)//BUG should be 10 instead of 11 la.add(new HashMap ()); System.out.println("\nValues in the array: "+la.size()); for (int j = 0; j < ls.size(); j++) { la.get((int)ls.get(j)/10).put((int)ls.get(j)%10, ls.get(j)); } System.out.println("\nThe ordered array is:"); System.out.print("["); for (int i = 0; i < la.size(); i++) for (int k = 1; k < 10; k++)//BUG needs to start with k=0 { if(la.get(i).containsKey(k)) System.out.print(la.get(i).get(k)+" "); } System.out.println("]"); } Bucket sort: With bugs!!

178 package bucket; import java.util.*; import java.math.*; public class BucketSort { public static void main(String[] args) { Random r = new Random(100); r.nextInt(100); LinkedList ls = new LinkedList (); for (int j = 0; j < 10; j++) ls.push(r.nextInt(100)); System.out.println(ls); // System.out.println((int) 17 / 5); bucketSort(ls); } public static void bucketSort(LinkedList ls) { LinkedList > la = new LinkedList >(); for (int i = 0 ; i<10; i++) la.add(new HashMap ()); System.out.println("\nValues in the array: "+la.size()); for (int j = 0; j < ls.size(); j++) { la.get((int)ls.get(j)/10).put((int)ls.get(j)%10, ls.get(j)); } System.out.println("\nThe ordered array is:"); System.out.print("["); for (int i = 0; i < la.size(); i++) for (int k = 0; k < 10; k++) { if(la.get(i).containsKey(k)) System.out.print(la.get(i).get(k)+" "); } System.out.println("]"); }} Bucket sort: Without bugs!!

179 class MergeSortAlgorithm { public static int [] mergeSort(int array[]) { int loop; int rightIndex = 0; int [] leftArr = new int[array.length / 2]; int [] rightArr = new int[(array.length - array.length) / 2]; if (array.length <= 1) return array; for (loop = 0 ; loop < leftArr.length ; ++loop) leftArr[loop] = array[loop]; for (; rightIndex < rightArr.length ; ++loop) rightArr[rightIndex++] = array[loop]; mergeSort(leftArr); rightArr = mergeSort(rightArr); array = merge(leftArr,rightArr); return array; } public static int [] merge(int [] leftArr, int [] rightArr) { int [] newArray = new int[leftArr.length + rightArr.length]; int leftIndex = 0; int rightIndex = 0; int newArrayIndex = 0; while (leftIndex < leftArr.length && rightIndex < rightArr.length) { if (leftArr[leftIndex] < rightArr[rightIndex]) newArray[newArrayIndex] = leftArr[leftIndex++]; newArray[newArrayIndex] = rightArr[rightIndex++];} while (rightIndex < rightArr.length) newArray[newArrayIndex++] = rightArr[rightIndex++]; while (leftIndex < leftArr.length) newArray[newArrayIndex++] = leftArr[leftIndex++]; return newArray;} public static void print(int [] arr, String caller) { int loop; for (loop = 0 ; loop < arr.length ; ++loop) System.out.println("caller " + caller + " : " + arr[loop]);} public static void main(String [] args) { int []originalArray = {1,5,4,3,7}; print(originalArray,"before"); originalArray = mergeSort(originalArray); print(originalArray,"after"); }}

180 class MergeSortAlgorithm { public static int [] mergeSort(int array[]) { int loop; int rightIndex = 0; int [] leftArr = new int[array.length / 2]; int [] rightArr = new int[(array.length - array.length) / 2]; // BUG if (array.length <= 1) return array; for (loop = 0 ; loop < leftArr.length ; ++loop) leftArr[loop] = array[loop]; for (; rightIndex < rightArr.length ; ++loop) rightArr[rightIndex++] = array[loop]; mergeSort(leftArr); // BUG rightArr = mergeSort(rightArr); array = merge(leftArr,rightArr); return array; } public static int [] merge(int [] leftArr, int [] rightArr) { int [] newArray = new int[leftArr.length + rightArr.length]; int leftIndex = 0; int rightIndex = 0; int newArrayIndex = 0; while (leftIndex < leftArr.length && rightIndex < rightArr.length) { if (leftArr[leftIndex] < rightArr[rightIndex]) newArray[newArrayIndex] = leftArr[leftIndex++]; newArray[newArrayIndex] = rightArr[rightIndex++]; // TWO BUGS} while (rightIndex < rightArr.length) newArray[newArrayIndex++] = rightArr[rightIndex++]; while (leftIndex < leftArr.length) newArray[newArrayIndex++] = leftArr[leftIndex++]; return newArray;} public static void print(int [] arr, String caller) { int loop; for (loop = 0 ; loop < arr.length ; ++loop) System.out.println("caller " + caller + " : " + arr[loop]);} public static void main(String [] args) { int []originalArray = {1,5,4,3,7}; print(originalArray,"before"); originalArray = mergeSort(originalArray); print(originalArray,"after"); }}


Download ppt "Softrare Reliability Methods Prof. Doron A. Peled Bar Ilan University, Israel And Univeristy of warwick, UK Version 2008."

Similar presentations


Ads by Google