1. 25/06/2015Marius Mikucionis, AAU SSE1/22 Principles and Methods of Testing Finite State Machines – A Survey David Lee, Senior Member, IEEE and Mihalis Yannakakis invited paper Part about Conformance Testing

2. 25/06/2015Marius Mikucionis, AAU SSE2/22 Overview Main concepts: FSM, sequences, equivalence Conformance testing problem and assumptions Separating family of sequences Status messages and reset Distinguishing sequences Identifying sequences Polynomial time randomised algorithm Heuristic procedures and optimisations Extensions: one machine, more states, partially specified

3. 25/06/2015Marius Mikucionis, AAU SSE3/22 Concepts: FSM Finite State Machine (Mealy machine): Is a tuple: M=(I, O, S, , ): –I, O – input and output symbol sets (alphabets), –S – state set, –  : S  I  S – transition function, – : S  I  O – output function. Example: –When the machine is in state s  S and receives input symbol a  I it moves to state  (s, a) and produces output symbol (s, a).

4. 25/06/2015Marius Mikucionis, AAU SSE4/22 Concepts: sequences and state equivalence We extend transition and output functions: –Let x be an input string: x=a 1,…,a k, a i  I, for i=1..k –then  (s 1, x)=s k+1 where s i+1 =  (s i, a i ), for i=1..k –and (s 1, x)=b 1 …b n where b i = (s i, x), for i=1..k x – is a sequence and b 1 …b n is a response to it. Equivalent s i, s j  S   x  I * : (s i, x) = (s j, x) If s i and s j are not equivalent then: –  separating sequence x  I * : (s i, x)  (s j, x)

5. 25/06/2015Marius Mikucionis, AAU SSE5/22 Machine equivalence Equivalent A, B  FSM  –  s A  S A  s B  S B : s B equivalent to s A –  s B  S B  s A  S A : s A equivalent to s B There are many equivalent machines Each equivalent machine class contains minimized machine M with minimum number of states (all states are unique). Minimized machine is unique.

6. 25/06/2015Marius Mikucionis, AAU SSE6/22 Conformance testing problem Given: –Complete information of specification machine A (states, transition and output function) –Implementation machine B, black box, only I/O is observable Goal: –Determine whether B is correct implementation of (conforms to, is equivalent to) A by applying a test sequence to B and observing the output. Checking sequence for machine A with n states is an input sequence x that distinguishes A from any non-equivalent machine B with n states: – (s B, x)  (s A, x)  B is not equivalent to A

7. 25/06/2015Marius Mikucionis, AAU SSE7/22 Conformance testing assumption Specification A is strongly connected –It must be possible to reach all states A is reduced (minimized) –We can determine equivalence only to minimized machine, since equivalent states are not distinguishable. B does not change during experiment and has the same input alphabet as A B has no more states than A –Assume, faults do not increase number of states, only: Wrong output on transition Wrong state in transition destination

8. 25/06/2015Marius Mikucionis, AAU SSE8/22 Conformance test structure Algorithm structure: –Initialization: move to some known state s 1 : If s 1 is given  verify it (not always possible) Else apply homing sequence that takes to some known state s 1 (possible for minimised machines) –Verify similarity of B to A –Verify each transition  (s i, a)=s j : Apply sequence that moves machine to s i Apply a Verify that machine is in s j Methods: –Status and Reset messages –Distinguishing sequences –Identifying sequences –Randomised sequences

9. 25/06/2015Marius Mikucionis, AAU SSE9/22 Separating family of sequences Separating family of sequences for A is a collection of n sets  i (one set for each state): –For every pair of states s i, s j (s i  s j ) there is sequence  : 1) (s i,  )  (s j,  ) 2)  is a prefix in some x i  i and a prefix in some x j  j  i is called a separating set of state s i Elements of  i are separating sequences of state s i Examples: –A has a preset distinguishing sequence x then all sets  i may be equal to {x}. –UIO sequences fulfil 1) but may violate 2).

10. 25/06/2015Marius Mikucionis, AAU SSE10/22 Separating family properties General construction algorithm for reduced FSMs: –If A is reduced   s i,s j  x – separating sequence –Partition states into blocks based on (s k, x): For every s k put x into Z k –Repeat procedure for each block until all blocks become singletons. Every pair of states has a separating sequence with common prefix. Z i contains <n-1 sequences of length  n. Key property: only one state at most gives the same response to all elements of particular  i

11. 25/06/2015Marius Mikucionis, AAU SSE11/22 State and machine similarity State similarity: –q i  S B is similar to s i  S A  (q i, x)= (s i, x)  x  i Because of key property: q i can be similar to at most one state of A Machine similarity: –B is similar to A   s i  S A  q i  S B q i is similar to s i All q i are distinct and B has at most n states  one- to-one correspondence between A and B states.

12. 25/06/2015Marius Mikucionis, AAU SSE12/22 Status messages and Reset FSM has a reset capability if special input r takes the machine from any state to initial s 1. If r input is defined for all states of B then reset is reliable. Status message tells the current state of machine without changing it. Reliable status message guarantees that state will stay the same as before message. Unreliable status messages must be applied twice when the state is expected to change. s status/[state] s1s1 r/

13. 25/06/2015Marius Mikucionis, AAU SSE13/22 Test with reliable reset Let  i be a family of separating sets Build a spanning tree with states in nodes from diagram of machine A B similarity to A check: –For every s i  S A for every x  Z i : –Reset B to state s 1 by applying r –Move to state s i according to tree path from s 1 to s i –Apply x  we are sure that B moved to a state similar to s i –  we are sure that B has states similar to A ones,  B has no more states than A has,  B is similar to A

14. 25/06/2015Marius Mikucionis, AAU SSE14/22 Test with reliable reset (2) Check all other transitions  (s i, a)=s j : –For every x  Z j : Reset to s 1 by r Move to state s i according to tree path from s 1 to s i Apply a Apply x –  we are sure that transition  (s i, a)=s j is OK  B is isomorphic to A

15. 25/06/2015Marius Mikucionis, AAU SSE15/22 Test with distinguishing sequences Let A have an adaptive distinguishing sequence, then Z i ={x i }, where x i is a path in decision tree from root to state s i  (s i, s j ) is a pre-computed transition sequence that takes machine from state s i to state s j t i =  (s i, x i ) – some state after separation Similarity test: –x 1  (t 1, s 2 )x 2  (t 2, s 3 )x 3 …x n  (t n, s 1 )x 1 –  we have visited all n states and observed all distinguishing responses to distinguishing sequences Transition  (s i, a)=s j test when in state t k : –  (t k, s i-1 )x i-1  (t i-1, s i ) a x j

16. 25/06/2015Marius Mikucionis, AAU SSE16/22 Test with identifying sequences Problem: to verify state s i similarity we need to apply separating sequence |Z i | times precisely on the same state. Example: –Separating family: Z i ={a, b} –Machine is in state s 1 –Apply “a  a  a  ”, observe “000” –B was in q 0, moved to q 1, q 2, q 3 –However, at least two of states are the same –  q 3 =q i for some i=0..2  we have already applied a on q 3  let’s apply b on it! –  we have successfully applied a and b on the same state,  B has state q 3 similar to state s 1 in A. 1 23 a/0 a/1 b/1 b/0

17. 25/06/2015Marius Mikucionis, AAU SSE17/22 Test with identifying sequences (2) A has n states Similarity to s i with separating set Z i ={z 1, z 2 }: –Let q r =  (s i, z 1  (t i, s i )) r ), where t i =  (s i, z 1 ) –Apply q n –Then  r<n : q r = q n, i.e. we have already applied z 1 successfully on q n –Apply z 2 –  sequence (z 1  (t i, s i )) n z 2 identifies a state in B which is similar to s i Transition check: reuse reliable reset idea, because identifying sequences actually reset the state.

18. 25/06/2015Marius Mikucionis, AAU SSE18/22 Polynomial time randomized test Similarity: –For i=1,…,n: Repeat k i times: –Apply sequence that takes to s i from current state –Choose a separating sequence z  Z i uniformly at random –Apply z Let x be random input formed sequence from similarity test with –k i =O(n |Z i | min(p, |Z i |) log n) Then it can be shown that B is similar to A with high probability

19. 25/06/2015Marius Mikucionis, AAU SSE19/22 Heuristic procedures and optimizations Checking sequences guarantee complete fault coverage, but sometimes they are too long. Success example: circuit testing is based on faults model significantly limit possible faults. Covering paths: –Transition checking with UIO sequences leads to Postman Tour Problem  NP-hard. –Random walk may be “trapped” if system has “narrow passages” or has just few faults  exponential. –Guided random walks records partial history and makes random choice based on priorities from history. –Test sequences from combinations of sub-sequences: overlap in sub-sequences.

20. 25/06/2015Marius Mikucionis, AAU SSE20/22 Summary of algorithms Complexity is very sensitive to preliminary conditions and requirements to the system: p=|I| – number of inputs, n=|S| - number of states SequencesLengthTime Reliable reset pn 3 Distinguishing pn 3 IdentifyingExponential RandomizedPolynomial

21. 25/06/2015Marius Mikucionis, AAU SSE21/22 Conformance test extensions One black-box only: –No difference for deterministic test algorithms –Randomized algorithms are more effective on a single fixed machine: fault probability is squared while test length is doubled More states than in specification: –Unknown states and transitions have “combination lock” property that require exponentially long sequences to cover. Partially specified machines: –Special treatment for undefined transitions may transform machine to fully specified machines. –One fault - exponentially many machines –Only randomised checking has polynomial complexity

22. 25/06/2015Marius Mikucionis, AAU SSE22/22 Evaluation and critics Valuable digest of completely solved problems. Good introduction to partially solved ones. Easy to read: concepts are presented following the pattern: motivation, definition and simple example. Most of algorithms are “encoded” in complex sentences instead of nested blocks – gives motivation, but looses clarity.

23. 25/06/2015Marius Mikucionis, AAU SSE23/22 Thank you for your attention! Questions?

24. 25/06/2015Marius Mikucionis, AAU SSE24/22 Interesting proposition Let A and B satisfy the assumptions, then the following are equivalent: 1)A and B are isomorphic (there exists a bijection between equivalent A and B states) 2)A and B are equivalent 3)At least one state of A has an equivalent state in B