1. Integration of Functional and Timed Testing of Real Time and Concurrent Systems
V. Kuliamin, A. Petrenko, N.!Pakoulin, I.!Bourdonov, A.!Kossatchev
ISP RAS, Russia
Hello!
I am Victor Kuliamin from Institute for System Programming (ISP RAS), Moscow, Russia.
My talk is concerned with integration of testing functional and temporal requirements for concurrent and real-time systems.

2. Classic testing scheme
stimulus s1
Test System
Target System r1 s2
reaction r2 s3 r3
Test =
Test sequence s4 r4
The classic scheme of testing based on state-transition model of the target system assumes that the test system provides stimulus to the target system and gets reactions from it.
The test is represented as a sequence of stimulus-reaction pairs. The correctness of the target system is evaluated on the base of reactions obtained, and the completeness of testing is assessed on the base of sequences tried and their relations with all possible sequences. s5 r5

3. Concurrency and real-time issues
stimulus1
Target System
stimulus
stimulus2
reaction1
reaction
reaction3
reaction2
reaction
Multiple access points
Simultaneous inputs
Multiple reactions
Spontaneous reactions
Temporal constraints
stimulus
Concurrent systems possess a number of properties that makes that scheme of testing insufficient.
They may have multiple access points for getting input and providing output.
Multiple access points make possible simultaneous input actions.
Multiple reactions may follow a single stimulus. They may come from one access point or from several ones.
System is active. Reactions may be provided without any previous inputs.
Real-time systems are almost always implemented as systems of concurrently working components, so all the previous issues are relevant for them. One more point should be considered for real-time – temporal constraints.
(In this presentation we do not consider problems specific to hard real-time, because it has additional difficulties for precise detection of conformance to hard temporal requirements and for testing other resources utilization. Our work is relevant mostly to soft real-time systems, which problems are mostly concerned with their concurrent nature and soft temporal constraints.)
Δt <= T0
reaction

4. Outline Theoretical background Practical considerations Case studies
Now we proceed to the theoretical background of our approach to testing concurrent systems.
This background concerns the formal models used for test constructions.

5. Proposed model IOSM – Input/Output State Machine States S Stimuli X
Reactions R
Transitions T
φ : T → S×(X ∪ R ∪ {i})× S ?b !z ?a
Internal transition !y
Spontaneous
reaction !x
We use Input/Output State Machines as a model of system under test.
Input/Output State Machine unlike usual State Machine has transitions labeled by stimuli or reactions separately.
This fact allows to model spontaneous reactions occurring without any input provided.
Multiple reactions following one stimulus can also be modeled. It is important that finite IOSM can describe a behavior that can be described only with infinite SMs.
IOSM also can specify internal actions not corresponding to any externally visible event, by influencing the further behavior of the system.
Multiple reactions

6. Empty stimulus X = I ∪ {e}
Empty stimulus e models the knowledge of system that it has no inputs
Asynchronous State Machine – IOSM with empty stimulus !x !y ?a ?e
Practical considerations lead to additional notion of empty stimulus.
Suppose, we have a system with input queue. This system gets stimuli from the input queue to process them, but may have some specific behavior when the input queue is empty.
Empty stimulus is used to model that the system knows that it has no input.
Input/Output State Machines with empty stimulus are called Asynchronous State Machines.

7. Behavior function B : X* → P (R*) ai ↦ { xi } B : X* → P (R*)!y !x !y ?a ?e ?a
B : X* → P (R*)
ai ↦ { xi }
B : X* → P (R*)
ai ↦ { yjxk…xlyn : k+…+l = i }
I have not said how Asynchronous State Machine behaves. Its behavior can be defined in many different ways depending on how we can resolve the choice between stimulus and reactions in mixed states. We also may consider empty stimulus case separately and make reaction issuing to have greater priority than empty stimulus acceptance.
In general semantics of an Asynchronous State Machine can be described with its behavior function, which maps some sequences of stimuli, including empty stimulus, into sets of reaction sequences.
We can show that Asynchronous State Machines without mixed states having outgoing transitions marked with both stimuli and reactions are enough to represent all possible behaviors.
It can be also show that pure IOSM without empty stimulus are less expressive.
I.Burdonov, A.Kossatchev, V.Kuliamin. Asynchronous State Machines: Classification and Applications to Testing. Proceedings of ISP RAS, Moscow, In Russian

8. Modeling parallel inputs?b
How to model responses
to all possible multistimuli?
?{a,a}
?{b,b}
?{a,b}
?{a,a,b}
?{a,b,b}
Plain concurrency : multistimulus is
equivalent to some sequence of the stimulus
contained
When we try to describe the behavior of the system accepting parallel inputs, we face with the problem of infinity of possible input actions even when the total number of individual stimuli is finite, or even equal to 1.
If we can give in parallel two stimuli a and b,
we can also give two a’s or two b’s.
We may also consider possible triples of stimuli and so forth. How to give a description of system behavior in all these cases that would not require describing separately each situation? The base of such an aspiration is the fact that real systems usually behave according to some short description, do not explicitly stating their properties in all the situations considered.
The base of our approach is the plain concurrency axiom. System satisfying this axiom behaves in response to a tuple of parallel stimuli exactly in the same way as in response to a sequence obtained by some ordering of this tuple.
Not all the systems have this property, but many practically significant ones do. We can also try to construct a plain concurrent model of a system using additional states, internal transitions and transitions accepting empty stimulus.

9. Outline Theoretical background Practical considerations Case studies
UniTesK test development method
Modifications for testing concurrent systems
Case studies
Let us now turn to practical issues concerning testing of concurrent systems.
We consider this problem from the point of implementing such a testing in the context of UniTesK test development technology successfully used in several industrial testing projects.

10. UniTesK test construction
Behavior Model
Testing Model ✕
Target System
UniTesK is model-based test development method. It uses two layers of models for test construction.
First is the behavior model of the system, or formal specifications of its functionality. It describes what the system should do.
Second is the testing model. It models the system in such a way that it is easy to derive tests for it.
Testing model is used construct sequences of test inputs in effective way. That means that, from one side, the sequences constructed should be representative, they should cover as many different variants of system behavior as possible and, from the other side, the number of sequences and their total length should as little as possible.
Behavior model is used to check the correctness of system behavior, which is performed by oracles generated from it. Oracle provides two types of checks – admissibility of the input sequence and correctness of the resulting behavior. If the sequence obtained is inadmissible, it is skipped.
The admissible one is applied to the target system and produces some response, which is checked for conformance with the behavior model. ✕ 
Behavior Correctness
Checking
Test Input Construction

11. UniTesK test architecture
Test sequence construction
Test Engine
Test sequence construction
Test Action Iterator
Test Scenario
Oracle
Specification  ✕
Mediator
This slide presents the architecture of tests used in UniTesK.
The main components of test are test sequence construction mechanism and oracle, which cecks the behavior of the target system.
Test system construction is provided by constructing a set of paths on some state-transition model of the system (testing model). This model is described in test action iterator, and the set of paths (transition tour, for example) is constructed by some general algorithm implemented in test engine component.
To be able to separate oracle interface and target system interface mediator component is introduced. Mediator allows to make oracle from the abstract model and use it for different implementations of the same functionality.
Oracle is generated automatically from the specifications of the system behavior in the form of software contracts – preconditions and postconditions of interface operations and invariants representing data integrity constraints.
Test action iterator is generated from the testing model represented as test scenario. Test scenarios used in UniTesK describe the state-transition model of the target system by defining the state data structure, state construction operation, and iteration of admissible input stimuli. They do not describe the complete transitions, particularly, their ends, which require a lot of work to calculate.
Target system

12. Example of specifications
specification int deposit ( int x ) {
pre
return x >= 0; }
post
return balance + x;
We use extensions of widely used programming languages to write contract specifications.
This is an example of pre- and postcondition for operation that deposit a sum of money equal to its parameter to an account.

13. Example of scenario scenario class AccountTest {
AcountSpecification model;
AbstractState getState()
{ return new IntState( model.balance ); }
scenario deposit()
iterate(int i = 0; i < 5; i++;) model.deposit(i); }
scenario withdraw()
iterate(int i = 0; i < 5; i++;) model.withdraw(i);

14. Test sequence construction
Test Engine
This slide demonstrates the construction of basic test sequence, which makes the structure of the state-transition model used explicit. During testing when test engine finds itself in a state, where it haven’t been yet, it extracts from the test action iterator the first admissible stimulus in this state and then applies it. When it visits some state not the first time, it extracts the next unapplied stimulus. If there are no such stimuli, it uses the information on state-transition system structure to come to the state having unapplied stimuli or to decide that all transitions are covered.
Test Action Iterator

15. Modifications for testing concurrency
Specifications
Asynchronous reactions
Temporal constraints
Testing
Providing parallel inputs
Collecting reactions
Checking the behavior as a whole
The approach presented should be extended to be able to test concurrent and real-time systems.
First, we should add constructs to specify asynchronous reactions and temporal constraints.
Second, we should some how extend testing mechanisms to be able to generate parallel inputs according to some coverage criteria, gather asynchronous reactions, and assess the whole behavior of the system.

16. Specifying asynchronous reactions
specification reaction UDPPacket UDPResponse ( ) {
pre { return !ModelUDPPackets.isEmpty ( ); }
post
return
@ModelUDPPackets.clone().contains(UDPResponse)
&& !ModelUDPPackets.contains(UDPResponse); }
This example demonstrates how the asynchronous reaction can be specified. We also use pre- and postconditions for this purpose.

17. Specifying temporal constraints
specification reaction UDPPacket UDPResponse ( ) {
pre { return !ModelUDPPackets.isEmpty ( ); }
post
return
@ModelUDPPackets.clone().contains(UDPResponse)
&& !ModelUDPPackets.contains(UDPResponse)
&& time –
@ModelUDPPackets.clone().time(UDPResponse)
<= 50*numberOfNodes; }
To specify temporal constraints we can use additional construct to refer the time of execution of some target operation. Another way is to attach automatically model timers to events and use their values to check the constraints.

18. Providing parallel inputs
Target System
s24
s23
s22
s21
s33
s32
s31
Multisequence is used instead of sequence of stimuli
To test system behavior in response to parallel input we should generate a sequence of stimuli for each access point available. Each sequence may contain empty stimuli.

19. Collecting asynchronous reactions11 21 31
Target System
r11
r12 12 22 32
r21
r22
r23 23
r31
r32
r33 33
Time
When we collect reactions it is important to know the order of their occurrence. Reactions coming from one access point are naturally ordered.
We may also use timestamps to refine the ordering. Due to different properties of the channels reaction observed later does not always occur later.
Usually reactions, which timestamps differ not greater than some time interval we consider as having no certain order, and if a reaction is observed later than another one by this interval, it is considered to occur later.
So, reactions can be organized in a partially ordered set.
Reactions form a partially ordered set

20. Constructing model action series
Stimuli
Reactions
Plain concurrency axiom       ✕
When we have a number of input sequences and a partially ordered set of reactions, we use plain concurrency axiom to evaluate the observed behavior against the specifications.
Plain concurrency allows us to conclude on the validity of the behavior based only on possible full orderings of the events observed. We construct all the different full orderings an check each event by its oracle.
If some check fails we proceed to another ordering until some is recognized as valid. 

21. Modified test architecture
Test sequence construction
Test Engine ✕ 
Test Action Iterator
Synchronization Manager
Serializer
Oracle
Mediator
For concurrency testing the test architecture is modified in he following way.
First, the system is tested in sequential way and its state-transition model is extracted.
Then, a special test engine produce multiple sequences of admissible inputs begin to work.
These sequences are delivered to synchronization manager, which task is to guarantee that the corresponding inputs are actually parallel.
Oracle component is not used now. The input is transformed from model representation to the implementation one by mediators and corresponding inputs are applied in parallel with the help of synchronization manager.
The reactions of the system are collected and delivered to the serializer, which constructs possible full orderings with the help of oracles of individual events.
Target system
Reaction Collector

22. Outline Theoretical background Practical considerations Case studies
Now I proceed to case studies where our approach was applied to testing industrial software.

23. Case studies Nortel Networks SOS – 1994-1996
about 30 serious bugs found
MSR IPv –
several serious bugs found
MS Mobile IPv –
At the first time some elements of the technique presented was applied to test switch operating system of Nortel Networks.
The result was about 30 serious bugs found, which cause cold restart of the system. This was quite a surprise, because SOS was used by that time about 10 years and was considered as actually reliable system.
The second significant application was testing of Microsoft Research implementation of IPv6 protocol. We took part in this work along with several other teams from all over the world, but, as afar as I know, only our group managed to discover serious bugs that allow to shutdown remotely any node in IPv6 network.
Then, the same technique was applied to test Microsoft implementation of IPv6 for mobile computers. The tests were extended to test the features related to mobility. The project is considered as successful, although no serious bugs were found.

24. References http://www.ispras.ru/groups/rv/rv.html
I. Burdonov, A. Kossatchev, V. Kuliamin. Asynchronous State Machines: Classification and Applications to Testing. Proceedings of ISP RAS, Moscow, In Russian
V. Kuliamin, A. Petrenko, I. Bourdonov, and A. Kossatchev. UniTesK Test Suite Architecture. Proceedings of FME LNCS 2391, pp.!77-88, Springer-Verlag, 2002.
A. Petrenko. Specification-based Testing: Towards Practice. Proceedings of PSI’01, LNCS, Springer-Verlag. v. 2244, pp
V. Kuliamin, A. Petrenko, I. Bourdonov, A. Demakov, A. Jarov, A. Kossatchev, S.!Zelenov. Java Specification Extension for Automated Test Development. Proceedings of PSI'01. LNCS, Springer-Verlag. v. 2244, pp
A. Petrenko, I. Bourdonov, A. Kossatchev, and V. Kuliamin. Experiences in using testing tools and technology in real-life applications. Proceedings of SETT'01, India, Pune, 2001.

25. Thank you!