1. 10/05/2007 1 Automated Software Test Generation: Model-Based Testing By Nedret ÖZAY Cmpe 516- Fault Tolerant Computing

2. 2 10/05/2007 Outline Introduction Model-Based Testing(MBT)  Models in SW Testing  Finite State Machines Fundamental Tasks of MBT  Choosing the Model  Building the Model  Generating Tests  Running Tests  Collecting Results Key Concerns and Closing Thoughts

3. 3 10/05/2007 What are the problems of Software Testing? Time is limited Applications are complex Requirements are fluid

4. 4 10/05/2007 What is wrong with manual testing? Testers are often perceived as bottlenecks to the delivery of software products. They are being asked to test more and more code in less and less time.

5. 5 10/05/2007 What is Test Automation? Test automation is the use of software to control the execution of tests, the comparison of actual outcomes to predicted outcomes, the setting up of test preconditions, and other test control and test reporting functions. Commonly, test automation involves automating a manual process already in place that uses a formalized testing process.

6. 6 10/05/2007 Model-Based Testing One way to generate test cases automatically is “model-based testing” where a model of the system is used for test case generation.

7. 7 10/05/2007 Model-Based Testing “Model-based testing is a testing technique where the runtime behavior of an implementation under test is checked against predictions made by a formal specification, or model.”- Colin Campbell, Microsoft Research In other words:  A model describes how a system should behave in response to an action.  Supply the action and see if the system responds as you expect.

8. 8 10/05/2007 Models Simply, a model of software is a depiction of its behavior. Behavior can be described in terms of the  input sequences accepted by the system,  the actions,  conditions,  output logic, or the flow of data through the application’s modules and routines. By using modeling a shareable, reusable, precise description of the system is acquired.

9. 9 10/05/2007 Models in SW Testing Finite State Machines Statecharts UML Markov Chains Grammars

10. 10 10/05/2007 Finite State Machines Formally a finite state machine representing a software system is defined as a quintuple (I, S, T, F, L), where I is the set of inputs of the system. S is the set of all states of the system. T is a function that determines whether a transition occurs when an input is applied to the system in a particular state. F is the set of final states the system can end up in when it terminates. L is the state into which the software is launched.

11. 11 10/05/2007 Example: Light Switch Consider a simple light switch simulator: The lights can be turned on and off using one input. The intensity of the light can be adjusted using two inputs for decreasing and increasing the intensity. There are three levels of light intensity: dim, normal, and bright. If the lights are bright, increasing the intensity should not affect the intensity. The simulator starts with the lights off. Finally, when the lights are turned on, the intensity is normal by default, regardless of the intensity of the light when it was last turned off.

12. 12 10/05/2007 Example: Light Switch “Light Switch” = (I, S, T, F, L), where: I = {,,, } S = {[off], [dim], [normal], [bright]} T: changes [off] to [normal] changes any of [dim], [normal], or [bright] to [off] changes [dim] to [normal], or [normal] to [bright] changes [bright] to [normal], or [normal] to [dim] The inputs do not affect the state of the system under any condition not described above F = [off] L = [off]

13. 13 10/05/2007 State Transition Matrix And State Transition Diagram State \ Input [off][normal] [dim][off][normal] [off][bright][dim] [bright][off][normal]

14. 14 10/05/2007 Example: Notepad

15. 15 10/05/2007 Fundamental tasks of MBT Understanding The System: Forming a mental representation of the system’s functionality is a prerequisite to building models. This is a nontrivial task as most systems today typically complex functionality. Moreover, software is deployed within gigantic operating systems among a clutter of other applications, dynamically linked libraries, and file systems all interacting with and/or affecting it in some manner.  Determine the components/features that need to be tested based on test objectives.  Start exploring target areas in the system.  Establish communication with requirements, design, and development teams if possible.  Identify the users of the system.  Enumerate the inputs and outputs of each user.  Study the domains of each input.  Document input applicability information.

16. 16 10/05/2007  Document conditions under which responses occur.  Study the sequences of inputs that need to be modeled.  Understand internal data interactions and computation.  Maintain one living document. Choosing the Model: Not surprisingly, there are no software models today that fit all intents and purposes. Consequently, for each situation decisions need to be made as to what model (or collection of models) are most suitable.  HTML files for a web browser or mathematical expressions for a scientific calculator: Grammar  Phone systems etc.: Finite State Machine  Parallel Systems: Statecharts (each component is a FSM)  Systems that have intention of statistical analysis of failure data: Markov Chain Fundamental tasks of MBT(cont’d)

17. 17 10/05/2007 Fundamental tasks of MBT(cont’d) Building the Model: The general procedure that underlies many MBT methodologies:  Make a list of the inputs.  input applicability constraints for each input. For example, consider a simple telephone. The input “take the handset off the hook” can only be applied when the handset is hung up. (It is impossible to pick up the handset if the handset has already been picked up.)  input behavior constraints for each input according to the current state context. For example, the input “take the handset of the hook” can cause a number of behaviors. If the context is that the phone is idle, the resulting behavior will be that the system will generate a dial tone. If the context is that the phone is ringing due to an incoming call, then the system will connect the two phones to allow conversation. Two different behaviors are caused by the same input.

18. 18 10/05/2007 Generating Tests A state machine (left) and a test path in the state machine (right) In the case of finite state machines, traverse the state transition diagram. The sequences of arc labels along the generated paths are, by definition, tests. For example, in the state transition diagram below, the sequence of inputs “a, b, d, e, f, i, j, k” qualifies as a test of the represented system.

19. 19 10/05/2007 Running Tests First, test scripts are written to simulate the application of inputs. Next, the test suite can be easily translated into an executable test script.

20. 20 10/05/2007 Collecting Results Evaluating test results is perhaps the most difficult testing process. Testers must determine whether the software generated the correct output given the sequence of test inputs applied to it. In practice, this means verifying screen output, verifying the values of internally stored data and establishing that time and space requirements were met. Verification of state can be used. For example, the model itself will catalog each state change that occurs (or should occur) in the software as test inputs are being applied. Thus, the model acts as a very precise and detailed specification, informing the tester what inputs should be available and the values of each data abstraction that comprises the state of the software under test.

21. 21 10/05/2007 Collecting Results (cont’d)

22. 22 10/05/2007 Key Concerns State Space Explosion: Number of states can be increased in a dramatical manner. Solutions:  Abstraction: Abstraction for MBT involves the incorporation of complex data into a simple structure. For example a dialog box that requires the entry of multiple fields of data followed by the OK button, could be modeled as two abstract inputs: “enter valid data” and “enter invalid data.”  Exclusion: Simply dropping information without bothering to abstract it. This is mostly done when decomposing a set of software behaviors into multiple models. Each model will contain certain inputs and exclude others. The inputs excluded by a modeler would have to be tested in some other way, obviously.

23. 23 10/05/2007 Key Concerns (cont’d) Meeting Coverage Criteria For state machines, coverage is expressed in terms of inputs, states, and transitions. For example, there is the criterion of covering all states.

24. 24 10/05/2007 Attractive Attributes of MBT Succesfull applications when MBT was employed:  Rosaria and Robinson (2000) on testing graphical user interfaces;  Agrawal and Whittaker (1993) on testing embedded controller software;  Avritzer and Larson (1993) and Dalal et al. (1998) on testing phone systems. A model serves as a unifying point of reference that all teams and individuals involved in the development process can share, reuse, and benefit from. For example, confusion as to whether the system under test needs to satisfy a particular requirement can be resolved by examining the model. Most popular models have a rich theoretical background that makes numerous tasks such as generating large suites of test cases easy to automate. For example, graph theory readily solves automated test generation for finite state machine models.

25. 25 10/05/2007 Difficulties and Drawbacks of MBT Skills, time, and other resources need to be allocated for making preparations, overcoming common difficulties, and working around the major drawbacks. This overhead needs to be weighed against potential rewards in order to determine whether a model-based technique is sensible to the task at hand. Certain skills of testers (basic familiarity with formal languages, automata theory, and perhaps graph theory and elementary statistics). The most prominent problem for state models (and most other similar models) is state space explosion. Briefly, models of almost any non-trivial software functionality can grow beyond management even with tool support.

26. 26 10/05/2007 Questions ?

27. 27 10/05/2007 References http://en.wikipedia.org/wiki/Test_automation http://en.wikipedia.org/wiki/Model-based_testing http://www.model-based-testing.org http:// Ibrahim K. El-Far and James A. Whittaker, “Model-based Software Testing”, Encyclopedia on Software Engineering (edited by J.J. Marciniak), Wiley, 2001 L. Apfelbaum and J. Doyle, “Model-Based Testing”. Software Quality Week Conference. May 1997 H. Robinson “MBT Tutorial”, StarWest 2006 Alan Hartman IBM Israel – Haifa Research Laboratory, “Ten Years of Model Based Testing A Sober Evaluation”, MBT 2006