1. Texas A&M University Department of Computer Science Sameh S. Sharkawi
Model Based Testing
Texas A&M University
Department of Computer Science
Sameh S. Sharkawi

2. Outline Introduction How MBT Works Benefits of MBT Problems of MBT
What is MBT
Motivation behind MBT
How MBT Works
Benefits of MBT
Problems of MBT
Current State of MBT in Industry
Software Tools for MBT
Conclusion
References

3. Introduction (What is MBT)
Definition
“Model-based testing refers to software testing where test cases are derived in whole or in part from a model that describes some (usually functional) aspects of the system under test (SUT) .” *
* Wikipedia Definition of Model Based Testing.

4. Introduction (What is MBT Contd.)
MBT is an approach in which you define the behavior of a system in terms of actions that change the state of the system.
Such a model of the system results in a well-defined Finite State Machine (FSM) which helps us to understand and predict the system’s behavior

5. Introduction (Motivation behind MBT)
Testing is expensive
30-50% of development costs
Poor testing is VERY expensive
Down time
Maintenance costs
Rework
Law suits
Model Based Testing promises
Increased effectiveness of testing
Similar or decreased costs
Reuse of design artifacts

6. Introduction (Motivation behind MBT Downtime Costs)
Brokerage operations $6,450,000
Credit card authorization $2,600,000
Ebay (1 outage 22 hours) $225,000
Amazon.com $180,000
Package shipping services $150,000
Home shopping channel $113,000
Catalog sales center $90,000
Airline reservation center $89,000
Cellular service activation $41,000
On-line network fees $25,000
ATM service fees $14,000

7. How MBT Works

8. How MBT Works?
The model is usually an abstract, partial presentation of the system under test's desired behavior.
The test cases derived from this model are functional tests on the same level of abstraction as the model.
These test cases are collectively known as the abstract test suite

9. How MBT Works? Contd.
The abstract test suite cannot be directly executed against the system under test because it is on the wrong level of abstraction.
Therefore an executable test suite must be derived from the abstract test suite that can communicate with the system under test.
This is done by mapping the abstract test cases to concrete test cases suitable for execution.
An example of a model-based testing workflow. IXIT refers to "implementation extra information" and denotes here the total package of information that is needed when the abstract test suite is converted into an executable one. Typically it includes information about test harness, data mappings and SUT configuration.

10. How MBT Works? Contd. Deriving Test Cases Algorithmically
Test Case Generation by Theorem Proving
Test Case Generation by Constraint Logic Programming
Test Case Generation by Model Checking
Test Case Generation by Symbolic Execution
The effectiveness of model-based testing is primarily due to the potential for automation it offers. If the model is machine-readable and formal to the extent that it has a well-defined behavioral interpretation, test cases can in principle be derived mechanically.
Often the model is translated to or interpreted as a finite state automaton or a state transition system. This automaton represents the possible configurations of the system under test. To find test cases, the automaton is searched for executable paths. A possible execution path can serve as a test case. This method works if the model is deterministic or can be transformed into a deterministic one.
Depending on the complexity of the system under test and the corresponding model the number of paths can be very large, because of the huge amount of possible configurations of the system. For finding appropriate test cases, i.e. paths that refer to a certain requirement to proof, the search of the paths has to be guided. For the test case selection multiple techniques are applied.

11. How MBT Works? Contd. Model Provides expected results for Generates
Spec Explorer (UW-MSR Summer Institute '04)
Model
Provides expected
results for
Generates
Pass Fail
Test Cases
Test Oracle
Provides
actual results for
Run
Implementation

12. Benefits of MBT

13. Benefits of MBT Starting from specification
Involves testers early in the development process
Teams testers with developers
Forces testability into product design
Building the test interface
Finds design and specification bugs - before code exists
The model is the test plan - and is easily maintained
Automated test suite generation
Coverage is guaranteed - increases testing thoroughness
Zero test suite maintenance costs
Automated test suite execution
Finds code and interface bugs
Includes a framework for the testing of distributed applications
Reduces test execution costs

14. Benefits of MBT Contd. These benefits can be summarized as: Machine
Time

15. Benefits of MBT Contd. Machines A typical test engineer
Earns € ($ USD) per year
Works 40 to 50 hours per week
A typical test machine
Costs €250 ($300 USD) per year to buy and operate
Can work 100 hours per week

16. Benefits of MBT Contd.
react.cs.uni-sb.de/mbt2006/talks/mbt4masses.pdf

17. Problems of MBT

18. Problems of MBT

19. Problems of MBT Contd.

20. Problems of MBT Contd.

21. Problems of MBT Contd.

22. Problems of MBT Contd. Process shift Personnel shift
Up front investment in test
Personnel shift
Higher education and sophistication
Spec Explorer (UW-MSR Summer Institute '04)

23. Current State of MBT in Industry

24. Current State of MBT in Industry
IBM Research
Most projects were done by PhD Holders
Very Little was successful (Just in the Lab, never actually went into market)
Too Complicated to use and analyze
Microsoft Research
The Spec# programming system is a new attempt at a more cost effective way to develop and maintain high-quality software

25. Current State of MBT in Industry
Spec# is a computer language which includes and extends C#
Adds pre/post conditions, contracts
Adds high-level data types with convenient notations
Adds logical quantifiers like FORALL and EXISTS
A Spec# model is just a program!
You can run it like a C# program
It can call framework code
NEW You can explore it (run all of its possible behavior) which is how we do MBT with Spec#
Spec Explorer (UW-MSR Summer Institute '04)

26. Current State of MBT in Industry (Spec Explorer)
A model exploration and testing tool for .NET:
Authoring of models in Word
Model exploration (generating FSM)
FSM visualization
FSM traversal and test suite generation
Automatic implementation binding
Online test-suite execution and offline test-suite code generation
Spec Explorer (UW-MSR Summer Institute '04)

27. Current State of MBT in Industry (Spec #)
Model: Calculator (viewpoint: only starting and stopping, and switching scientific mode)
bool Running = false;
bool Scientific = false;
[Action] void SetRunning(bool newRunning)
requires Running != newRunning; {
Running = newRunning; }
[Action] void SetScientific(bool newScientific)
requires Running;
requires Scientific != newScientific; {
Scientific = newScientific;
System State
Spec#: Pre-Condition describes when action is enabled
State Update
Tells Spec Explorer this is an action. Action invocations do appear in tests.
Spec Explorer (UW-MSR Summer Institute '04)

28. Software Tools for MBT

29. Software Tools for MBT
Conformiq Test Generator is a model-based testing tool using UML state charts that represent testing strategies
Leirios Test Generator is a model-based testing tool that generates tests automatically from deterministic system specifications
Reactis Tester is another model-based testing tool that focuses on control systems
TGV is a tool for the generation of conformance test suites for protocols
TorX is also a prototype testing tool for conformance testing of reactive software
Lurette is an automated testing tool of reactive programs written in Lustre
AsmL Test Tool can generate tests directly from an AsmL model
AutoFocus (in german) is a graphical tool for developing and modeling distributed systems with integrated testing facilities

30. Software Tools for MBT (AsmL)
AsmL is the Abstract State Machine Language. It is an executable specification language based on the theory of Abstract State Machines. The current version, AsmL 2 (AsmL for Microsoft .NET), is embedded into Microsoft Word and Microsoft Visual Studio.NET. It uses XML and Word for literate specifications. It is fully interoperable with other .NET languages. AsmL generates .NET assemblies which can either be executed from the command line, linked with other .NET assemblies, or packaged as COM components.

31. Conclusion

32. Conclusion Modeling is not easy What needs to happen
What details do you leave out?
How do you check the model?
There are no recipes for success
What needs to happen
Easier notation
Incremental approach
Pilots and examples

33. Questions
How do we guarantee that the generated expected outputs and inputs are enough and cover all the cases?
How to make for the errors and exceptions that may occur due to running the program on different environments and different OSs?
How is automated generation of inputs and outputs work when an actual human interaction is required as part of the system testing?
Does MBT cover stress testing?
Does MBT cover how user friendly the system is or it only covers functionality?

34. References

35. References http://en.wikipedia.org/wiki/Model-based_testing
sb.de/mbt2006/talks/ModelBasedTestingSoberEvaluation.pdf
Colin Campbell, Wolfgang Grieskamp, Lev Nachmanson, Wolfram Schulte, Nikolai Tillmann, Margus Veanes “Spec Explorer: An Integrated Environment for Model-Based Testing.” Foundations of Software Engineering Microsoft Research, Redmond