1. Methodological Issues in Model-Based Testing (MBT)

2. Related definitions
What is MBT?
What about handling the real-world complexity in models?
What about generating the production code from models?

3. Definitions
Test Case – structure of input and expected ouput behaviours
Deterministic systems – finite sequence
Non-deterministic systems – tree-like structure
Deterministic
Output and next state are unambigously determined by the current state and input
Non-deterministic
Reaction for input is non-deterministic, ie
More than one outgoing transition may be available for some states
Test Suite – set of test cases
Test Purpose – a property that has to be tested. May be expressed formally (statement, branch, path, … coverage) or informally (reference to system requirement or sub-requirement)

4. Model-based testing - components
Results
Model
Test Generator Tool
Test Suite
Test Execution
Tool/
Human Tester
Drivers/
Adapters
SUT
<<Implements>>
Does the implementation conform to specification model?

5. Model-based testing
Based on the specification model of SUT’s behaviour
Therefore model-based testing is Black-Box testing
Specification model
Allows to automatically generate tests
Can be used as the oracle that checks whether the SUT behaves correctly
Theoretically, specification models allow us to generate production code also! (model-based development)

6. Types of testing/test generation
Offline / a priori
Test suite is generated before executing it (offline)
Online / on-the-fly
Each test is generated after executing the previous test during the testing

7. Why bother?! Modeling-the-real-world issue
Model has to be valid. Why build a model, validate it, derive tests, run tests rather than directly validating the SUT?
Model-Based Development issue
Can we use the same model for generating tests and generating production code?
If we can generate production code from the valid model does it eliminates the need for testing because generated code is valid itself?

8. Modeling-the-real-world issue

9. Modeling the real “world”
Model is a simplification of the real world. Models do not reflect all attributes of the real world
Two basic approaches for simplification
Omission of details
Encapsulation of details
We can’t understand and describe the whole “world” at once
We must break the “world” into reasonable pieces

10. Omission of details Abstract models are easier to understand
Stepwise refinement is used
High-level models include fundamental ideas
High-level models are too abstract for testing
Abstract models are detailed, if necessary, and with relevant details for some aspect of the SUT only
Detailed models with respect of requirements to be validated, limited set of requirements at a time
Drivers/adapters insert the implemention-specific information so that tests can be executed against the real SUT

11. Drivers/Adapters
Used for interaction between abstract test cases and the real-world SUT via
Concretization
Abstraction
Abstract TCs
Test Execution
Tool/
Human Tester
Drivers/
Adapters
SUT
Concretization
Concrete results
Abstraction

12. Drivers/Adapters
Test Cases generated from abstract model are too abstract to be fed to the real SUT
Concretization is needed to be able to pass information to the real SUT
Actual results from the real SUT are too detailed to be compared to the expected results from the model
Abstraction is needed to be able to validate real-world concrete results against the abstract model

13. Encapsulation of details
In other words – using references to detailed models of certain aspects, for example
ISO/OSI model, where each layer relies on lower layers
Libraries in programming languages
“Parts” of the real world are put togehter when executing tests

14. Model-Based Development issue

15. Models in SW development
Testing is not the only discipline using models
The main question – can we use the same model for test and production code generation?
Let’s investigate two alternatives
Common model
Separate models

16. The common model approach
Test
Generator
Tool
Suite
Requirements
Code
With common model - the code is tested against itself since the source (model) is the same!
Only Code generator and code-environment interaction can be tested with this Test Suite!

17. The separate models approach
Requirements
Pricipial problem of the common model approach is resolved by different sources
Development Model
Test Model
Development of models and manual development must be carefully synchronized!
This approach is expensive
Code
Test
Suite
Code
Generator
Tool
Test
Generator
Tool

18. Summary Modeling itself helps to reveal defects in logic
In MBT, Models are executable artifacts written in a very high-level programming language
Test code can be generated
Production code can be generated
Models need to be abstract enough to be maintainable – can’t model the whole world at once
Omission of details
Encapsulation of details
Adapters/drivers are responsible for adding/removing Implemetation-specific details for interaction with real-world SUT
Great interest on MBT in embedded systems industry

19. Sources and further reading
Methodological issues in Model-Based testing
Alexander Pretschner
Jan Phillips
MBT Example - Synthesis of Test Purpose Directed Reactive Planning Tester for Non-deterministic Systems
Jüri Vain,
Kullo Raiend,
Andres Kull,
Juhan P. Ernits