1. Verification and Validation in the Context of Domain-Specific Modelling Janne Merilinna

2. VTT TECHNICAL RESEARCH CENTRE OF FINLAND 2 Conclusion (OOPSLA DSM’09 group work) Sun was shining The beer was good Loud discussions Which was good However To be continued in the next SPLASH (OOPSLA) DSM’10

3. VTT TECHNICAL RESEARCH CENTRE OF FINLAND 3 Overview Definition of Verification and Validation (V&V) What to V&V in the context of DSM V&V adapted into the context of DSM (some) Means to facilitate V&V Conclusions

4. VTT TECHNICAL RESEARCH CENTRE OF FINLAND 4 Definitions 1/2 IEEE Standard Dictionary: Validation ‘the process of evaluating a system or component during or at the end of the development process to determine whether it satisfies specified requirements.’ Model Validation ‘the process of determining the degree to which the requirements, design or implementation of a model are a realization of selected aspects of the system being modelled.’

5. VTT TECHNICAL RESEARCH CENTRE OF FINLAND 5 Definitions 2/2 IEEE Standard Dictionary: Verification ‘(1) the process of evaluating a system or component to determine whether the products of a given development phase satisfy the conditions imposed at the start of that phase. (2) Formal proof of program correctness.’ Model Verification ‘the process of determining the degree of similarity between the realization steps of a model; for example, between the requirements and the design or between the design and its implementation.’

6. VTT TECHNICAL RESEARCH CENTRE OF FINLAND 6 The Process to V&V A process of transforming end-user requirements to a system fulfilling such requirements with a DSM approach The development process includes phases for developing Modelling infrastructure Metamodels, Generators, and Application models which are transformed into the target format running on a software platform.

7. VTT TECHNICAL RESEARCH CENTRE OF FINLAND 7 Verification Adapted to the DSM Context -Metamodel 1/2 The verification of metamodels (MM) means Determining if a metamodel encapsulates the problem domain, and the rules of the target platform that are necessary to take into account in the metamodel

8. VTT TECHNICAL RESEARCH CENTRE OF FINLAND 8 Verifying the domain coverage requires neural processing but guidelines for language development help Verifying that only correctly designed applications can be modelled ‘Correctly designed’ denotes that the modelled applications are syntactically correct and semantically sound in the sense that they appear to be correct from the interpreter perspective, i.e. the application models can be generated into working executables running on a set platform Developing an extensive test suite of application models might help Verification Adapted to the DSM Context -Metamodel 2/2

9. VTT TECHNICAL RESEARCH CENTRE OF FINLAND 9 Verification Adapted to the DSM Context -Model 1/2 The verification of models (M) means Determining whether a model encapsulates a product under development based on the requirements set for it

10. VTT TECHNICAL RESEARCH CENTRE OF FINLAND 10 Verification Adapted to the DSM Context -Model 2/2 The verification of models means scrutinizing whether the requirements set for an application should be possible to be covered by a model simply by considering the model itself Maintaining a traceability-link between the requirements and models help Analysing the models with advanced analysis tool might also help Transforming a model to a format understood by analysis tools is a risk!

11. VTT TECHNICAL RESEARCH CENTRE OF FINLAND 11 Verification Adapted to the DSM Context -Generator 1/3 The verification of generators (G) means Determining whether the generators produce correct output (GTR) from the input models

12. VTT TECHNICAL RESEARCH CENTRE OF FINLAND 12 Verification Adapted to the DSM Context -Generator 2/3 The verification of code generators [Kuster et al.] The syntactic correctness of a model transformation: in order to ensure that the outcome of the transformation is syntactically correct The termination and confluence of a model transformation: in order to ensure that a model always produces the same outcome The safety and liveness properties: in order to verify that e.g. specific structural properties are preserved during the transformation. The semantic equivalence of a model transformation: in order to ensure that the semantics are preserved during the model transformation

13. VTT TECHNICAL RESEARCH CENTRE OF FINLAND 13 Verification Adapted to the DSM Context -Generator 3/3 Syntax and format in general Metamodel coverage of the generator should be checked Semantics The semantics between the source model and target model has to be preserved during the transformation A third metamodel to connect the both worlds? -> Verification done when verification of GTR is considered?

14. VTT TECHNICAL RESEARCH CENTRE OF FINLAND 14 Verification Adapted to the DSM Context -GTR 1/2 The verification of generated textual representation (GTR) means Evaluating whether or not the intention of a model is realised in the outcome, i.e. in the GTR produced by a generator

15. VTT TECHNICAL RESEARCH CENTRE OF FINLAND 15 Verification Adapted to the DSM Context -GTR 2/2 Notice: The intention of the model should be considered to be a fact, thus the model must be considered to be correct from the modeller’s perspective in order to make any sense in the GTR verification Maintaining traceability-link between GTR and models should help to verify that the intention is realised Testing should not be overlooked although testing cannot formally state that the GTR always functions as planned

16. VTT TECHNICAL RESEARCH CENTRE OF FINLAND 16 Validation Adapted to the DSM Context 1/2 The validation is the process of evaluating the GTR, including the target platform, to determine whether it satisfies the specified requirements It must be noted that one application model can only be seen as one validation case for the whole modelling infrastructure Multiple application models are required!

17. VTT TECHNICAL RESEARCH CENTRE OF FINLAND 17 Validation Adapted to the DSM Context 2/2 We see validation as the challenge of maintaining a bidirectional traceability link between the end-user requirements and the generated deliverable Maintaining a bidirectional traceability link between requirements and GTR should help

18. VTT TECHNICAL RESEARCH CENTRE OF FINLAND 18 There are methods, techniques and some tools for verification and validation No comprehensive methodology backed up with tools still exists especially intended for DSM And then what? Methods and tools for the language developer Metamodel Method to check what kinds of erroneous models can be modelled? Code generator Method to check what parts of the metamodel the code generator can reach? Method and tools for the language user Bi-directional traceability link From requirements databases (DOORS, OSRMT) to domain-specific models to source code and back Comprehensive test-suite generation from application models Conclusions