Model Based Testing implementing with tools Ruud van Houwelingen 1 December 2, 2009

2 Agenda Introduction Model Based Testing – Definition Tool Overview Implementation Conclusions

3 Introduction Who am I Why This Presentation? Updated Version of This Presentation on:

4 Model Based Testing – Definition Model-Based Testing is the automatic generation of efficient test procedures/vectors using models of system requirements and specified functionality. Model-based testing is software testing in which test cases are derived in whole or in part from a model that describes some (usually functional) aspects of the system under test (SUT). (Wikipedia) Model-based testing is the automation of black-box test design. A modelbased testing tool uses various test generation algorithms and strategies to generate tests from a behavioral model of the SUT. (Utting / Legeard)

5 Why Use MBT? Raise Quality Complex systems (System Interaction) ‘Exploding’ number of Test cases (> 100,000) Testing cannot keep up with the speed of Software Development or Deployment High Maintainability of Test Cases Test automation Possible Reduce Testing Time Reduce Costs

6 Automation of the Test Process ExecutionEvaluation

7 Model Based Testing Process Model TestManager Test scenarios SUT test execution test evaluation Pass / Fail Adapter test generation Report

8 ClassicalMBT create tests execute tests evaluate outcome result100 tests in 2 weeks specification modeling tests in 1 night Model Based Testing Process

9 Advantages / Challenges Advantages Automatically generate test cases in a short period of time Experiment with test cases No ambivalent test base Well defined coverage on the specification level Maintainability of the model Test scripts in formal notation (for test execution tools) Validation through simulation Challenges Selection of test cases to be generated (and oracles as well!!) (Skills for) making/defining the model The building of the model itself (abstraction level) Collecting the requirements for the model

10 What is the position of MBT today? Where does MBT fit in the “V-model” What is the relation between the models for development and testing?

11 MBT and the V-model CUSTOMER NEEDS & BUSINESS CASE REQUIREMENTS & CONSTRAINTS FUNCTIONAL DESIGN TECHNICAL DESIGN DEVELOPMENT & CONFIGURATION COMPONENT/UNIT TESTING USER ACCEPTANCE TESTING IMPLEMENTATION FUNCTIONAL ACCEPTANCE SUPPLIERS SYSTEM TESTING BUSINESS PURPOSE NEEDS & LIMITATION SOLUTION DEFINITION CUSTOMER SYSTEM INTEGRATION TESTING

Product requirements Process requirements & design Syst. Req. & design `Build Unit Test System tests Acceptance tests Deployment Operational time MBT and the V-model

13 Tools What kind of Tools Making the model Test Generation Tools Test Execution What can be automated by which tool? Rational / Borland (Micro Focus)  SmarTesting  QTP Conformiq  QTP Testmanager (Axini) Interoperabiliy needs to be worked on

14 Generating Test Cases How can Risk Based Testing be applied? Do we test all possibilities? Do we make a representative subset? What about the Efficiency of the Generator?

15 Immigration and Naturalisation Service We want to test more thorough but thorough testing costs too much time/effort Classical test automation is not sufficient Research University Twente, Enschede, Radbout, Nijmegen Many more…. model-based testing

16

17 INDiGO Architecture ESB BVVBuZaGBA..... Network Partners

18 BeInformed Suite

19 BeInformed Suite II

20 Concerns At this point in the project it is impossible to test all modeled laws, rules and regulations; Because of the incremental development/deployment of the system this characteristic must also reflect on testing strategy; The ultimate goal is to have all modeled laws, rules and regulations tested with maximum coverage, to minimize the risk of knowledge modeling.

21 Testing the BeInformed Legal Based Models BeInformed is part of the INDiGO Architecture; BeInformed functionality is part of the Siebel user-interface Legal Base Models in BeInformed are, through a ‘instrument’, published as a Web Service; The Web Services functionality is ‘well’ defined and is used within a ‘Application Workflow’; With the definition of each “Application Workflow’ the scope of the Web Services is determined and can be tested.

22 INDiGO architecture ESB BVVBuZaGBA..... Involved Organisations

23 Testing of the BeInformed system It is not about testing the way the models were build…

24 Testing of the BeInformed system … but about testing the way the information is used

25 Testing of the BeInformed system … with the same legal basis

Axini Testmanager 26

27 Testing per Instrument De kennismodellen worden middels instrumenten gepubliceerd op de ESB/GKI en zijn beschikbaar voor INDiGO concept Instrument Application Workflow Examples are: Price of a Single Document Term for a process step to be ended Legal end Preferred end Documents needed for a single application Countries / Nationality Classification (European Citizen)

28 Testing per Instrument Each Instrument covers a part of the Legal Based Models concept Instrument Application Workflow

29 Testing per Instrument per Application Workflow concept Instrument Each Application Workflow covers a part of the Legal Based Models per instrument Application Workflow

30 ? / ? Some parts of the decision tree do not apply to this specific subject under test ? / ? Within each Application Workflow you only cover a part of the model ? / ? Testing per Instrument per Application Workflow

31 The more we test, the more we cover the modeled laws and rules concept Instrument Application Workflow Testing per Instrument per Application Workflow

32 Testing per Instrument per Application Workflow – All Workflows Tested If all Application Workflows have been tested, we cover (nearly) all modeled laws and rules concept Instrument Application Workflow

33 Testing per Instrument / System Functions The Ultimate Goal of this approach is to cover all Legal based models, but at least to have a maximum coverage of all instruments. concept Instrument Application Workflow

34 Conclusions Highly Qualified key-users should not / must not be involved in “easy but intensive” testing process; Experience, Knowledge and Insight must give input for MBT in stead of writing Test scripts; Using MBT-tools improves coverage of all possible paths through the model; In an Agile, iterative and complex environment: MBT-tools can easily adapt to changes;

ORDINA N.V. Ringwade 1Tel: www.ordina.nl 3439 LM NieuwegeinFax: