Presentation is loading. Please wait.

Presentation is loading. Please wait.

Multi-Paradigmatic Model-Based Testing Wolfgang Grieskamp Microsoft Research.

Similar presentations


Presentation on theme: "Multi-Paradigmatic Model-Based Testing Wolfgang Grieskamp Microsoft Research."— Presentation transcript:

1 Multi-Paradigmatic Model-Based Testing Wolfgang Grieskamp Microsoft Research

2 Wolfgang GrieskampMicrosoft Research2Agenda I. Model-Based Testing at Microsoft II. Lessons Learned III. A Multi-Paradigmatic Approach & Tool

3 Wolfgang GrieskampMicrosoft Research3Agenda I. Model-Based Testing at Microsoft II. Lessons Learned III. A Multi-Paradigmatic Approach & Tool

4 Wolfgang GrieskampMicrosoft Research4 Model-Based Testing at Microsoft Applied since around 1999 Applied since around 1999 Mailing list > 700 members (~10 % of MS testers) Mailing list > 700 members (~10 % of MS testers) Number of internal tools around Number of internal tools around Microsoft Research tools: Microsoft Research tools: AsmL-T (2002) AsmL-T (2002) Spec Explorer 1 (2004) Spec Explorer 1 (2004) Spec Explorer 2 (in development) Spec Explorer 2 (in development) A success story, but MBT is not yet mainstream A success story, but MBT is not yet mainstream

5 Wolfgang GrieskampMicrosoft Research5 How current MS tools work Based on EFSM or ASM Based on EFSM or ASM A set of parameterized actions A set of parameterized actions A set of parameterized guarded update rules associated with actions A set of parameterized guarded update rules associated with actions Update rules fire in states where they are enabled Update rules fire in states where they are enabled Try all parameter combinations as defined by the user Try all parameter combinations as defined by the user State transitions are labeled with associated actions State transitions are labeled with associated actions EFSM vs ASM (Spec Explorer): state space can grow dynamically EFSM vs ASM (Spec Explorer): state space can grow dynamically

6 Wolfgang GrieskampMicrosoft Research6 A Glance of Spec Explorer [Campbell, Grieskamp, Nachmanson, Schulte, Tillmann, Veanes 2005] Actions and update rules are defined as parameterized methods of an AsmL or Spec# model program Actions and update rules are defined as parameterized methods of an AsmL or Spec# model program Actions are partitioned into controlled and observed ones Actions are partitioned into controlled and observed ones Controlled = Input, Observed = Output Controlled = Input, Observed = Output Alternating refinement provides the conformance notion Alternating refinement provides the conformance notion SUT must simulate all control steps of Model SUT must simulate all control steps of Model Model must simulate all observation steps of SUT Model must simulate all observation steps of SUT Quick demo Quick demo

7 Wolfgang GrieskampMicrosoft Research7Agenda I. Model-Based Testing at Microsoft II. Lessons Learned III. A Multi-Paradigmatic Approach & Tool

8 Wolfgang GrieskampMicrosoft Research8 Lessons Learned 1. Adoption Problems AuthoringAuthoring Executable Specification LanguagesExecutable Specification Languages Scaling up to Model-Based DevelopmentScaling up to Model-Based Development Education and DocumentationEducation and Documentation 2. Technological Problems State ExplosionState Explosion Test SelectionTest Selection Test ManagementTest Management

9 Wolfgang GrieskampMicrosoft Research9 Lessons Learned 1. Adoption Problems 2. Technological Problems

10 Wolfgang GrieskampMicrosoft Research10 Authoring Where is my IntelliSense? Where is my IntelliSense? Context-sensitive editing, incremental compilation, refactoring, etc. ubiquitous today Context-sensitive editing, incremental compilation, refactoring, etc. ubiquitous today Environments that support this probably more relevant than the actual languages Environments that support this probably more relevant than the actual languages If you provide your own modeling notation, you better support decent authoring If you provide your own modeling notation, you better support decent authoring But: Building decent authoring support for a language is probably by an order of magnitude more complex than writing its compiler

11 Wolfgang GrieskampMicrosoft Research11 Executable Specification Languages Why do I have to learn this new language? Why do I have to learn this new language? High-level language constructs hard to digest for many testers High-level language constructs hard to digest for many testers Mainstream programming languages catch up Mainstream programming languages catch up Let the language guys market this Let the language guys market this Dont mix up evangelizing MBT and your idea of language design Dont mix up evangelizing MBT and your idea of language design But: some stakeholders like those high-level languages, in particular outside of test organizations; therefore it is best to be agnostic regarding choice of languages.

12 Wolfgang GrieskampMicrosoft Research12 Scaling up to Model-Based Development Isnt this very low-level? Isnt this very low-level? Where is the test plan? Where is the test plan? Scenario-oriented (=interaction-based) modeling is ubiquitous in the process Scenario-oriented (=interaction-based) modeling is ubiquitous in the process Need to support scenarios, best using diagram notations (UML) Need to support scenarios, best using diagram notations (UML) But: Scenarios need to be combined (composed) with state-based models.

13 Wolfgang GrieskampMicrosoft Research13 Education and Documentation I do not have time to learn this! I do not have time to learn this! Where is the textbook for this? Where is the textbook for this? People in the industry have not enough time to educate themselves People in the industry have not enough time to educate themselves Practical education material for MBT is poor Practical education material for MBT is poor Need better coverage at universities, text books, and individual tool documentation Need better coverage at universities, text books, and individual tool documentation But: chicken-and-egg problem: mainstream in university education requires proof of feasibility in industry

14 Wolfgang GrieskampMicrosoft Research14 Lessons Learned 1. Adoption Problems 2. Technological Problems

15 Wolfgang GrieskampMicrosoft Research15 State Explosion Cool! That stuff generates a large number of tests on the click of a button! … Cool! That stuff generates a large number of tests on the click of a button! … Test suite execution time is a significant productivity factor Test suite execution time is a significant productivity factor On-the-fly only works in particular test applications On-the-fly only works in particular test applications Need better techniques to reduce the state space Need better techniques to reduce the state space Partly solved by test purposes, but more work is required (apply POR, state symmetries, etc. from MC)

16 Wolfgang GrieskampMicrosoft Research16 Test Selection I want to have these tests from my model I want to have these tests from my model Typical request: find the shortest path to a given state; from then on, apply link coverage Typical request: find the shortest path to a given state; from then on, apply link coverage Give users more fine-grained control over test selection, in particular, composition of test selection strategies Give users more fine-grained control over test selection, in particular, composition of test selection strategies Partly solved by test purposes, provided context sensitive selection strategies can be controlled with test purposes

17 Wolfgang GrieskampMicrosoft Research17 Test Management How does this integrate with my test management tool X? How does this integrate with my test management tool X? Test generation only part of the testing problem Test generation only part of the testing problem A diversity of test management tools and processes around A diversity of test management tools and processes around Cannot prescribe test management from an MBT tool, but must be able to plug-in a variety of such frameworks Cannot prescribe test management from an MBT tool, but must be able to plug-in a variety of such frameworks

18 Wolfgang GrieskampMicrosoft Research18Agenda I. Model-Based Testing at Microsoft II. Lessons Learned III. A Multi-Paradigmatic Approach & Tool

19 Wolfgang GrieskampMicrosoft Research19Multi-Paradigmatic? par·a·digm (plural par·a·digms) noun 1.typical example: a typical example of something 2.model that forms basis of something: an example that serves as a pattern or model for something, especially one that forms the basis of a methodology or theory 3.relationship of ideas to one another: in the philosophy of science, a generally accepted model of how ideas relate to one another, forming a conceptual framework within which scientific research is carried out

20 Wolfgang GrieskampMicrosoft Research20 Principles Do not stick to a single paradigm Do not stick to a single paradigm State-based and scenario-based modeling State-based and scenario-based modeling Textual and diagrammatic notations Textual and diagrammatic notations Programming languages and specification languages Programming languages and specification languages Support combination of paradigms Support combination of paradigms Model composition Model composition Fully integrate into a modern development environment Fully integrate into a modern development environment IntelliSense, refactoring, and so on IntelliSense, refactoring, and so on

21 Wolfgang GrieskampMicrosoft Research21 Quick Glance Model Explore Traverse Test Model with Charts

22 Wolfgang GrieskampMicrosoft Research22 Quick Glance Model Explore Traverse Test Model with Programs

23 Wolfgang GrieskampMicrosoft Research23 Quick Glance Model Explore Traverse Test Explore Individually or in Composition

24 Wolfgang GrieskampMicrosoft Research24 Quick Glance Model Explore Traverse Test Explore for Model- Checking

25 Wolfgang GrieskampMicrosoft Research25 Quick Glance Model Explore Traverse Test Traverse for Test-Selection

26 Wolfgang GrieskampMicrosoft Research26 Quick Glance Model Explore Traverse Test Test Execution

27 Wolfgang GrieskampMicrosoft Research27 Demo

28 Wolfgang GrieskampMicrosoft Research28 Foundations: Action Machines [Grieskamp, Kicillof, Tillmann 2005/2006] A combination of LTS and automata A combination of LTS and automata Given: Given: T a set of terms with variables to represent values and action labels T a set of terms with variables to represent values and action labels E a set of environments mapping locations to terms E a set of environments mapping locations to terms Special location i containing the action label Special location i containing the action label E forms a complete lattice with meet and join E forms a complete lattice with meet and join Action machine M = (C,A,I,T): Action machine M = (C,A,I,T): C a set of control points C a set of control points A C a set of accepting control points A C a set of accepting control points I E E C initialization relation: e (e,c) I I E E C initialization relation: e (e,c) I T E C E C stepping relation: (e,c) (e,c) T T E C E C stepping relation: (e,c) (e,c) T Remark: the action label is contained in the special location i Remark: the action label is contained in the special location i

29 Wolfgang GrieskampMicrosoft Research29 Given Action Machines Guarded-update machine (encapsulates model program) Guarded-update machine (encapsulates model program) Defined by a set of guarded-update rules Defined by a set of guarded-update rules Steps for each rule in each environment where the guard is true Steps for each rule in each environment where the guard is true Step updates the environment Step updates the environment Two control points: accepting and non-accepting Two control points: accepting and non-accepting Determined by a predicate over the environment Determined by a predicate over the environment Single-step machine (building block for scenarios): Single-step machine (building block for scenarios): Defined by an action term Defined by an action term Two control points: before-step and after-step Two control points: before-step and after-step Before-step: enables transition into after-step labeled with action term; is non-accepting Before-step: enables transition into after-step labeled with action term; is non-accepting After-step: nothing enabled; is accepting After-step: nothing enabled; is accepting Completely parametric over environment Completely parametric over environment

30 Wolfgang GrieskampMicrosoft Research30 Composed Action Machines Synchronized Parallel Machine Synchronized Parallel Machine Control points are product of control points of composed machines Control points are product of control points of composed machines Steps if both machines step into an environment which has a non-empty join: Steps if both machines step into an environment which has a non-empty join: (e,(c1,c2)) (e1 e2,(c1,c2)) iff (e,(c1,c2)) (e1 e2,(c1,c2)) iff (e,c1) (e1,c1) (e,c1) (e1,c1) (e,c2) (e2,c2) (e,c2) (e2,c2) e1 e2 e1 e2 Alternating Refinement Machine Alternating Refinement Machine Similar as product machine, but: Similar as product machine, but: (e,(c1,c2)) (e,(c1,c2)) iff (e,(c1,c2)) (e,(c1,c2)) iff (e,c1) (e1,c1) (e,c1) (e1,c1) (e,c2) (e2,c2) (e,c2) (e2,c2) control step e1 e2, e = e1 control step e1 e2, e = e1 observation step e1 e2, e = e2 observation step e1 e2, e = e2

31 Wolfgang GrieskampMicrosoft Research31 Composed Action Machines Sequential machine Sequential machine Control points are disjoint union of control points of composed machines Control points are disjoint union of control points of composed machines Each step of the first machine leading into an accepting state is duplicated to lead also into an initial state of the second machine (simplified description) Each step of the first machine leading into an accepting state is duplicated to lead also into an initial state of the second machine (simplified description) Justifies the need of the initialization relation: a machine can be initialized in the environment reached by another machine Justifies the need of the initialization relation: a machine can be initialized in the environment reached by another machine

32 Wolfgang GrieskampMicrosoft Research32 Composed Action Machines Interleaving parallel composition Interleaving parallel composition Repetition Repetition Action translation Action translation Declaration Declaration Constraints Constraints Hiding Hiding Traversals Traversals …

33 Wolfgang GrieskampMicrosoft Research33 Implementation: XRT [Grieskamp, Tillmann, Schulte 2005] Virtual execution framework for.NET intermediate code Virtual execution framework for.NET intermediate code Implements mixed concrete/symbolic state space exploration Implements mixed concrete/symbolic state space exploration Action machines a thin layer on top of XRT Action machines a thin layer on top of XRT

34 Wolfgang GrieskampMicrosoft Research34Agenda I. Model-Based Testing at Microsoft II. Lessons Learned III. A Multi-Paradigmatic Approach & Tool

35 Wolfgang GrieskampMicrosoft Research35Conclusions MBT successful but not mainstream MBT successful but not mainstream Application at MS suggests that tools need to be more powerful and simpler at the same time Application at MS suggests that tools need to be more powerful and simpler at the same time Let users model in mainstream languages (making modeling simpler) and different paradigms (adapting to roles in the process) Let users model in mainstream languages (making modeling simpler) and different paradigms (adapting to roles in the process) Use model-composition as the cornerstone to provide additional power Use model-composition as the cornerstone to provide additional power Multi-paradigmatic MBT is a premier case study for model-based development in general Multi-paradigmatic MBT is a premier case study for model-based development in general Current plans at MSR: Current plans at MSR: Ship the new Spec Explorer tool to internal customers Ship the new Spec Explorer tool to internal customers Possibly ship as emerging technology externally Possibly ship as emerging technology externally Push integration and/or spin-off for mainstream Visual Studio Push integration and/or spin-off for mainstream Visual Studio

36 Wolfgang GrieskampMicrosoft Research36 The End Checkout http://research.microsoft.com/projects/specexplorer for the current Spec Explorer tool and updates regarding the new tool Checkout http://research.microsoft.com/projects/specexplorer for the current Spec Explorer tool and updates regarding the new tool http://research.microsoft.com/projects/specexplorer Checkout my website for papers Checkout my website for papers Thanks! Thanks!


Download ppt "Multi-Paradigmatic Model-Based Testing Wolfgang Grieskamp Microsoft Research."

Similar presentations


Ads by Google