1. Microsofts Protocol Documentation Program: A Success Story for Model-Based Testing Yiming Cao, Lead Software Design Engineer Windows Interoperability Engineering Team, Microsoft Corporation Dagstuhl SeminarDagstuhl Seminar, October 2010, Leibniz-Zentrum für Informatik, Germany.

2. Part 1: Background Microsoft's Protocol Documentation Program: A Success Story for Model-Based Testing Copyright © 2010, Microsoft Corporation 9/4/2010 2

3. Microsoft Open Protocols Microsoft's Protocol Documentation Program: A Success Story for Model-Based Testing Copyright © 2010, Microsoft Corporation http://www.microsoft.com/protocols 9/4/2010 3

4. BlueLine Technical Document Testing Program of Windows (as of 03/09) 222 protocols/technical documents tested 22,847 pages studied and converted into requirements 36,875 testable requirements identified and converted into test assertions 69% tested using MBT 31% tested using traditional test automation 66,962 person days (250+ years) Hyderabad: 250 test engineers Beijing: 100 test engineers Microsoft's Protocol Documentation Program: A Success Story for Model-Based Testing Copyright © 2010, Microsoft Corporation 9/4/2010 4

5. Formalization Technical Document Test Suite English Statements (Requirements) Assertions Checked against Windows (Accuracy) Validate consistency Simulate implementation process Validate consistency Simulate implementation process Test-Driven Document Quality Assurance Microsoft's Protocol Documentation Program: A Success Story for Model-Based Testing Copyright © 2010, Microsoft Corporation 9/4/2010 5

6. Microsoft's Protocol Documentation Program: A Success Story for Model-Based Testing Copyright © 2010, Microsoft Corporation Study Document study Test strategy (modeled or traditional) Requirements spec (initial) Estimates of Effort Plan Requirements spec (final) Test plan Architecture Scenarios Design Test design Model, or Test case descriptions Adapter Design Final Test case code Generated, or Manual Adapter Implementation Test logs and report Network captures Test setup artifacts Process: Managing a Project with 350 Test Engineers Review after every phase High-profile reviewer team consisting of industry experts and Microsoft senior engineers, build along the model of a program committee Review based on process report and test suite artifacts Verdict accept, conditional accept, reject 9/4/2010 6

7. Requirements Tracking Microsoft's Protocol Documentation Program: A Success Story for Model-Based Testing Copyright © 2010, Microsoft Corporation Technical Document Requirements Spec Model or Test Design Test Suite Logs Network Captures Statements identified in document and tracked with requirement ID IDs referenced from model or test assertion IDs logged at test execution time IDs send to network as beacon packages for network monitor capturing manual automated 9/4/2010 7

8. Typical Requirements Specification Microsoft's Protocol Documentation Program: A Success Story for Model-Based Testing Copyright © 2010, Microsoft Corporation Verification Comment Use this column to explain the reason why the requirement is non-verifiable or unverified (for Normative, Server requirements). Req ID Doc Sect Descrip tion Pos Neg Derive d Scenari os Behavi or Scope Priority Inform ative/ Norma tive Verifica tion Verifica tion Comm ent MS- CHAP_R14 3.2.5When a peer receives a Failure-Request message, if the embedded MSCHAPv2 packet indicates a password expiration error, the peer MAY reply with a Change- Password-Response message. ProtocolClientp2NormativeUnverifiedClient behavior not verified. MS- CHAP_R15 3.3.1In addition, the authentication phase is terminated by sending an EAP-Success/EAP-Failure packet to the peer. S1,S2,S3,S4, S5,S6 ProtocolServerp2Informativ e Test Case MS- CHAP_R16 3.3.5 An EAP server MUST transmit a Challenge- Request message to initiate an Extensible Authentication Protocol Method for Microsoft CHAP authentication procedure. S1,S2,S3,S4, S5,S6,S7,S8 ProtocolServerp0NormativeTest Case 9/4/2010 8

9. Part 2: Model-Based Test Suite Development Microsoft's Protocol Documentation Program: A Success Story for Model-Based Testing Copyright © 2010, Microsoft Corporation 9/4/2010 9

10. Model-Based Document Analysis Model Implementation Expected Outputs (Test Oracle) Inputs (Test Sequences) Control Document Generate Observe Feedback Author Issue Feedback Verdict Feedback Microsoft's Protocol Documentation Program: A Success Story for Model-Based Testing Copyright © 2010, Microsoft Corporation 9/4/2010 10

11. Process Drilldown Microsoft's Protocol Documentation Program: A Success Story for Model-Based Testing Copyright © 2010, Microsoft Corporation Gather Requirements Define ActionsDefine Adapters Model & Explore Generate TestsImplement Adapters Run Tests 9/4/2010 11

12. Gathering Requirements Microsoft's Protocol Documentation Program: A Success Story for Model-Based Testing Copyright © 2010, Microsoft Corporation Gather Requirements Define ActionsDefine Adapters Model & Explore Generate TestsImplement Adapters Run Tests 9/4/2010 12

13. Developing and Implementing Adapters Microsoft's Protocol Documentation Program: A Success Story for Model-Based Testing Copyright © 2010, Microsoft Corporation Gather Requirements Define Actions Define Adapters Model & Explore Generate Tests Implement Adapters Run Tests 9/4/2010 13

14. Adapter Interface TC Test Cases (traditional or MBT) TC Adapter Implementation SUT (System under test) Abstracts SUT functionality Contract between teams Test case team Adapter team Pluggable Different server setups Different transports Test Adapters Microsoft's Protocol Documentation Program: A Success Story for Model-Based Testing Copyright © 2010, Microsoft Corporation 9/4/2010 14

15. Develop Model and Generate Tests Microsoft's Protocol Documentation Program: A Success Story for Model-Based Testing Copyright © 2010, Microsoft Corporation Gather Requirements Define Actions Define Adapters Model & Explore Generate Tests Implement Adapters Run Tests Potential dependency on data types 9/4/2010 15

16. Spec Explorer 2010 Technology Breakdown Model programs Guarded state update rules Rich object-oriented model state (collections, object graphs) Language agnostic (Based on.Net intermediate language interpretation) Trace patterns Regular style language to represent scenarios Slicing of model program by composition Symbolic state exploration and test generation Expands parameters using combinatorial interaction testing Extracts a finite interface automaton (IA) from composed model Traverses IA to generate standalone test code –or- Runs on-the-fly tests from IA Integrated into Visual Studio 2010 Microsoft's Protocol Documentation Program: A Success Story for Model-Based Testing Copyright © 2010, Microsoft Corporation 9/4/2010 16

17. Spec Explorer 2010 Look & Feel Explore Analyze Generate Execute C# Model (or other.Net Language) Model Graph Test Suite VSTT Result Microsoft's Protocol Documentation Program: A Success Story for Model-Based Testing Copyright © 2010, Microsoft Corporation 9/4/2010 17

18. Spec Explorer Walkthrough: Chat Example Users can Enter the session Exit the session List all session users Broadcast a message Received by all session users 18 Chat Room Server User1 SUT User2User3 What happens if two users broadcast a message at virtually the same time?

19. Chat Requirements 19 R1: User MUST receive response for logon request R2: User MUST receive response for logoff request R3: User MUST receive response for list request R4: List response MUST contain the list of logged-on users if successful R5: All logged-on users MUST receive broadcast message R6: Messages from one sender MUST be received in order

20. Chat traces Is the trace correct? (Assume two users (user1, user2) are logged on) 20 T1 Broadcast(user1,1) Broadcast(user2,2) BroadcastAck (user2,1) BroadcastAck (user1,2) BroadcastAck (user1,1) BroadcastAck (user2,2) T2 Broadcast (user1,1) Broadcast (user2,2) BroadcastAck (user1,2) BroadcastAck (user1,1) BroadcastAck (user2,1) BroadcastAck (user2,2) T3 Broadcast (user1,1) Broadcast (user2,2) BroadcastAck (user1,2) BroadcastAck (user2,2) BroadcastAck (user1,1) BroadcastAck (user2,1) If each user sends one message, any receiving order is correct!

21. Chat traces Is the trace correct? (Assume two users (user1, user2) are logged on) 21 T4 Broadcast(user1,1a) Broadcast(user1,1b) BroadcastAck(user2,1a) BroadcastAck(user1,1a) BroadcastAck(user1,1b) BroadcastAck(user2,1b) T5 Broadcast(user1,1a) Broadcast(user1,1b) BroadcastAck(user1,1a) BroadcastAck(user2,1b) BroadcastAck(user1,1b) BroadcastAck(user2,1a) Local order consistency: messages sent by one user must be received in order

22. 22

23. 23

24. Spec Explorers Conformance Notion Alternating simulation: SUT must simulate all stimuli of model Model must simulate all responses of SUT (responses are buffered) 24 Broadcast(1,1) Broadcast(1,2) ModelSUT Broadcast(1,1) Broadcast(1,2) BroadcastAck(1,1)BroadcastAck(2,1) BroadcastAck(1,1) BroadcastAck(2,1) BroadcastAck(1,2)

25. Spec Explorers Approach to State Explosion Slicing techniques: Parameter selection (Constraints, Pairwise, etc.) State filtering Trace patterns Requirement coverage Slicing requires human intervention Smart slicing is an art Spec Explorer tailored for slicing design 25

26. Where a Trace can end: Accepting State Condition Is the following a valid word? Micr Does the following trace represents a useful test? Broadcast(1,1); Broadcast(1,1) An Accepting state condition characterizes those states in which a trace can end Used to ensure that a trace does not stop at arbitrary points Used to ensure that a test leaves the system in a good state Accepting state condition for Chat: All messages have been delivered to recipients 26

27. Demo: Chat Model What the demo shows OO model state Using helper methods in models Modeling with events Slicing

28. Demo: Server Message Block Protocol v2 What the demo shows: Modeling a key Microsoft product Using C# to model state transition systems Using trace patterns to slice test purposes from models Generating and executing tests Microsoft's Protocol Documentation Program: A Success Story for Model-Based Testing Copyright © 2010, Microsoft Corporation 9/4/2010 28

29. Part 3: Evaluation Microsoft's Protocol Documentation Program: A Success Story for Model-Based Testing Copyright © 2010, Microsoft Corporation 9/4/2010 29

30. Comparison MBT vs Traditional Microsoft's Protocol Documentation Program: A Success Story for Model-Based Testing Copyright © 2010, Microsoft Corporation In % of total effort per requirement, normalizing individual vendor performance Vendor 2 modeled 85% of all test suites, performing relatively much better than Vendor 1 9/4/2010 30

31. Remarks on Comparison Numbers backed up by correlation analysis conducted by empirics expert Engineers applying MBT were lay folks, most of them freshly hired from college Few power modelers achieve much higher efficiency rates Though model authoring takes only a smaller part of overall effort per requirement, modeling approach seems to determine efficiency Microsoft's Protocol Documentation Program: A Success Story for Model-Based Testing Copyright © 2010, Microsoft Corporation 9/4/2010 31

32. Factors for Success Tool support Modeling in mainstream languages and environments Visualization of exploration and actual generated test suite (no magic as in online testing) Dealing with state explosion by scenario slicing Training and Guidance One week class for every new hire (including both general methodology and model based testing) Size of project generated critical mass for community help Reviewing process added additional guidance Microsoft's Protocol Documentation Program: A Success Story for Model-Based Testing Copyright © 2010, Microsoft Corporation 9/4/2010 32

33. Pain Points Observed Delta Testing Support A new set of test cases generated based on a new version of model: Which old cases can be re-used? Which old cases are not valid anymore? Which new cases need to be added? By-passing assertion at test execution time Push-button technology for requirement coverage with minimum transitions Good heuristics is needed when behavior is infinite 9/4/2010 Microsoft's Protocol Documentation Program: A Success Story for Model-Based Testing Copyright © 2010, Microsoft Corporation 33

34. Pain Points Observed (continued) Ambiguity in Requirement Coverage What does it mean to cover a requirement in model? Covering a transition once/multi-times? Or even path coverage or pairwise coverage? Rule Debugging Support C# provides rich programming experience But rules can get so complex that a debugger is needed to analyze whats going wrong 9/4/2010 Microsoft's Protocol Documentation Program: A Success Story for Model-Based Testing Copyright © 2010, Microsoft Corporation 34

35. Conclusions Biggest application of behavioral model- based testing known so far Probe big enough for relevant empiric results, proving efficiency gain of 42% Will lead to more momentum on modeling in general inside of Microsoft Model-based testing rocks! Microsoft's Protocol Documentation Program: A Success Story for Model-Based Testing Copyright © 2010, Microsoft Corporation 9/4/2010 35

36. Advertisement… Microsoft offers Spec Explorer 2010 as preview technology free of charge via Visual Studio Gallery: http://visualstudiogallery.msdn.microsoft.com/ en-us/271d0904-f178-4ce9-956b-d9bfa4902745 Licensing allows for commercial use Academic institutions can join MSDN Academic Alliance for access to Visual Studio 2010 Microsoft's Protocol Documentation Program: A Success Story for Model-Based Testing Copyright © 2010, Microsoft Corporation 9/4/2010 36

37. © 2009 Microsoft Corporation. All rights reserved. Microsoft, Windows, Windows Vista and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries. The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION. Microsoft's Protocol Documentation Program: A Success Story for Model-Based Testing Copyright © 2010, Microsoft Corporation 9/4/2010 37