1. April 20, 2006 Model Program Based Black-Box Testing, Lentedagen, Vught, The Netherlands 1 Model Program Based Black-Box Testing Margus Veanes Foundations of Software Engineering Microsoft Research, Redmond

2. April 20, 2006Model Program Based Black-Box Testing, Lentedagen, Vught, The Netherlands 2 System-level black-box testing ► Requires model of system behavior ► Behavior is often reactive/nondeterministic  Implementation is multi-threaded or distributed  Thread scheduling hard to control ► State space is typically infinite  Objects, unbounded value types ► Traditional FSM-based testing does often not scale

3. April 20, 2006Model Program Based Black-Box Testing, Lentedagen, Vught, The Netherlands 3 Model based testing Model Implementation Test OracleTest Cases Are run by Provides actual results for Pass Fail Provides expected results for Generates User Info

4. April 20, 2006Model Program Based Black-Box Testing, Lentedagen, Vught, The Netherlands 4 Our approach ► Behavior is described by model programs  Written in Spec# or AsmL  Describe reactive system behavior as model automata  Use alternating simulation for conformance ► Model exploration, validation and model-based testing are provided by the Spec Explorer tool developed at MSR  Supports scenario control  Provides offline as well as online testing  Views test cases as game strategies  Checks conformance violations

5. April 20, 2006Model Program Based Black-Box Testing, Lentedagen, Vught, The Netherlands 5 The big picture Model program Constrained model Test cases Pass/Fail Implementation Scenario control Modeling Test generation Test executionCoding Online testing

6. April 20, 2006Model Program Based Black-Box Testing, Lentedagen, Vught, The Netherlands 6 Conformance checking component view Constrained model Model IUT wrapper IUT Refinement checker Controllable actions Observable actions Pass/Fail d1 …… c1 Object mapping

7. April 20, 2006Model Program Based Black-Box Testing, Lentedagen, Vught, The Netherlands 7 Example: Chat IUT wrapper IUT Ch at ser ver Controllable actions: Observable actions: Login … Send Receive Refinement checker Constrained chat model b.Send(“hi”) m.Send(“hello”) b.Send(“bye”) m.Rcv(“hi”) m.Rcv(“bye”) b.Rcv(“hello”) m.Rcv(“bye”) m.Rcv(“hi”) b.Rcv(“hello”) Bob Marym b Object mapping

8. April 20, 2006Model Program Based Black-Box Testing, Lentedagen, Vught, The Netherlands 8 Refinement relation ► Conformance between the model and the system under test is alternating refinement b.Send(“hi”) m.Send(“hello”) SpecSystem b.Send(“hi”) m.Send(“hello”) m.Receive(b,“hi”)b.Receive(m,“hello”) m.Receive(b,“hi”)b.Receive(m,“hello”)

9. April 20, 2006Model Program Based Black-Box Testing, Lentedagen, Vught, The Netherlands 9 Example: Valid chat scenario BobServerMary B.Send(“hi”) M.Send(“hello”) M.Receive(B,“hi”) B.Receive(M,“hello”) Time Service should guarantee local consistency of message delivery (FIFO wrt sender) Send is a controllable action Receive is an observable action B.Send(“bye”) M.Receive(B,“bye”)

10. April 20, 2006Model Program Based Black-Box Testing, Lentedagen, Vught, The Netherlands 10 Model programs and automata ► States are mappings of variables to values  first-order structures ► Initial state is given by initial assignment to variables ► Actions are defined by method invocations  Actions are either controllable or observable ► An action m(v) is enabled is a state S if the precondition of m(v) is true in S ► An action m(v) transitions from state S 1 to state S 2 if m(v) is enabled in S 1 and its execution in S 1 yields S 2

11. April 20, 2006Model Program Based Black-Box Testing, Lentedagen, Vught, The Netherlands 11 Model program : Chat service State: Actions: class Client { bool entered = false; Map > unreceivedMsgs = Map{}; } class Client { void Send(string message) requires entered; { foreach (Client c in enumof(Client), c != this, c.entered) c.unreceivedMsgs[this] += Seq{message}; } void Receive(Client sender, string message) requires sender != this && unreceivedMsgs[sender].Length > 0 && unreceivedMsgs[sender].Head == message; { unreceivedMsgs[sender] = unreceivedMsgs[sender].Tail;} … } Controllable: Observable:

12. April 20, 2006Model Program Based Black-Box Testing, Lentedagen, Vught, The Netherlands 12 Online testing ► For large reactive systems deriving an exhaustive test-suite is infeasible ► Merge test generation and test execution into one process ► Online testing is a form of model-based stress testing ► Purely random exploration is wasteful  Use scenario control to direct online testing

13. April 20, 2006Model Program Based Black-Box Testing, Lentedagen, Vught, The Netherlands 13 Exploration & Scenario control ► Exploration is the process of unfolding a rule- based model program into a transition system  Actions move the system from state to state  State comes from variables containing values  Compound data types are possible (sets, maps, sequences, etc…)  Objects are identities ► Scenario control  Imposes limits on exploration ► Using filters, state groupings, scenario actions, …

14. April 20, 2006Model Program Based Black-Box Testing, Lentedagen, Vught, The Netherlands 14 Example Demo: exploration of a coffee machine

15. April 20, 2006Model Program Based Black-Box Testing, Lentedagen, Vught, The Netherlands 15 Scenario control ► In general the transition system is infinite, but we can impose limits ► Goal  Create a state space of manageable size that satisfies a given testing goal ► Two main tasks  Restrict action parameter domains to interesting values  Restrict the state space to interesting states ► Note: The two tasks are not necessarily independent!

16. April 20, 2006Model Program Based Black-Box Testing, Lentedagen, Vught, The Netherlands 16 Scenario control (cont.) ► The following techniques are used in Spec Explorer  State filters  Stopping conditions  State groupings  Search order  Enabling conditions on actions  Scenario actions ► Usually a combination of all (or some) of the techniques is needed to achieve desired effect

17. April 20, 2006Model Program Based Black-Box Testing, Lentedagen, Vught, The Netherlands 17 Example of scenario actions: Chat initialization ► Use a parameterized scenario action Start.  Invoking Start( n ) produces a sequence of n Create() invocations followed by n Enter() invocations [Action(Kind=Scenario)] void Start(int nrOfMembers) requires enumof(Client) == Set{}; { Seq clients = Seq{}; // 1-- The given number of // clients are created for (int i = 0; i < nrOfMembers; i++) clients += Seq{Create()}; // 2-- all clients enter the session for (int i = 0; i < nrOfMembers; i++) clients[i].Enter(); } Start(2) Create()/c0; Create()/c1; c0.Enter(); c1.Enter();

18. April 20, 2006Model Program Based Black-Box Testing, Lentedagen, Vught, The Netherlands 18 Test generation ► The result of exploration is a finite transition system with observable and controllable actions : a model automaton A. ► Testing can be viewed as a game between two players  Testing tool (TT) ► Making controllable transitions  Implementation under test (IUT) ► Making observable transitions ► The “rules of the game” are dictated by A ► Tests are generated from A. Test cases are strategies  A strategy tells what move TT should make in a state where its actions are enabled  The purpose of a strategy may be to achieve transition coverage or to reach a certain goal state

19. April 20, 2006Model Program Based Black-Box Testing, Lentedagen, Vught, The Netherlands 19 Demo: chat

20. April 20, 2006Model Program Based Black-Box Testing, Lentedagen, Vught, The Netherlands 20 Experiences ► Most tool features driven by demands of internal users at MS (mostly testers) ► Models help discover more bugs during modeling (design bugs) than testing  Testers do not get credit for finding those bugs !!! ► During testing models help discover deep system- level bugs where manual test scenarios don’t  Such bugs are hard to understand and fix

21. April 20, 2006Model Program Based Black-Box Testing, Lentedagen, Vught, The Netherlands 21 Experiences (cont.) ► Bugs appear in both models and implementations (the ratio is roughly 50-50) ► Code coverage is a poor measure for testing concurrent software, often a single execution thread provides the same coverage. ► New versions of implementation usually require only local changes to models, whereas manual tests often need to be rewritten completely ► The tool is built on.NET but has been used to test distributed C++ applications. Most test harnesses at MS are built in managed code.

22. April 20, 2006Model Program Based Black-Box Testing, Lentedagen, Vught, The Netherlands 22 Outlook ► User wish list:  Language independence for modeling ► Use C#, VB, …  Better control over exploration ► Scenario-oriented models  Model composition ► Model features separately and compose the models  Model validation ► Validate properties on models Drives the focus of current research at FSE

23. April 20, 2006Model Program Based Black-Box Testing, Lentedagen, Vught, The Netherlands 23 Related foundational work ► Abstract state machines ► Interface automata ► Input-Output COnformance (IOCO) theory ► Markov Decision Process (MDP) theory

24. April 20, 2006Model Program Based Black-Box Testing, Lentedagen, Vught, The Netherlands 24 Thank you! ► FSE web site:  http://research.microsoft.com/foundations http://research.microsoft.com/foundations ► Downloading Spec Explorer:  http://research.microsoft.com/specexplorer http://research.microsoft.com/specexplorer Questions?