1. Event-B in a Nutshell Test Data Generation 13th CREST Open Workshop 12th-13th of May 2011, London 13th CREST Open Workshop 12th-13th of May 2011, London *) joint work with colleagues from DEPLOY project Alin Stefanescu - University of Pitesti, Romania SBT Challenges Search-Based Software Engineering for Model-Based Testing Test Generation Approaches Finite Model Learning Conclusions

2. Page 2 [Event-B in a Nutshell]

3. Page 3 Event-B history Jean-Raymond Abrial (1938- ) Inventor of the Z and B formal methods. Z – developed in the 70s B – developed in the 90s, successfully deployed in industry Event-B – born with the 21st century Evolution of B for system level specification Developement supported by French and European projects: FP6 RODIN and FP7 DEPLOY

4. Page 4 DEPLOY project (2008-2012) – funded by FP7 DEPLOY :: Industrial deployment of advanced system engineering methods for high productivity and dependability using formal methods 4 industrial partners ■ Bosch, Siemens, SAP, SSF 3 industrial service providers ■ Systerel, ClearSy, Cetic 7 academic partners ■ Newcastle, Aabo, Düsseldorf, ETH Zurich, ■ Southampton, Pitesti, Bucharest http://www.deploy ‐ project.eu

5. Page 5 Rodin platform for Event-B Extension of Eclipse IDE (Java-based) Theorem proving as core technology Many other Rodin Plug-ins  ProB: animation, consistency and model-checking  Animators (AnimB)  Decomposition  Modularisation  Team-work  Code generation  UML-B  etc.

6. Page 6 Event-B in a nutshell ■ State-transition model (like ASM, B, Z) ■ set theory as mathematical language ■ refinement as basic modeling approach ■ Contexts ■ carrier sets (domains) ■ constants ■ axioms ■ Machines ■ global variables ■ invariants ■ events that update the variables ■ Events ■ local parameters ■ guards ■ actions ITEMS := CONTEXT {{it1}, {it34}, {it36}, {it67}, {it89}, {it11}, {it354}, {it876}, {it321}, {it333}, {it78}, {it787}, {it7878}, {it2342}, {it3453}, {it6786}, {it1232}, {it7765}, {it7098}) items : Powerset(ITEMS) ITEMS := CONTEXT {{it1}, {it34}, {it36}, {it67}, {it89}, {it11}, {it354}, {it876}, {it321}, {it333}, {it78}, {it787}, {it7878}, {it2342}, {it3453}, {it6786}, {it1232}, {it7765}, {it7098}) items : Powerset(ITEMS) Event-B model

7. Page 7 [Test Generation Approaches]

8. Page 8 Test generation based on Event-B We investigate search-based testing (SBT) techniques for Event-B. Model-based testing (MBT) is a newly introduced topic in DEPLOY priority topic for industrial partners like SAP challenges due to the sheer size of the state space of real-life scenarios Model-Based Testing (MBT)

9. Page 9 Future MBT plugin in RODIN MBT Plug-in University of Pitesti and University of Dusseldorf Extra test information { Model-checking } Event-B model { Search-based } { Constraint-based } Test cases MBT Users Tool developers:

10. Page 10 Test generation from Event-B Event-B model Global variables: var1, var2, var3,... Events ev1(p11,...), ev2(p21,...), ev3(p31,...),... ev2(..),ev5(..),...ev3() ev4(..),ev2(..),...ev4(..)... ev3(..),ev7(..),...ev5(..)... ev6(..),ev5(..),...ev8 1. Generate a set of tests (sequence of events with concrete param.) 2. Optimize test suite (according to some criteria) – if still needed ev2(..),ev5(..),...ev3() ev4(..),ev2(..),...ev4(..)... ev3(..),ev7(..),...ev5(..) SBT Opportunity!

11. Page 11 What is the explicit state space Event-B model Global variables: var1, var2, var3,... Events ev1(p11,...), ev2(p21,...), ev3(p31,...),... State Space of the Event-B model State Space of the Event-B model... (3,4,{a,b},...) ev3(5) States given by the values of global variables Transitions labeled by events with concrete parameters Abstract machine

12. Page 12 Test generation from Event-B SBT Opportunity! State Space of the Event-B model State Space of the Event-B model... (3,4,{a,b},...) ev3(5) Approach 1: Explore the state space using the ProB model checker state space explosion mainly due to data Try: guide the search

13. Page 13 Test generation from Event-B – part II SBT Opportunities! State Space of the Event-B model State Space of the Event-B model... (3,4,{a,b},...) ev3 Approach 2: Explore state space ignoring the data (i.e. local parameters) Problem 1: still large state space Then: construct approximations of state space up to depth K using finite automata Try 1: using machine learning and static analysis Try 2: using evolutionary algorithms? Problem 2: infeasible sequences Try 1: constraint solving for path feasibility Try 2: test data generation with metaheuristics

14. Page 14 [Search-Based Testing Challenges]

15. Page 15 More details in: A. Stefanescu, F. Ipate, R. Lefticaru, C. Tudose. Towards Search-Based Testing for Event-B Models. To appear in Proc. of 4th International Workshop on Search-Based Software Testing (SBST), 2011. Let’s take a look at some of specific challenges for Event-B...

16. Page 16 No explicit state space Fact Event-B has no explicit states like the EFSMs no control state (as in EFSMs) Problem Large (possibly infinite) state space testing coverage criteria must be defined only recent work addressing SBT for EFSMs Possible ideas: coverage of all events (or a given subset of them) or coverage of all test paths of length < K many other coverages possible, so industrial guidance is needed consider the class of Event-B models with a special state variable (see industrial use cases from SAP, SSF, Bosch and UML-B models)

17. Page 17 Non-numerical types Fact Event-B is based on set theory set relations, powersets, functions, set comprehensions, products, records, etc. Complex structured data (e.g. business domain) Problem fitness functions in literature mostly defined for numerical types Possible solutions design new fitness functions for set-based (non-numerical) types efficient encoding of mixed non-numerical/numerical test data

18. Page 18 Hierarchical models Fact Event-B supports different types of hierarchy refinement from abstract to concrete levels model decomposition modularity most industrial models use some sort of hierarchy (due to size) Problem no much previous work on SBT addressing hierarchical models Possible ideas: adapt existing work on test selection for hierarchical state machines use the existing ProB model checker that can partially deal with hierarchy

19. Page 19 Non-determinism Fact Event-B has different types of non-determinism :| or : ∈ operators (e.g. x : ∈ {item1,..., item20}) non-deterministic choice of the event to be executed when several enabled non-deterministic choice of parameters (ANY construct) non-deterministic initialisation of variables satisfying the set of invariants Problem no much previous work on SBT addressing non-deterministism Possible ideas: devise fitness functions that improve the chance of choosing a given path in a non-deterministic model (under certain assumptions) make the non-determinism visible (model instrumentation)

20. Page 20 [Test Data Generation]

21. Page 21 Generating test data for a path Problem Given one path of events, provide the test data (event parameters) that enables the execution of the path. Approach genetic algorithms encoding of sets into binary genes mixed choromosomes (numerical and binary genes) More details in: I. Dinca, A. Stefanescu, F. Ipate, R. Lefticaru, C. Tudose. Test Data Generation for Event-B Models using Genetic Algorithms. In Proc. of 2nd International Conference on Software Engineering and Computer Systems (ICSECS'11). CCIS Series, vol. 181, pp. 76-90, Springer, 2011.

22. Page 22 Test data generation with genetic algorithms Simulator (ProB) Fitness evaluation ITEMS := CONTEXT {{it1}, {it34}, {it36}, {it67}, {it89}, {it11}, {it354}, {it876}, {it321}, {it333}, {it78}, {it787}, {it7878}, {it2342}, {it3453}, {it6786}, {it1232}, {it7765}, {it7098}) items : Powerset(ITEMS) ITEMS := CONTEXT {{it1}, {it34}, {it36}, {it67}, {it89}, {it11}, {it354}, {it876}, {it321}, {it333}, {it78}, {it787}, {it7878}, {it2342}, {it3453}, {it6786}, {it1232}, {it7765}, {it7098}) items : Powerset(ITEMS) Encoding of variables Mutation Selection Crossover Event-B model items … 011010 Step 1 Purchase Step 2 ValidateLarge Step 3 CheckSpecial 0110 10101 00 1 0110 0111 101 0 Fitness functions “Chromosome” End?

23. Page 23 Fitness functions for one path fitness := approach level + normalized branch level Classical Tracey’s objective for numerical types New objective functions for set types

24. Page 24 Examples from the benchmark

25. Page 25 Statistical results Statistical comparison of  Genetic Algorithms (GA) and  Random Testing (RT)  on 18 paths covering 5 Event-B models  using statistical test like t-test and U-test And (of course) the winner is:  GA performs significantly better than RT on most paths Note: We are currently evaluating constraint- solving (mature for Event-B). It It seems to be quicker for small to medium path (with exceptions).

26. Page 26 [Finite Model Learning]

27. Page 27 Generating finite models from Event-B Problem There is no explicit state space of an Event-B model Approach Finite automata learning (adapted L* algorithm ) Aproximation through cover automata K-bound on the length of executions Use finite automata for conformance test generation More details in: F. Ipate, I. Dinca, A. Stefanescu: Model Learning and Test Generation for Event-B using Cover Automata. Submitted to SEFM’11.

28. Page 28 First experiments Preliminary approach Approximation through cover automata for bound l Incremental -> fits very well with model refinements Minimal finite automata Sometimes difficult to find counterexamples (to the approximation) Scales for medium size models:... ev3 Bound l SBT Opportunity?!

29. Page 29 [Conclusions]

30. Page 30 Opportunities for Search-Based Techniques To wrap-up opportunities: ■ Test suite minimisation with multi-objective optimisation ■ Test data generation for one path with search-based algorithms ■ Construct finite models with evolutionary algorithms ■ Combine ProB model-checker with meta-heuristics ■ Combine ProB constraint-solver with meta-heuristics ■ Experiment with different search algorithms (PSO, ACO, SA,...) To be answered until end of project (April 2012): Which of the above work good in practice?