1. © author(s) of these slides 2014 including research results of the research network ES and TU Darmstadt otherwise as specified at the respective slide Malte Lochau TU Darmstadt Malte.lochau@es.tu-darmstadt.de ES Real-Time Systems Lab Prof. Dr. rer. nat. Andy Schürr Dept. of Electrical Engineering and Information Technology Dept. of Computer Science (adjunct Professor) www.es.tu-darmstadt.de Multi-Objective Test Suite Optimization for Incremental Product Family Testing (joint work with Hauke Baller, Sascha Lity and Ina Schaefer) 03.03.2014, Copenhagen

2. ES – Real-Time Systems Lab 2 | 03.03.2014 | PLE Contents  „What is a good ordering for testing the members of a product family?“  Extension of the classical Test Suite Minimization Problem  Weights  Bounds  Variability  Approx. optimal solution(s) for Multi-Objectives via ILP Solving  Incremental heuristic solution based on dynamic programming to appear at ICST‘2014

3. ES – Real-Time Systems Lab 3 | 03.03.2014 | PLE Test Suite Minimization Problem [Harrold et al., 1993] t1 r1 r2 r3 r4 r5 r6 t2 t3 t4 … Test Cases Test Requirements Complete Test Suite

4. ES – Real-Time Systems Lab 4 | 03.03.2014 | PLE  All-Transitions Coverage R = {r 1,r 2, …, r 8 }  Complete Test Suite TS (4 Test Cases)  Minimal Complete Test Suite TS ‘ (1 Test Case) Example: Model-Based Testing r1r1 r2r2 r3r3 r4r4 r5r5 r6r6 r7r7 r8r8 test model Test Suites  TS = {(r 1 r 2 ),(r 1 r 4 ),(r 3 r 7 r 8 ),(r 3 r 5 r 6 )}  TS ‘ = {(r 1 r 2 r 3 r 5 r 4 r 7 r 8 r 5 r 6 )}

5. ES – Real-Time Systems Lab 5 | 03.03.2014 | PLE Minimum Set Cover Problem Assumptions:  All Test Cases are relevant:  Each Requirement is satisfiable:

6. ES – Real-Time Systems Lab 6 | 03.03.2014 | PLE Cost/Profit Test Suite Optimization [Malishevsky et al. 2006, Xu et al. 2012, Rout et al. 2013] t1 r1 r2 r3 r4 r5 r6 t2 t3 t4 … Test Cases Test Requirements Adequate Test Suite 2 2 1 1 4 4 3 3 1 1 2 2 Testing Profits p TS 4 4 2 2 1 1 4 4 Testing Costs c TS +++++= +++= ≥ Profit Goal k p ≤ Cost Bound k c ∑=7 Minimal Test Suite ∑=8

7. ES – Real-Time Systems Lab 7 | 03.03.2014 | PLE Test Suite Profits p TS  TS = {(r 1 r 2 ),(r 1 r 4 ),(r 3 r 7 r 8 ),(r 3 r 5 r 6 )}  p TS =(2+1)+(2+1)+(2+2+4)+(2+2+1)=15  TS ‘ = {(r 1 r 2 r 3 r 5 r 4 r 7 r 8 r 5 r 6 )}  p TS =(2+1+2+2+1+2+4+2+1)=15  TS ‘‘ = {(r 1 r 4 r 5 r 2 ),(r 3 r 7 )}  p TS =(2+1+2+1)+(2+2)=10 TS ‘‘ is Adequate Test Suite for k p = 10 Example: Model-Based Testing with Costs/Profits Test Suite Costs c TS  TS = {(r 1 r 2 ),(r 1 r 4 ),(r 3 r 7 r 8 ),(r 3 r 5 r 6 )}  c TS = 2+2+3+3 = 10  TS ‘ = {(r 1 r 2 r 3 r 5 r 4 r 7 r 8 r 5 r 6 )}  c TS = 9  TS ‘‘ = {(r 1 r 4 r 5 r 2 ),(r 3 r 7 )}  c TS = 5+2 = 7 r 1 (2) r 2 (1) r 3 (2) r 4 (1) r 5 (2 ) r 6 (1) r 7 (2 ) r 8 (4 ) Profits are counted once k p =10

8. ES – Real-Time Systems Lab 8 | 03.03.2014 | PLE Minimum Partial Weighted Set Cover Problem [Könemann et al., 2011]

9. ES – Real-Time Systems Lab 9 | 03.03.2014 | PLE Product Line Analysis

10. ES – Real-Time Systems Lab 10 | 03.03.2014 | PLE Sample-based Product Line Analysis [Cohen et al. 2006], [Gustafsson 2007], [Oster et al., 2010], [Perrouin et al, 2010], [Kim et al. 2011], [Johansen et al., 2012], [Henard et al., 2013], [Haslinger et al., 2013], ….

11. ES – Real-Time Systems Lab 11 | 03.03.2014 | PLE Family-based Product Line Analysis [Krishnamurti et al., 2006], [Gruler et al., 2010], [Classen et al. 2010], [Gnesi et al. 2011], [Cordy et al., 2013], …

12. ES – Real-Time Systems Lab 12 | 03.03.2014 | PLE Incremental Product Line Analysis [Batory et al. 2007], [Engström et al. 2011], [Lochau et al. 2012], …

13. ES – Real-Time Systems Lab 13 | 03.03.2014 | PLE T={t 1, …, t 15 }  t 7 =(r 1 r 6 r 8 r 7 )  t 8 =(r 1 r 4 r 5 r 2 )  t 11 =(r 1 r 4 r 7 )  t 12 =(r 9 r 10 r 8 )  t 13 =(r 3 r 11 )  t 14 =(r 1 r 4 r 11 )  t 15= (r 3 r 7 )} Family-based Product Line Test Suite Design  4 Product Variants r 1 (2) r 2 (1) r 3 (2) r 4 (1)r 5 (2) r 6 (1) r 7 (2) r 8 (4) r 9 (2) r 10 (2) r 11 (8)

14. ES – Real-Time Systems Lab 14 | 03.03.2014 | PLE Complete Minimal Product Line Test Suite [Cichos et al. 2011]

15. ES – Real-Time Systems Lab 15 | 03.03.2014 | PLE Constraint Minimum Set Cover Problem t1 r1 r2 r3 r4 r5 r6 t2 t3 t4 … Test Cases Test Requirements p1 Product Variants p2 p3 5 5 2 2 3 3 Product Costs/Profits pp TS 2 2 1 1 4 4 3 3 1 1 2 2 4 4 2 2 4 4 1 1 Testing Costs Testing Profits ++ = ≥ Profit Goal k pp p TS ≥ k p rp TS ≥ k rp

16. ES – Real-Time Systems Lab 16 | 03.03.2014 | PLE Constraint (Minimum) Partial Weighted Set Cover Problem

17. ES – Real-Time Systems Lab 17 | 03.03.2014 | PLE Integer Linear Programming (ILP)

18. ES – Real-Time Systems Lab 18 | 03.03.2014 | PLE ILP for Product Line Test Suite Optimization Product p j selected  x j =1 Test Case t selected into TS  y t =1 Requirement r i satisfied  z i =1 Binary Variables pp TS Product selected  Requirements must be satisfied p j  select Test Case from Test Case Set NOT p j OR One Test Case from Test Case Set Product selected  Requirements must be satisfied p j  select Test Case from Test Case Set NOT p j OR One Test Case from Test Case Set t t p p For all Requirements (of a Product) For all Requirements (of a Product) Requirement satisfied  For all Products, a Test Case must be selected Again ( r i  Set)  (NOT r i OR Set) Requirement satisfied  For all Products, a Test Case must be selected Again ( r i  Set)  (NOT r i OR Set) r r t t Test Cases must be applied to a Product t t p p Priotization pp TS ≥ k pp  pp TS - Δ pp = k pp pp TS ≥ k pp c TS ≤ k c rp TS ≥ k rp Differences must not be negativ Normalization Value

19. ES – Real-Time Systems Lab 19 | 03.03.2014 | PLE Re-Formulation as Product Selection Problem  ILP finds Adequate Test Suite (if exists), but Problem is NP-hard  Another Point-of-View: Incremental Product Selection t1 r1 r2 r3 r4 r5 r6 t2 t3 t4 p1 p2 p3 rp TS c TS pp TS

20. ES – Real-Time Systems Lab 20 | 03.03.2014 | PLE Adequate Product Sub Set

21. ES – Real-Time Systems Lab 21 | 03.03.2014 | PLE Incremental Product Selection p1 p2 p3 p4 p1 p2 p3 p4 p5 p7 p6 p8 p9 p5 + + profit goals reached? select next product cost bounds exceeded? no adequate solution found

22. ES – Real-Time Systems Lab 22 | 03.03.2014 | PLE Product Selection Heuristic 1.Are Profit Goals reached? ( rp TS ≥ k rp and pp TS ≥ k pp ) 2.No: a. For each unselected Product i.While unsatisfied Requirements exist, select one:  With high profit and uniqueness/rareness  Select a satisfying Test Case :  With the ability to satisfy many other open Requirements for many other Products and low cost b.Rank Products and Select  By: Product profit, resulting Test Case costs, and Requirement profit (weighted) c.Recalculate pp TS, rp TS and c TS d.Is Test budget excelled ( c TS > k c ) i.Yes: “Error”  End e.Goto 1. 3.Yes: “Success”  End r r n n t t n n cf. [Harrold et al., 1993] cf. [Harrold et al., 1993] [Harrold et al., 1993] + Products [Harrold et al., 1993] + Products

23. ES – Real-Time Systems Lab 23 | 03.03.2014 | PLE Experimental Results  Varying:  Number of: Requirements, Test Cases, and Products  Mapping: Requirements/Test Cases and Test Cases/Products  Weights  In Total: 51 generated Data Sets  Time Limit: 24h t1 r1 r2 r3 r4 r5 r6 t2 t3 t4 p1 p2 p3 50, 100, and 500 Test Cases 50, 100, and 500 Products 50, 100, and 500 Requirements Mean and Variance -/o/+ Heuristic is 21.5 times faster 68.6% less accuracy Efficency improvement factor: 127.48

24. ES – Real-Time Systems Lab 24 | 03.03.2014 | PLE Discussion & Future Work Summary  Heuristic produces useful results and scales  Feasible tradeoff between accuracy and efficiency Ongoing and Future Work  Backtracking if bounds are exceeded  Search-based optimization techniques  Methods and criteria to find appropriate weights and bounds  Beyond ILP:  Dynamic weight function,  On-the-fly generation of test artifacts, …  Application to Real-World Case Studies

25. ES – Real-Time Systems Lab 25 | 03.03.2014 | PLE Industrial Case Study: Heidelberg Ion-Beam Therapy Center  DCU Case Study for Beam Cycle Creation  Complex Automation Engineering Software  Highly-Configurable and Runtime Adaptive  Re-Engineered as (D)SPL  ~ 200 Features (Boolean and Non-Boolean)  150% Test Model ~ 640 Transitions  SPL Implementation in C ~ 175.000 LOC Treatment Rooms Gantry DCU-Z DCU-R DCU-P

26. ES – Real-Time Systems Lab 26 | 03.03.2014 | PLE Model-Driven Development of DCU Code Parameters + User Data.h Hand-written ECA Code.c.h + States Transition Trigger.h Automata Stubs.c insert deploy generate SFT

27. ES – Real-Time Systems Lab 27 | 03.03.2014 | PLE DCU SPL Test Model Specification 150% SFT +

28. ES – Real-Time Systems Lab 28 | 03.03.2014 | PLE Thank Your For Your Attention! Integrated Model-based Testing of Continuously Evolving Software Product Lines Multi-Mechanismen Adaption für das künftige Internet