1. Modeling the Covering Test Problem Brahim Hnich, Steven Prestwich, and Evgeny Selensky Cork Constraint Computation Center UCC Supported by SFI

2. Overview Motivation Problem description CP models Naïve model Alternative model Integrated model Symmetry Results Local search approach

3. Motivation Software and hardware testing: important activity in product development process E.g., s/w testing consumes up to 50% of overall cost Exhaustive testing is infeasible Large number of possible test cases 10 on/off switches lead to 2 10 possible combinations

4. Motivation To be practical small number of possible test cases such that A small number (3) on/off switches get exercised in all possible 2 3 combinations The question becomes How many test cases do we need? This problem is an instance of the covering arrays problem

5. Covering array A covering array of size b is: An kxb array (k vectors of length b) Each entry is drawn from {0,…,g-1} The projection of any t coordinates contains all g t possibilities

6. Covering array: example A covering array of size 10 is: 00000 00011 00101 01001 01110 10001 10110 11010 11100 11111

7. Covering array: example A covering array of size 10 is: 00000 00011 00101 01001 01110 10001 10110 11010 11100 11111

8. Covering array: example A covering array of size 10 is: 00000 00011 00101 01001 01110 10001 10110 11010 11100 11111

9. Covering array: example A covering array of size 10 is: 00000 00011 00101 01001 01110 10001 10110 11010 11100 11111

10. Covering array: example A covering array of size 10 is: 00000 00011 00101 01001 01110 10001 10110 11010 11100 11111

11. Covering array: example A covering array of size 10 is: 00000 00011 00101 01001 01110 10001 10110 11010 11100 11111

12. Covering array: example A covering array of size 10 is: 00000 00011 00101 01001 01110 10001 10110 11010 11100 11111

13. Covering array: example A covering array of size 10 is: 00000 00011 00101 01001 01110 10001 10110 11010 11100 11111

14. Covering array: example A covering array of size 10 is: 00000 00011 00101 01001 01110 10001 10110 11010 11100 11111

15. CP models: naïve model {0,…,g-1} … … … b k M[b,k]

16. Coverage constraints For every t columns c1,…,ct For every row r For each possible combination p Introduce a Boolean variable B Use reification constraints to state that combination p is true iff Boolean variable B is set to 1 B=1 iff (M[r,c1]= 0 and M[r,c2]=0 and …. And M[r,ct]=0) Enforce that the sum of the Booleans associated with each combination p is at least 1 Too many auxiliary variables Inefficient propagation

17. Row and column symmetry The naïve matrix model has both row and column symmetry Any permutation of the rows does not affect the set of solutions Any permutation of the columns does not affect the set of solutions Reduce this symmetry: double-lex Lexicographically order the rows Lexicographically order the columns

18. CP models: alternative model …… 0{0..7}{0,…,g t -1}……… 0{0..7} ……… 1 ……… 2 ……… 3 ……… 4 ……… 5 ……… 6 ……… 7 ….…… b k choose t

19. Coverage constraints For each column State a global cardinality constraint enforcing that each value occurs between 1..b-2 t +1 times Global constraint Efficient propagation Effective pruning

20. Intersection constraints Because our columns intersect E.g. and Their values should agree 0123456701234567 0123456701234567

21. Symmetry in alternative matrix Rows are still symmetric Lexicographic ordering the rows However, columns are no longer symmetric But, the column symmetry in the naïve model is now a complex symmetry A complex combination of partial column symmetry and value symmetry Difficult to break!

22. Comparison Naïve model Coverage constraints  Symmetry Row symmetry Column symmetry Alternative model Coverage constraints Intersection constraints  Symmetry Row symmetry Complex symmetry 

23. Integrated model Naïve model Coverage constraints  Symmetry Row symmetry Column symmetry Alternative model Coverage constraints Intersection constraints  Symmetry Row symmetry Complex symmetry 

24. Integrated model Naïve model Coverage constraints  Symmetry Row symmetry Column symmetry Alternative model Coverage constraints Intersection constraints  Symmetry Row symmetry Complex symmetry  Channelling constraints  Increased number of variables 

25. Integrated model Naïve model Symmetry Row symmetry Column symmetry Alternative model Coverage constraints Channelling constraints  Increased number of variables 

26. CP models: channelling Column Column 1Column 2Column 3 0000 1001 2010 3011 4100 5101 6110 7111

27. Experiments Moderate sized instances Naïve model is very inefficient The integrated model is al least one order of magnitude faster than the alternative model New results for t=3 and k <12 Larger Instances Beyond 2000 variables Amount of search is computationally prohibitive

28. A local search approach An almost “SAT-encoding” of the integrated model The upper bound on the coverage constraints are hard to express (implied constraint) No symmetry breaking No at most one clauses … details in the paper

29. Local search: results Run model on Walksat reproduced bounds found by solver Solves several larger instances with better results than the state of the art No proof of optimality though!

30. Conclusion The covering test problem A core problem in combinatorial software testing CP approach Integrated model was able to prove optimality and improve bounds for moderate size instances Local search approach Improved bounds for larger instances Future Push the models further to solver larger instances