1. Combinatorial Testing Review “Combinatorial Software Testing”, Kuhn, Kacker, Lei, Hunter, IEEE Computer, Aug. 2009 Compared “traditional” software test development with simple combinatorial pairwise testing (2-way combinations) Measures: – fault detection effectiveness, – fault detection rate

2. Pairwise covering array All pairs of values appear somewhere in array

3. 10 Project Study Two teams of testers One group – ad hoc test development (common practice) – use cases based on requirements documents Second group – all pairwise combinations of system input values

4. Results Higher fault detection effectiveness (13% better) for 2-way tests Much higher fault detection per hour 2.4X

5. Questions to think about Why were results better for 2-way test? Why was fault detection rate so much higher? Input model refers to the set of parameter names and values used in testing. For a given input model, when might the “use case” scenario based test method be better than 2- way?

6. Fault distribution for various application domains What does this graph suggest about the effectiveness of pairwise (2-way) testing?

7. Oracle-free Testing with Two-layer Covering Arrays Rick Kuhn National Institute of Standards and Technology Gaithersburg, MD East Carolina University NSF Research Experiences for Undergraduates June 29, 2015

8. Some current approaches

9. New method Consider equivalence classes Example: shipping cost based on distance d and weight w, with packages 10 in a third class. Then for cost function f(d,w), f(d, 0.2) = f(d, 0.9), for equal values of d. But f(d, 0.2) ≠f(d, 5.0), because two different weight classes are involved.

10. Basic property of equivalence classes

11. Can we use this property for testing? Let’s do an example: access control. access is allowed if (1) subject is employee and time is in working hours and it’s a weekday; or (2) subject is an employee with administrative privileges; or (3) subject is an auditor and it is a weekday. Equivalence classes for time of day and day of the week time = minutes past midnight (0..0539), (0540..1020), (1021..1439). Days of the week = weekend and weekdays, designated as (1,7) and (2..6) respectively.

12. Code we want to test int access_chk() { if (emp && t >= START && t = MON && d <= FRI) return 1; else if (emp && p) return 2; else if (aud && d >= MON && d <= FRI) return 3; else return 0; }

13. Establish equivalence classes emp: boolean day: (1,7), (2,6) A1 A2 time: (0,100,539),(540,1020),(1021,1439) B1 B2 B3 priv: boolean aud: boolean emp (bool) : 0,1 day (enum) : A1,A2 time (enum): B1,B2,B3 priv (bool): 0,1 aud (bool) : 0,1

14. All of these should be equal Eq. class A1 Eq. class B1

15. These should also be equal Eq. class A2 Eq. class B1 Now we’re using class A2

16. Covering array Primary array: 0,A2,B1,1,1 1,A1,B1,0,0 0,A1,B2,1,0 1,A2,B2,0,1 0,A1,B3,0,1 1,A2,B3,1,0 One secondary array for each row Class A2 = (2,6) Class B1 = (0,539) emp: boolean day: (1,7), (2,6) A1 A2 time: (0,539),(540,1020),(1021, 1439) B1 B2 B3 priv: boolean aud: boolean 0 2 0 1 1 0 6 0 1 1 0 2 539 1 1 0 6 539 1 1

17. Run the tests Correct code output: 3333 0000 1111 0000 2222 Faulty code: if (emp && t>=START && t==END && d>=MON && d<=FRI) return 1; Faulty code output: 3333 0000 3311 0000 2222

18. What’s happening here? Input domain Incorrect boundary We simply detect inconsistency between partitions

19. Can this really work on practical code? Primary x secondary#teststotal faults detected 3-way x 3-way285x822806 4-way x 3-way970x8776022 Experiment: TCAS code (same used in earlier model checking tests) Small C module, 12 variables Seeded faults in 41 variants Results: More than half of faults detected Large number of tests -> but fully automated, no human intervention We envision this type of checking as part of the build process; can be used in parallel with static analysis, type checking

20. Prototype tool

21. Next Steps Realistic trial use Different constructions for secondary array, e.g., random values Formal analysis of applicability – range of applicability/effectiveness, limitations, special cases Determine how many faults can be detected this way Develop tools to incorporate into build process

23. Why this works for testing If Ai, Bi, Ci, etc. are equivalence classes, A1 = 0,1,2; A2 = 300,400; etc. B1 = 22,33,55; B2 = 12, 50,99 etc. C1 = 1000, 2000, 4000 …. etc. then: f(0, 22, …) = f(1, 22,1000,…) = f(2, 22,1000,…) and since eq. class B1 includes 22,33,55, then f(0, 22, …) = f(0, 33,1000,…) = f(0, 55,1000,…) = f(1, 22,1000,…) = f(1, 33,1000,…) = f(1, 55,1000,…) … and on and on … using all possible combinations of equivalent values We use a covering array of t-way combinations to make it tractable

24. Why this works for testing If Ai, Bi, Ci, etc. are equivalence classes, A1 = 0,1,2; A2 = 300,400; etc. B1 = 22,33,55; B2 = 12, 50,99 etc. C1 = 1000, 2000, 4000 …. etc. then: