Andrea Calvagna – University of Catania - IT Angelo Gargantini – University of Bergamo – IT TAP 2009 – Zürich – 2nd July 2009 C OMBINING S ATISFIABILITY S OLVING AND H EURISTICS TO C ONSTRAINED C OMBINATORIAL I NTERACTION T ESTING

O UTLINE Background on Combinatorial Interaction Testing constraints logic approach to CIT [TAP 08] Test generation by satisfiability solving in the presence of constraints – by using Yices Heuristics - for ordering test predicates Experimental assessment C ALVAGNA & G ARGANTINI - C OMBINING S ATISFIABILITY S OLVING AND H EURISTICS TO C ONSTRAINED C OMBINATORIAL I NTERACTION T ESTING

I NTERACTION TESTING IT : interaction testing identify independent features and representative values (the inputs) test the interaction between them highly configurable systems Software for families of products (like cell phones) Sw like gcc compiler > 1400 optional features … C ALVAGNA & G ARGANTINI - C OMBINING S ATISFIABILITY S OLVING AND H EURISTICS TO C ONSTRAINED C OMBINATORIAL I NTERACTION T ESTING

H OW MUCH TO TEST ? 34 switches = 2 34 = 1.7 x possible inputs = 1.7 x tests? Every possible combination of inputs? Which interactions cause faults? C ALVAGNA & G ARGANTINI - C OMBINING S ATISFIABILITY S OLVING AND H EURISTICS TO C ONSTRAINED C OMBINATORIAL I NTERACTION T ESTING

C OMBINATORIAL APPROACH Pairwise combination instead of exhaustive Generate combinations that efficiently cover all pairs of values Extended by t-wise: test all the combinations of t values Rationale: most failures are triggered by single values or combinations of a few values. Covering pairs (triples,…) reduces the number of test cases, but reveals most faults C ALVAGNA & G ARGANTINI - C OMBINING S ATISFIABILITY S OLVING AND H EURISTICS TO C ONSTRAINED C OMBINATORIAL I NTERACTION T ESTING

E XAMPLE 3 variables with 3 values each: 3 3 = 27 possible combinations Combinatorial testing with much fewer tests Display ModeColorScreen size full-graphicsMonochromeHand-held Low resolution16-bitLaptop text-onlyTrue-colorFull-size C ALVAGNA & G ARGANTINI - C OMBINING S ATISFIABILITY S OLVING AND H EURISTICS TO C ONSTRAINED C OMBINATORIAL I NTERACTION T ESTING

T EST S UITE TestColorDisplay Mode Screen Size 1MonochromeFull- graphics Hand-held 216-bitText-onlyLaptop 3True-colorFull- graphics Hand-held 4……… C ALVAGNA & G ARGANTINI - C OMBINING S ATISFIABILITY S OLVING AND H EURISTICS TO C ONSTRAINED C OMBINATORIAL I NTERACTION T ESTING One test covers many combinations: e.g. Test 1 covers 3 pairs: (Monochrome, Full-graphics) (Monochrome, Hand-held) (Full-graphics, Hand-held) pairwise testing can be achieved by only 9 tests combinations with 10 tests; tests; …

CIT EFFECTIVENESS Experiments show that CIT is effective finds faults that traditional testing may be not able to find efficient A low degree of interaction between inputs can already discover most faults Pairwise is the most used Never with interaction > 6 C ALVAGNA & G ARGANTINI - C OMBINING S ATISFIABILITY S OLVING AND H EURISTICS TO C ONSTRAINED C OMBINATORIAL I NTERACTION T ESTING

M ETHODS FOR BUILDING CIT TEST SUITES Methods and tools IPO, AETG, PICT, TestCover, … Main goal: producing small test suites Because the problem of generating a minimum test suit for combinatorial testing is NP-complete, most methods and tools use a greedy approach C ALVAGNA & G ARGANTINI - C OMBINING S ATISFIABILITY S OLVING AND H EURISTICS TO C ONSTRAINED C OMBINATORIAL I NTERACTION T ESTING

A DDING CONSTRAINTS Constraints express relations among inputs: a valid test must not violate the constraints E.g. if the display mode is in text-only, the color is monochrome Constrained CIT Only few test generation methods support constraints, all of them only as forbidden tuples Goal: supporting expressive constraints Easier to obtain from the requirements C ALVAGNA & G ARGANTINI - C OMBINING S ATISFIABILITY S OLVING AND H EURISTICS TO C ONSTRAINED C OMBINATORIAL I NTERACTION T ESTING

L OGIC APPROACH TO CCIT [ TAP 08] formalize pairwise testing: express each pair as a corresponding logical expression, a test predicate p1 = v1 /\ p2 = v2 p1 and p2 are inputs, v1 and v2 their values Similarly, the t-wise coverage can be modeled by a set of test predicates, each of the type p1 = v1 /\ p2 = v2 /\ … /\ pt = vt C ALVAGNA & G ARGANTINI - C OMBINING S ATISFIABILITY S OLVING AND H EURISTICS TO C ONSTRAINED C OMBINATORIAL I NTERACTION T ESTING

T EST = LOGICAL MODEL test: an assignment to all the inputs of the system a test ts covers a test predicate tp if and only if it is a model of tp: ts |= tp Finding a test suite finding a model for each test predicate C ALVAGNA & G ARGANTINI - C OMBINING S ATISFIABILITY S OLVING AND H EURISTICS TO C ONSTRAINED C OMBINATORIAL I NTERACTION T ESTING

A DDING CONSTRAINTS With the constraints c1,…, cn, a test is a model of tp and the constraints c_i ts |= tp /\ c1 /\ … /\ cn the constraints become first class citizens and they can be represented by logical expressions too Finding tests becomes a problem of finding a model of a complex expression many techniques like, constraint solvers, model checkers [AFM08], SAT solver, SMT solvers … C ALVAGNA & G ARGANTINI - C OMBINING S ATISFIABILITY S OLVING AND H EURISTICS TO C ONSTRAINED C OMBINATORIAL I NTERACTION T ESTING

Y ICES [ CSL. SRI. COM ] Efficient SMT solver that decides the satisfiability of arbitrary formulas Expressive language for constraints and input models Powerful algorithms to find models C ALVAGNA & G ARGANTINI - C OMBINING S ATISFIABILITY S OLVING AND H EURISTICS TO C ONSTRAINED C OMBINATORIAL I NTERACTION T ESTING YICES Test predicate constraints Model = test

T EST S UITE GENERATION A SIMPLIFIED VERSION Test predicates Input model Test suite builder Test predicate Model= test constraints YICES C ALVAGNA & G ARGANTINI - C OMBINING S ATISFIABILITY S OLVING AND H EURISTICS TO C ONSTRAINED C OMBINATORIAL I NTERACTION T ESTING

M ONITORING AND REDUCTION Monitoring: A test covers may cover several test predicates which can be removed from the pool Reduction: at the end, a minimal set of tests that cover all the test predicates can be found by a greedy algorithm t1 t2 C ALVAGNA & G ARGANTINI - C OMBINING S ATISFIABILITY S OLVING AND H EURISTICS TO C ONSTRAINED C OMBINATORIAL I NTERACTION T ESTING

C OLLECTING Test suite builder Model = test that covers all the test predicates collected YICES Collected test predicates tp1 /\ tp2 /\ tp3 Instead of one test for every tp, collect the tps to build a conjoint C ALVAGNA & G ARGANTINI - C OMBINING S ATISFIABILITY S OLVING AND H EURISTICS TO C ONSTRAINED C OMBINATORIAL I NTERACTION T ESTING

O RDERING In which order test predicates can be collected? First simple two policies: 1. the order in which tp are generated 2. random order C ALVAGNA & G ARGANTINI - C OMBINING S ATISFIABILITY S OLVING AND H EURISTICS TO C ONSTRAINED C OMBINATORIAL I NTERACTION T ESTING

R ANDOM O RDERING non deterministic ordering Like most CIT approaches: run the method 50 times and then take the best result C ALVAGNA & G ARGANTINI - C OMBINING S ATISFIABILITY S OLVING AND H EURISTICS TO C ONSTRAINED C OMBINATORIAL I NTERACTION T ESTING GOAL: finding a deterministic ordering that performs as better as the random ordering Best result = min test suite Worst result = max test suite

D ETERMINISTIC O RDERING P OLICIES C ALVAGNA & G ARGANTINI - C OMBINING S ATISFIABILITY S OLVING AND H EURISTICS TO C ONSTRAINED C OMBINATORIAL I NTERACTION T ESTING Several criteria for ordering the test predicates can be defined Looking at the tests already generated and at the test predicates still to cover

D ETERMINISTIC O RDERING P OLICIES touchCounting explored assignments count the number of assignment var = val contained in the tps which are already contained in a test of the test suite and take the tp which has fewer assignments already touched minLeast used assignments Consider only the least used assignment in a tp maxMost used assignment Consider only the most used assignment in the tp devEven usage of assignments Consider the standard deviation in the usage count an try to keep the usage even C ALVAGNA & G ARGANTINI - C OMBINING S ATISFIABILITY S OLVING AND H EURISTICS TO C ONSTRAINED C OMBINATORIAL I NTERACTION T ESTING /c variants Consider not only the tests in the test suite (test already generated) but also the other test predicates in the collected test predicate

C ASE S TUDIES 13 specifications with constraints taken from the literature From 3 ^3 to 8.3 x 10^6 combinations From 6 to 50 constraints (forbidden tuples) C ALVAGNA & G ARGANTINI - C OMBINING S ATISFIABILITY S OLVING AND H EURISTICS TO C ONSTRAINED C OMBINATORIAL I NTERACTION T ESTING

E XPERIMENTAL R ESULTS C ALVAGNA & G ARGANTINI - C OMBINING S ATISFIABILITY S OLVING AND H EURISTICS TO C ONSTRAINED C OMBINATORIAL I NTERACTION T ESTING Among policies: One policy (touch/c) performed always better except in two cases in which touch performed better As generated: bad policy

C OMPARISON WITH RANDOM C ALVAGNA & G ARGANTINI - C OMBINING S ATISFIABILITY S OLVING AND H EURISTICS TO C ONSTRAINED C OMBINATORIAL I NTERACTION T ESTING random max min avg Bad policies: worse than average of random ordering Good policies: better than average of random ordering

C OMPARISON WITH RANDOM Best heuristicstouch/cmeaning better than minneverRandom is still a viable option if one can run the algorithms several (at least 50) times like min3/13 better than avg9/13Touch/c gives good results better than max12/13 Like max1/13If one can run the algorithms only once, a deterministic policy can guarantee better or equal results Worst than maxNeverNo risk in using touch/c C ALVAGNA & G ARGANTINI - C OMBINING S ATISFIABILITY S OLVING AND H EURISTICS TO C ONSTRAINED C OMBINATORIAL I NTERACTION T ESTING

C ONCLUSIONS Logic approach can be applied to constrained combinatorial testing It allows a more natural (and expressive) formalization of constraints Good heuristics are defined to order the test predicates and obtain small test suites Random ordering still useful in case of unlimited resources Prototype tool available at C ALVAGNA & G ARGANTINI - C OMBINING S ATISFIABILITY S OLVING AND H EURISTICS TO C ONSTRAINED C OMBINATORIAL I NTERACTION T ESTING