Download presentation

Presentation is loading. Please wait.

Published byClinton Harrington Modified about 1 year ago

1
Combinatorial Testing Using Covering Arrays - Going Beyond Pairwise Testing Raghu Kacker, NIST Yu Lei, UTA 5/15/06

2
Combinatorial Testing Using Covering Arrays 2 Outline Introduction The IPO Strategy FireEye: An N-Way Testing Tool Related Work Conclusion

3
Combinatorial Testing Using Covering Arrays 3 Why Software Testing? Modern society is increasingly dependent on the quality of software systems. A NIST study reports that software failure costs the US economy billions of dollars every year. Testing (or dynamic analysis) is the most widely used approach to ensuring software quality Other approaches like static analysis and formal verification are more difficult to apply and do not seem to scale

4
Combinatorial Testing Using Covering Arrays 4 Testing Process The testing process consists of three stages: Test Generation: Generate test data For model-based testing, a model (or abstraction) of the system has to be built. Test Execution: Test setup and the actual test runs Test Evaluation: Check if the output is in line with expectations

5
Combinatorial Testing Using Covering Arrays 5 All About Trade-Off Testing is labor intensive and can be very costly estimated to often consume more than 50% of the development cost Exhaustive testing is often impractical due to resource constraints From a certain perspective, testing is basically about making a good trade-off between test effort and quality assurance.

6
Combinatorial Testing Using Covering Arrays 6 Combinatorial Testing Generates tests from an input parameter model by combining the values of the parameters. Requires lightweight specification and no knowledge about the implementation structure Can be virtually applied to any software and at different levels of abstraction Can be implemented as a push-button feature Usually performed to achieve t-way coverage, i.e. guarantees to cover every t-way interaction Motivation: Not every parameter contributes to every fault Can dramatically reduce the number of tests while still preserving important fault detection capabilities.

7
Combinatorial Testing Using Covering Arrays 7 The ASTUCA Project Develop new methods and tools for efficient t-way testing for up to 6-way testing Create new algorithms for efficient test set construction Provide adequate support for parameter relations and constraints Explore integration with other software testing tools Conduct empirical evaluation of t-way testing in an industrial setting A collaborative effort among the following institutions: The US National Institute of Standards and Technology George Mason University The University of Texas at Arlington

8
Combinatorial Testing Using Covering Arrays 8 State-of-the-Art Existing work has mainly focused on pairwise testing Many failures are caused by the interaction involving more than two parameters For certain software, pairwise testing discovers a relatively low percentage of faults e.g., For the RAX in NASA Deep Space 1 mission, pairwise testing only discovers 54 percent of interface faults, and 47 percent of engine faults. Increased coverage leads to a higher level of assurance Many applications, e.g., security protocols, have strict requirements on test coverage

9
Combinatorial Testing Using Covering Arrays 9 Technical Challenges The computational complexity for t-way testing grows rapidly as the value of t increases New algorithms must strike a balance between the time and space requirements and the optimality of the resulting test sets The number of tests also grows rapidly as the value of t increases Impractical to manually execute and inspect the results of a large number of test runs Test generation, test execution, and test evaluation must integrate together to enable test automation

10
Combinatorial Testing Using Covering Arrays 10 Terminology N-Way Test set -> N-Way Covering array Tests -> Rows Parameters -> Factors or Columns Values -> Levels

11
Combinatorial Testing Using Covering Arrays 11 Outline Introduction The IPO Strategy FireEye: An N-Way Testing Tool Related Work Conclusion

12
Combinatorial Testing Using Covering Arrays 12 The Framework (1) Builds a t-way test set in an incremental manner A t-way test set is first constructed for the first t parameters, Then, the test set is extended to generate a t-way test set for the first t + 1 parameters The test set is repeatedly extended for each additional parameter. Two steps involved in each extension for a new parameter: Horizontal growth: extends each existing test by adding one value of the new parameter Vertical growth: adds new tests, if necessary

13
Combinatorial Testing Using Covering Arrays 13 The Framework (2) Strategy In-Parameter-Order begin /* for the first t parameters p 1, p 2, …, p t */ T := {(v 1, v 2, …, v t ) | v 1, v 2, …, v t are values of p 1, p 2, …, P k, respectively} if n = t then stop; /* for the remaining parameters */ for parameter p i, i = t + 1, …, n do begin /* horizontal growth */ for each test (v 1, v 2, …, v i-1 ) in T do replace it with (v 1, v 2, …, v i-1, v i ), where v i is a value of p i /* vertical growth */ while T does not cover all the interactions between p i and each of p 1, p 2, …, p i-1 do add a new test for p 1, p 2, …, p i to T; end

14
Combinatorial Testing Using Covering Arrays 14 Example (1) Consider a system with the following parameters and values: parameter A has values A1 and A2 parameter B has values B1 and B2, and parameter C has values C1, C2, and C3

15
Combinatorial Testing Using Covering Arrays 15 Example (2) A B A1 B1 A1 B2 A2 B1 A2 B2 A B C A1 B1 C1 A1 B2 C2 A2 B1 C3 A2 B2 C1 ABC A1B1C1 A1B2C2 A2B1C3 A2B2 C1 A2B1C2 A1B2C3 Horizontal GrowthVertical Growth

16
Combinatorial Testing Using Covering Arrays 16 Comparison to AETG (1) A commercial tool developed by Telcordia, and protected by a US patent Starts with an empty set and adds one (complete) test at a time Each test is locally optimized to cover the most number of missing pairs: Generate a random order of the parameters Use a greedy algorithm to construct a test that covers the most uncovered pairs Repeat the above two steps for a given number of times (suggested 50), and select the best one

17
Combinatorial Testing Using Covering Arrays 17 Comparison to AETG (2) ABC A1B1C1 ABC A1B1C1 A1B2C2 ABC A1B1C1 A1B2C2 A2B1C3 A2B2 C1 A2B1C2 A1B2C3 Adds the 1st testAdds the 2nd testAdds the last test ABCABC

18
Combinatorial Testing Using Covering Arrays 18 IPO vs AETG IPO is deterministic, whereas AETG is inherently non-deterministic IPO has a lower order of complexity, both in terms of time and space, than AETG IPO constructs a test set one parameter at a time and in a more incremental nature. The results generated by IPO are still competitive to those generated by AETG. IPO is more flexible than AETG IPO can take a test set for a subsystem (i.e., for a subset of parameters) and then extend it to a complete set for the entire system

19
Combinatorial Testing Using Covering Arrays 19 Outline Introduction The IPO Strategy FireEye: A Prototype Tool Related Work Conclusion

20
Combinatorial Testing Using Covering Arrays 20 Major Features Uses Java as the programming language Relatively easier to program, leading to reduced development time and ease of maintenance Supports the concept of “write-once-run-everywhere” Data structures are carefully designed to optimize the runtime performance A hierarchical structure is used to manage the possible interactions Allows an incomplete test set to be extended to a complete one Add new tests or parameters, if necessary, to achieve t- way coverage

21
Combinatorial Testing Using Covering Arrays 21 Initial Results (1) [System] Name: Test Configuration for TCAS [Parameter] -- only compare with MINSEP and MAXALTDIFF Cur_Vertical_Sep : 299, 300, 601 High_Confidence : TRUE, FALSE Two_of_Three_Reports_Valid : TRUE, FALSE -- Low and High, only compare with Other_Tracked_Alt Own_Tracked_Alt : 1, 2 Other_Tracked_Alt : 1, 2 -- only compare with OLEV Own_Tracked_Alt_Rate : 600, 601 Alt_Layer_Value : 0, 1, 2, 3 -- compare with each other (also see NOZCROSS) and with ALIM Up_Separation : 0, 399, 400, 499, 500, 639, 640, 739, 740, 840 Down_Separation : 0, 399, 400, 499, 500, 639, 640, 739, 740, 840 Other_RAC : NO_INTENT, DO_NOT_CLIMB, DO_NOT_DESCEND Other_Capability : TCAS_TA, OTHER Climb_Inhibit : TRUE, FALSE

22
Combinatorial Testing Using Covering Arrays 22 Initial Results (2) All the experiments are performed on a desktop with 1.2GHZ CPU and 1GB memory.

23
Combinatorial Testing Using Covering Arrays 23 Outline Introduction The IPO Strategy FireEye: An N-Way Testing Tool Related Work Conclusion

24
Combinatorial Testing Using Covering Arrays 24 Classification Search-Based methods that are mainly developed by computer scientists AETG (from Telcordia), TCG (from JPL/NASA), DDA (from ASU), PairTest Algebraic methods that are mainly developed by mathematicians Orthogonal Arrays Recursive Construction

25
Combinatorial Testing Using Covering Arrays 25 Orthogonal Arrays Orthogonal arrays can be constructed very fast and are always optimal Any extra test will cause a pair to be covered for more than once However, there are several limitations: Orthogonal arrays do not always exist Every parameter must have the same number v of values Existing methods often require v be a prime power. Every t-way interaction must be covered at the same number of times

26
Combinatorial Testing Using Covering Arrays 26 Recursive Construction Covering arrays are a more general structure, which requires every t-way interaction be covered at least once Constructing a covering array from one or more covering arrays with smaller parameter sets Recursive construction can be fast, but it also has restrictions on the number of parameters and the domain sizes

27
Combinatorial Testing Using Covering Arrays 27 Search-Based vs Algebraic Methods Search-based methods: Advantages: no restrictions on the input model, and very flexible, e.g., relatively easier to support parameter relations and constraints Disadvantages: explicit search takes time, the resulting test sets are not optimal Algebraic methods: Advantages: very fast, and often produces optimal results Disadvantages: limited applicability, difficult to support parameter relations and constraints The advantages and disadvantages of the two types of methods seem to complement with each other

28
Combinatorial Testing Using Covering Arrays 28 Outline Introduction The IPO Strategy FireEye: An N-Way Testing Tool Related Work Conclusion

29
Combinatorial Testing Using Covering Arrays 29 Conclusion Combinatorial testing is a well-defined problem and has been used widely in practice. The IPO strategy has a lower order of complexity than AETG, and still produces competitive results. Algebraic methods, if applicable, are fast and can be optimal, whereas search-based algorithm are very flexible. Going beyond 2-way testing presents challenges and opportunities to the area of combinatorial testing.

30
Combinatorial Testing Using Covering Arrays 30 References 1. Boroday S. Y. and Grunskii I. S., “Recursive generation of locally complete tests,” Cybernetics and Systems Analysis 28 (1992), 20-25. 2. K. A., Bush, “Orthogonal arrays of index unity,” Annals of Mathematical Statistics, 23 (1952), 426-434. 3. D. M. Cohen, S. R. Dalal, M. L. Fredman, and G. C. Patton, “The AETG System: An Approach to Testing Based on Combinatorial Design,” IEEE Transactions on Software Engineering, 1997, Vol. 23, No. 7.The AETG System: An Approach to Testing Based on Combinatorial Design 4. M. B. Cohen, C. J. Colbourn, P. B. Gibbons and W. B. Mugridge, “Constructing test suites for interaction testing,” In Proc. of the Intl. Conf. on Software Engineering, (ICSE 2003), 2003, pp. 38-48, Portland. 5. R. Kuhn, D. Wallace, A. Gallo, “Software Fault Interactions and Implications for Software Testing,” IEEE Transactions on Software Engineering, June 2004, Vol. 30, No. 6.Software Fault Interactions and Implications for Software Testing 6. Alan Hartman, Leonid Raskin, “Problems and algorithms for covering arrays,” Discrete Mathematics 284(1-3): 149-156 (2004)Leonid RaskinDiscrete Mathematics 284 7. Y. Lei and K. C. Tai, “In-parameter-order: a test generation strategy for pairwise testing,” Proceedings Third IEEE Intl. High-Assurance Systems Engineering Symosium., 1998, pp. 254-261.In-parameter-order: a test generation strategy for pairwise testing 8. K. C. Tai and Y. Lei, “A Test Generation Strategy for Pairwise Testing,” IEEE Transactions on Software Engineering, 2002, Vol. 28, No. 1.A Test Generation Strategy for Pairwise Testing

Similar presentations

© 2017 SlidePlayer.com Inc.

All rights reserved.

Ads by Google