1. January 27, 2002 ECEN5033 University of Colorado -- Class Testing 1 Specifying interactions Remainder of slides assume Operations defined by a class are specified by preconditions, post conditions, and class invariants contract design approach

2. January 27, 2002 ECEN5033 University of Colorado -- Class Testing 2 Contract vs. defensive approach & testing Contract More responsibility on human designer Less class-level testing due to fewer paths due to less error-checking code MORE interaction testing required to ensure human designer complied with client side of contract using precondition constraints – (are preconditions in receiver met by sender?) “illegal” for test cases to violate preconditions Defensive More responsibility on error-checking code More class-level testing due to increased paths due to more code Once the class-level error- checking code is tested, the interaction testing is simplified since there is no need to test if preconditions are met Appropriate to violate preconditions to see if receiver catches that

3. January 27, 2002 ECEN5033 University of Colorado -- Class Testing 3 Sampling Exhaustive testing: every possible test case covering every combination of values Not reasonable much of the time Want to select the ones that will find the faults in which we are the most interested Without prior information, random selection is as good as we can do A sample is a subset of a population (e.g. all possible test cases) that has been selected based on some probability distribution

4. January 27, 2002 ECEN5033 University of Colorado -- Class Testing 4 Use profile (operational profile) Gave us a way to associate each use of the system with a frequency These can be ranked by frequency Ranks can be turned into probabilities The higher the frequency of use, the larger the probability of selection

5. January 27, 2002 ECEN5033 University of Colorado -- Class Testing 5 Operational profile – stratified sample Stratified sample is a set of samples in which each sample represents a specific subpopulation Example Select test cases that exercise each component of the architecture Divide a population of tests into subsets so that a subset contains all of the tests that exercise a specific component. Sample on each subset independent of the others Need a basis for stratifying

6. January 27, 2002 ECEN5033 University of Colorado -- Class Testing 6 A basis for stratification of test cases Use the actors from the use cases Select a sample of test cases from the uses of each actor in the use cases Each actor uses some subset of the possible uses with some frequency; rank by frequency Stratifying the test case samples by each actor provides a way to increase the reliability of system Running selected tests uses the system the way it will be used in typical situations; finds defects likely to be found in typical use gives largest possible increase in reliability with least effort

7. January 27, 2002 ECEN5033 University of Colorado -- Class Testing 7 Test data generation for a range of values Generate test data first using the specification as basis Select values based on boundary values just within and just on the boundaries if contract just outside the boundaries also if defensive Select values within intervals by sampling

8. January 27, 2002 ECEN5033 University of Colorado -- Class Testing 8 Test data generation for a range of values, cont. Consider a random function over range 0.. N such as int (random( ) * N) which generates a pseudo random value between 0 and 1 according to a uniform distribution. +: many different values will be tested over many iterations Need to log actual values used in case of failure so that the failed test case can be re-created. A randomly chosen value causing failure is explicitly added to test suite and used to test the repaired software

9. January 27, 2002 ECEN5033 University of Colorado -- Class Testing 9 Systematic sampling (+ boundary value testing) Want to be able to increase level of coverage systematically. We use boundary values coupled with sampling over ranges We never omit boundary value test data

10. January 27, 2002 ECEN5033 University of Colorado -- Class Testing 10 Issues about sampling for interaction testing Sampling issues Different states can cause two objects from the same class to behave differently A class family is a subset of a larger family Possibility of combinatorial explosion in the number of test configurations

11. January 27, 2002 ECEN5033 University of Colorado -- Class Testing 11 Impact of inheritance hierarchy on s.t.d.’s Each member of the family may have different states that can cause two objects from the same class to behave differently. In families of classes, the state transition tables are related along the lines of the inheritance hierarchy. as we look down the inheritance hierarchy, there will be the same number of states or more states in the derived class as in the base class should cover the states defined for each class, emphasizing the new states added at that level in inheritance hierarchy.

12. January 27, 2002 ECEN5033 University of Colorado -- Class Testing 12 Class families as subsets of others If a class family is a subset of a larger family, after the tests are designed, they can be applied to any of the classes in the family, assuming the substitution principle has been followed during design (see next slide)

13. January 27, 2002 ECEN5033 University of Colorado -- Class Testing 13 Substitution Principle a reminder Only the following changes are allowed in defining the behavior associated with a new subclass: Preconditions for each operation must be the same or weaker – less constraining – than those of the superclass Post conditions for each operation must be the same or stronger – do at least as much as defined by the superclass Class invariant – must be the same or stronger; add more constraints

14. January 27, 2002 ECEN5033 University of Colorado -- Class Testing 14 Orthogonal array testing applied to OO Orthogonal array testing is a sampling technique to limit the explosion of test cases Define pair-wise combinations of a set of interacting objects because most faults result from interactions due to two-way interactions An orthogonal array is an array of values in which each column represents a factor.

15. January 27, 2002 ECEN5033 University of Colorado -- Class Testing 15 Why PAIR-wise Goal: choose a fairly small subset of test cases that finds a great many defects. (duh!) How? Test all pairs of whatever it is that is varying. Pairs are fewer: Suppose 4 input parameters, each of which could take on 3 values. Number of combinations is 3^4 = 81. Can cover all pairwise input combinations in 9 tests 13 input params, each of which as 3 values; number of comb.’s is 3^13. Can cover all pairwise input combinations in 15 tests.

16. January 27, 2002 ECEN5033 University of Colorado -- Class Testing 16 More on pairwise AT&T wanted to test a LAN-based electronic mail system. Original plan was to develop and execute 1500 test cases. Switched to pairwise testing – 50 percent less effort and found 28% more defects. NIST reviewed 15 years of defect data on recalled medical devices. They concluded: 98% of the reported software flaws would have been detected by testing all pairs of parameter settings. Why does it work so well?

17. January 27, 2002 ECEN5033 University of Colorado -- Class Testing 17 Orthogonal array testing applied to OO An orthogonal array is an array of values in which each column represents a factor. A factor is a variable in an experiment; in our case, a specific class family in the software system Or states of that class family Each variable takes on a certain set of values called levels (rows); in our case, specific classes or states of those classes Each cell is an instance of the class, that is, a specific object or a particular state

18. January 27, 2002 ECEN5033 University of Colorado -- Class Testing 18 What are orthogonal arrays Consider numbers 1 and 2 – how many pair combinations are there? An orthogonal array is a 2-dimensional array of numbers that has this property: Choose any two columns of the array; all the pairwise combinations of its values will occur in every pair of columns.

19. January 27, 2002 ECEN5033 University of Colorado -- Class Testing 19 Example Subscript is number of rows in the array. Superscript is number of columns. 2 says each value is a 1 or a 2.

20. January 27, 2002 ECEN5033 University of Colorado -- Class Testing 20 another

21. January 27, 2002 ECEN5033 University of Colorado -- Class Testing 21 Mixed arrays

22. January 27, 2002 ECEN5033 University of Colorado -- Class Testing 22 How to use Identify the variables Determine the number of choices for each variable Locate an orthogonal array with a column for each variable and values within the columns corresponding to the number of choices. Map the test problem onto the array Construct test cases with these combinations

23. January 27, 2002 ECEN5033 University of Colorado -- Class Testing 23 Sample – see excerpt

24. January 27, 2002 ECEN5033 University of Colorado -- Class Testing 24

25. January 27, 2002 ECEN5033 University of Colorado -- Class Testing 25 When mapped to all of array Each row becomes a test case Decode the level numbers for a row in the array back to the individual lists for each factor Sometimes, you don’t need all of the columns We can ignore them because we don’t need them. Or we may decide we can include one more class family and its states in the future

26. January 27, 2002 ECEN5033 University of Colorado -- Class Testing 26 What about mismatches? When there is a difference in the number of levels, we can use a column that matches or exceeds the maximum What about a class with 2 states but there can be a 3 in the array we picked Interpret the 3 as if it were either 1 or 2. What do you want to emphasize in the testing? Arbitrary assignment High frequency High risk

27. January 27, 2002 ECEN5033 University of Colorado -- Class Testing 27 Other what-ifs? See the Copeland excerpt on Class Lecture materials page of web site

28. January 27, 2002 ECEN5033 University of Colorado -- Class Testing 28 OATS adequacy criteria Ability to vary how completely the software under test is covered Exhaustive – all combinations of all factors Minimal – only interactions between the base classes from each hierarchy are tested Random – tester haphazardly selects cases from several of the classes; not statistically random Representative – uniform sample ensures every class is tested to some level Weighted representative – adds cases to the representative approach based on relative importance or risk associated with the class