1. January 27, 2002 ECEN5033 University of Colorado -- Class Testing 1 Adequacy of a Class Test Suite Ideally – exhaustively test each class Practically – impossible or too hard Worth it to exhaustively test some classes with high risk Measures of adequacy to increase confidence that we have tested enough state-based coverage constraint-based coverage Constraints are pre and post conditions and the class invariant code-based coverage

2. January 27, 2002 ECEN5033 University of Colorado -- Class Testing 2 State-based coverage How many of the transitions in a state transition diagram are covered by the test suite? “Covered” = touched at least once May reveal each transition was covered but test values do not adequately cover value ranges If test cases were generated correctly from a state transition diagram with typical values and good boundary values, the test cases will achieve adequate state-based coverage If test cases were generated from pre- and post conditions, then it is useful to check them against the state transition diagram to ensure each transition is covered.

3. January 27, 2002 ECEN5033 University of Colorado -- Class Testing 3 State-based coverage: object interaction Note how operations interact w.r.t. transitions Current State Input or event ActionOutputNext State State AEvent 1Output 1 State C State BEvent 2Output 2 State C Event 3State D Test cases for the transition from State C to State D may work if State C was reached from State A but not if State C was reached from State B. “State” is a function of history.

4. January 27, 2002 ECEN5033 University of Colorado -- Class Testing 4 State-based coverage: transition pairs Concerning problem on previous page: Check that the test cases cover all pairs of transitions in the state transition diagram. In previous table, create test cases to test: Transition from State A to State C and transition from State C to State D Transition from State B to State C and transition from State C to State D In other words, the event sequences: Event 1 followed by Event 3 Event 2 followed by Event 3

5. January 27, 2002 ECEN5033 University of Colorado -- Class Testing 5 Statechart for Elevator Control

6. January 27, 2002 ECEN5033 University of Colorado -- Class Testing 6 Hierarchical statechart for Elevator Control

7. January 27, 2002 ECEN5033 University of Colorado -- Class Testing 7 Portion enlarged

8. January 27, 2002 ECEN5033 University of Colorado -- Class Testing 8 Constraint-based coverage How many pairs of pre- and post conditions have been covered? Using the technique described earlier for generating test cases from pre and post conditions, if one test case is generated to satisfy each requirement, then the test suite meets this measure

9. January 27, 2002 ECEN5033 University of Colorado -- Class Testing 9 Constraint-Based coverage: object interaction For each operation opn that is not an accessor operation, identify the operations op1, op2, etc. for which their preconditions are met when the post conditions for opn hold. That is, post condition(opn) satisfies (>=) precondition(op1), precondition(op2), etc. Then execute test cases for operation sequences opn-op1, opn-op2, etc.

10. January 27, 2002 ECEN5033 University of Colorado -- Class Testing 10 Class Diagram ElevatorElevator Doors Timer Elevator Button Up Floor Button Floor 1111 1 11 11 m 0…n Travels-to Travels-from Opens- Closes Controls Contains velocity: ±m/s position: meters floor: integer on: Boolean open: Boolean closed: Boolean jammed: Boolean time: seconds floor: integer position: meters on: Boolean Down Floor Button 1 1 Contains on: Boolean Light 1 on: Boolean 1 Turns-on 1 1 1 1 Elevator Controller 1 0…n Controls Requests-elevator-of Indicates-arrival-to

11. January 27, 2002 ECEN5033 University of Colorado -- Class Testing 11 How do we complete the class test? States do not correspond 1-1 to methods or to classes so the state transition diagram gives a different perspective We want to adequately test the class Look at the class diagram and see which classes send messages to it (“it” is the class under test) It could be difficult to create a driver to simulate each class sending msgs to the class-under- test If you only test each association in this application, it may be a weak level of testing for that class Test each state transition, better yet, the pairs

12. January 27, 2002 ECEN5033 University of Colorado -- Class Testing 12 Constraint coverage completes testing it If you test class A calling every msg it can in class B, that’s more coverage but … If you use existing classes to do that, it’s hard to get the right combination set up to make it happen Alternative: Write a driver with sequences of messages to access sequences of methods in the class you are testing. Which sequences?

13. January 27, 2002 ECEN5033 University of Colorado -- Class Testing 13 Which sequences?? Look at the post conditions of each method in the class you want to test Look at preconditions of each Find sequences where the Post(m1) >= pre(m2) the Post(m2) >= pre(m3) Send msgs to m1; m2; m3 Do this for all possible combinations

14. January 27, 2002 ECEN5033 University of Colorado -- Class Testing 14 Completion check See which associations were tested by comparing to the class diagram See which state transitions were covered this way by comparing to the statechart If some state transitions were missed or if some post/pre condition pairs or associations were missed, add a few tests

15. January 27, 2002 ECEN5033 University of Colorado -- Class Testing 15 Where do the pre and post conditions come from? Use cases?? The methods themselves? The class invariant?

16. Class Testing Testing Interactions, OATS (Lecture 3 on Class Testing) Software Engineering of Standalone Programs University of Colorado, Boulder

17. January 27, 2002 ECEN5033 University of Colorado -- Class Testing 17 Overview Class Testing Testing Interactions between objects Testing Class Hierarchies

18. January 27, 2002 ECEN5033 University of Colorado -- Class Testing 18 Testing Interactions Class Testing Testing Interactions between objects Identifying & specifying object interactions Testing object interactions Collection classes Collaborator classes Testing and Design Approach Sampling COTS testing Patterns Testing exceptions Testing Class Hierarchies

19. January 27, 2002 ECEN5033 University of Colorado -- Class Testing 19 Testing Interactions with other classes An OO program is made of a collection of objects that collaborate to solve a problem. Correct collaboration is necessary for program correctness. In this section, we assume interactions are sequential, not concurrent (that’s a subject for next semester) Focus of interaction testing: Ensure that messaging occurs correctly between objects whose classes have already been tested separately. Can happen embedded in application or in special environment using a test harness

20. January 27, 2002 ECEN5033 University of Colorado -- Class Testing 20 What is an object interaction? Object interaction is: request by a sender object to a receiver object to perform one of the receiver’s operations all of the processing performed by the receiver to complete the request including a possible reply Includes messages between an object and its components an object and other objects with which it is associated

21. January 27, 2002 ECEN5033 University of Colorado -- Class Testing 21 Object interaction impact During the processing of any single method invocation on a receiving object Multiple object interactions can occur Want to consider impact of these on the internal state of the receiving object those objects with which it has an association Impact may be “no change” changes in attribute values in 1 or more objects involved state changes in 1 or more objects including state changes of creation or deletion of objects

22. January 27, 2002 ECEN5033 University of Colorado -- Class Testing 22 Primitive and Non-Primitive Classes Interactions implied in a class spec where there are references to other objects Primitive class can be instantiated and used without needing to create any other instance Relatively few primitive classes in a program Since an OO program should model the objects in a problem and all the relationships between them, non-primitive classes are common & essential Non-primitive classes require the use of other objects in some or all of their operations Identify the classes of those objects

23. January 27, 2002 ECEN5033 University of Colorado -- Class Testing 23 Identifying Interactions Identify interactions by association relationships in the class diagram regular associations aggregation composition “Includes messages between an object and its components an object and other objects with which it is associated “

24. January 27, 2002 ECEN5033 University of Colorado -- Class Testing 24 Recognizing collaborators Collaborators may be addressed directly, e.g. using a variable name by a pointer or a reference dynamic type of the object may be different from the static type associated with the pointer pointers and references are polymorphic, bound to an instance of any number of classes Pre- and post conditions for operations in the public interface typically refer to states and/or specific attribute values of any collaborating objects

25. January 27, 2002 ECEN5033 University of Colorado -- Class Testing 25 Collection and collaboration Collection class maintains associations with instances of other classes but never actually interacts with those instances that is, it never requests a service Collaborating class Class with more extensive interactions Collection classes are less common

26. January 27, 2002 ECEN5033 University of Colorado -- Class Testing 26 Testing Collection Classes What collection classes do store references to these objects, representing one-to-many relationships between objects in a program create instances of these objects delete instances of these objects Specification refers to other objects does not refer to values computed based on those objects

27. January 27, 2002 ECEN5033 University of Colorado -- Class Testing 27 Identifying Collaborating Classes Non-primitive classes that are not collection classes Use other objects in 1 or more operations as part of their implementation When a post condition of an operation in a class’ interface refers to the state of an instance of an object and/or specifies that some attribute of that object is used or modified, that class is a collaborating class.

28. January 27, 2002 ECEN5033 University of Colorado -- Class Testing 28 Choosing testing “chunk” size Number of potential collaborations can grow too large very quickly During class test: test composing object’s interactions with its composed attributes As successive layers of aggregation are integrated, test the interaction between an object and its associated objects Defect visibility issue – If too large a “chunk” is chosen, intermediate results may be incorrect but not seen at the level of test-result verification The more complex the objects, the fewer one should integrate prior to a round of testing

29. January 27, 2002 ECEN5033 University of Colorado -- Class Testing 29 What factors increase object complexity? Number of parameters for each method Number of methods Number of state attributes in each object Trying to test a chunk that is too complex can result in defects that are hidden from the tests – for example, leaving an object in an inappropriate state that is not expected and therefore not looked at in the test.

30. January 27, 2002 ECEN5033 University of Colorado -- Class Testing 30 Location of testing emphasis What are the implications of defensive and contract design approaches when testing interactions?

31. January 27, 2002 ECEN5033 University of Colorado -- Class Testing 31 Specifying interactions Remainder of slides assume Operations defined by a class are specified by preconditions, post conditions, and class invariants contract design approach

32. January 27, 2002 ECEN5033 University of Colorado -- Class Testing 32 Contract vs. defensive approach & testing Contract More responsibility on human designer Less class-level testing due to fewer paths due to less error-checking code MORE interaction testing required to ensure human designer complied with client side of contract using precondition constraints – (are preconditions in receiver met by sender?) “illegal” for test cases to violate preconditions Defensive More responsibility on error-checking code More class-level testing due to increased paths due to more code Once the class-level error- checking code is tested, the interaction testing is simplified since there is no need to test if preconditions are met Appropriate to violate preconditions to see if receiver catches that

33. January 27, 2002 ECEN5033 University of Colorado -- Class Testing 33 Sampling Exhaustive testing: every possible test case covering every combination of values Not reasonable much of the time Want to select the ones that will find the faults in which we are the most interested Without prior information, random selection is as good as we can do A sample is a subset of a population (e.g. all possible test cases) that has been selected based on some probability distribution

34. January 27, 2002 ECEN5033 University of Colorado -- Class Testing 34 Use profile (operational profile) Gave us a way to associate each use of the system with a frequency These can be ranked by frequency Ranks can be turned into probabilities The higher the frequency of use, the larger the probability of selection

35. January 27, 2002 ECEN5033 University of Colorado -- Class Testing 35 Operational profile – stratified sample Stratified sample is a set of samples in which each sample represents a specific subpopulation Example Select test cases that exercise each component of the architecture Divide a population of tests into subsets so that a subset contains all of the tests that exercise a specific component. Sample on each subset independent of the others Need a basis for stratifying

36. January 27, 2002 ECEN5033 University of Colorado -- Class Testing 36 A basis for stratification of test cases Use the actors from the use cases Select a sample of test cases from the uses of each actor in the use cases Each actor uses some subset of the possible uses with some frequency; rank by frequency Stratifying the test case samples by each actor provides a way to increase the reliability of system Running selected tests uses the system the way it will be used in typical situations; finds defects likely to be found in typical use gives largest possible increase in reliability with least effort

37. January 27, 2002 ECEN5033 University of Colorado -- Class Testing 37 Test data generation for a range of values Generate test data first using the specification as basis Select values based on boundary values just within and just on the boundaries if contract just outside the boundaries also if defensive Select values within intervals by sampling

38. January 27, 2002 ECEN5033 University of Colorado -- Class Testing 38 Test data generation for a range of values, cont. Consider a random function over range 0.. N such as int (random( ) * N) which generates a pseudo random value between 0 and 1 according to a uniform distribution. +: many different values will be tested over many iterations Need to log actual values used in case of failure so that the failed test case can be re-created. A randomly chosen value causing failure is explicitly added to test suite and used to test the repaired software

39. January 27, 2002 ECEN5033 University of Colorado -- Class Testing 39 Systematic sampling (+ boundary value testing) Want to be able to increase level of coverage systematically. We use boundary values coupled with sampling over ranges We never omit boundary value test data

40. January 27, 2002 ECEN5033 University of Colorado -- Class Testing 40 Issues about sampling for interaction testing Sampling issues Different states can cause two objects from the same class to behave differently A class family is a subset of a larger family Possibility of combinatorial explosion in the number of test configurations

41. January 27, 2002 ECEN5033 University of Colorado -- Class Testing 41 Impact of inheritance hierarchy on s.t.d.’s Each member of the family may have different states that can cause two objects from the same class to behave differently. In families of classes, the state transition tables are related along the lines of the inheritance hierarchy. as we look down the inheritance hierarchy, there will be the same number of states or more states in the derived class as in the base class should cover the states defined for each class, emphasizing the new states added at that level in inheritance hierarchy.

42. January 27, 2002 ECEN5033 University of Colorado -- Class Testing 42 Class families as subsets of others If a class family is a subset of a larger family, after the tests are designed, they can be applied to any of the classes in the family, assuming the substitution principle has been followed during design (see next slide)

43. January 27, 2002 ECEN5033 University of Colorado -- Class Testing 43 Substitution Principle Only the following changes are allowed in defining the behavior associated with a new subclass: Preconditions for each operation must be the same or weaker – less constraining – than those of the superclass Post conditions for each operation must be the same or stronger – do at least as much as defined by the superclass Class invariant – must be the same or stronger; add more constraints

44. January 27, 2002 ECEN5033 University of Colorado -- Class Testing 44 Orthogonal array testing applied to OO Orthogonal array testing is a sampling technique to limit the explosion of test cases Define pair-wise combinations of a set of interacting objects because most faults result from interactions due to two-way interactions An orthogonal array is an array of values in which each column represents a factor.

45. January 27, 2002 ECEN5033 University of Colorado -- Class Testing 45 Orthogonal array testing applied to OO An orthogonal array is an array of values in which each column represents a factor. A factor is a variable in an experiment; in our case, a specific class family in the software system Or states of that class family Each variable takes on a certain set of values called levels (rows); in our case, specific classes or states of those classes Each cell is an instance of the class, that is, a specific object or a particular state

46. January 27, 2002 ECEN5033 University of Colorado -- Class Testing 46 OATS – orthogonal array testing system The factors (columns) are combined pair-wise rather than representing all possible combinations of the levels for the factors. Suppose we have three factors – A, B, C – and each has three levels How many possible combinations of these values are there? 3 x 3 x 3 How many pair-wise combinations? That is, how many combinations where a given level appears exactly twice? factor=class family; level=class; cell=instance

47. January 27, 2002 ECEN5033 University of Colorado -- Class Testing 47 How many pair-wise combinations? ABC 1113 2122 3131 4212 5221 6233 7311 8323 9332

48. January 27, 2002 ECEN5033 University of Colorado -- Class Testing 48 Mapping the terms of OATS to OO sw arrayfactor: class family factor: class family state level: specific class an instance of the specific class A state of that instance level: specific class an instance of the specific class A state of that instance level: specific class an instance of the specific class A state of that instance A state of that nstance

49. January 27, 2002 ECEN5033 University of Colorado -- Class Testing 49 OATS – uses a balanced design Every level of a factor appears exactly the same number of times as every other level of that factor Think of the rows of a table as test cases This is a systematic way of reducing the number of test cases. If we decide to add more later, we know which ones have not been run. Also logical – most errors are between pairs of objects rather than among several objects

50. January 27, 2002 ECEN5033 University of Colorado -- Class Testing 50 Look at elevator example (M&S, p. 230) Want to test interactions between elevator controller, elevator, and wall button-panel Suppose: Elevator can be basic, hi-speed, or express Elevator Controller can control basic, hi-speed, or express Wall button-panel can be one-button panel or two-button panel Elevator states: Moving to a floor, stopped, idle Elevator Controller states: idle, scheduling, dispatching Wall button-panel states?

51. January 27, 2002 ECEN5033 University of Colorado -- Class Testing 51 Steps from M & S, p. 230-232 1. Identify all factors (class families in this case) 2. Determine levels for each factor (states) 3. Locate a standard orthogonal array that fits the problem 1. We have 2 class families that each have 3 levels and three states 2. We have one class family that has _a_ levels and _b_ states  That’s how many possible test cases?  3 x 3 x 3 x 3 x a x b

52. January 27, 2002 ECEN5033 University of Colorado -- Class Testing 52 Excerpt from Fig. 6.16, M&S, 2 1 x 3 7 123456 1111111 2112222 3113333 4121122 5122233 6123311 7131213 8132321 9133132 10211332 11212113

53. January 27, 2002 ECEN5033 University of Colorado -- Class Testing 53 Need a way to map Fig 6.16 to this ElevatorElevator State Elev Ctrl Elev Ctrl State Wall Button Panl Wall Button PnlStte 1 2 3 4 5 6 7 8

54. January 27, 2002 ECEN5033 University of Colorado -- Class Testing 54 Mapping classes and states The class family with two levels (Wall Button Panel) maps to Column 1 Its states map to Column 2 Column 3 can have a class family with three levels; either elevator or elevator controller will do; Map Elevator Controller family to column 3 Map its states to column 4 What’s left? Map Elevator family to column 5 Map its states to column 6

55. January 27, 2002 ECEN5033 University of Colorado -- Class Testing 55 Mapping levels – Wall Button Panel For Wall Button Panel 1 = 1-button panel 2 = 2-button panel One-button Panel states (column 2) 1 = off 2 = on For two-button panel (column 2) 1 = both off 2 = 1 requesting 3 = both requesting

56. January 27, 2002 ECEN5033 University of Colorado -- Class Testing 56 Mapping levels – Elevator Controller Column 3 – elevator controller class family 1 = basic 2 = hi-speed 3 = express Column 4 – elevator controller class states 1 = idle 2 = scheduling 3 = dispatching

57. January 27, 2002 ECEN5033 University of Colorado -- Class Testing 57 Mapping levels -- Elevator Column 5 = elevator class family 1 = basic 2 = hi-speed 3 = express Column 6 = elevator class states 1 = Moving to a floor 2 = stopped 3 = idle

58. January 27, 2002 ECEN5033 University of Colorado -- Class Testing 58 Label Fig 6.16 with headers that “fit” Wall- Button Panel Wall- Button Pnl stte Elev Ctrlr Elev Ctrlr State ElevatorElevator State 1111111 2112222 3113333 4121122 5122233 6123311 7131213 8132321 9133132

59. January 27, 2002 ECEN5033 University of Colorado -- Class Testing 59 When mapped to all of Fig 6.16 Each row becomes a test case Decode the level numbers for a row in the array back to the individual lists for each factor In Fig 6.16, there are actually two more columns. We can ignore them because we don’t need them. Or we may decide we can include one more class family and its states in the future

60. January 27, 2002 ECEN5033 University of Colorado -- Class Testing 60 What about mismatches? When there is a difference in the number of levels, we can use a column that matches or exceeds the maximum What about the Wall-Button Panel at level 1 where there is only one button? That only has two states but there can be a 3 in column 2 associated with the 1 in the column 1 We interpret the 3 as if it were either 1 or 2. What do you want to emphasize in the testing? Arbitrary assignment High frequency High risk

61. January 27, 2002 ECEN5033 University of Colorado -- Class Testing 61 Other what-ifs? See M & S, p. 232

62. January 27, 2002 ECEN5033 University of Colorado -- Class Testing 62 OATS adequacy criteria Ability to vary how completely the software under test is covered Exhaustive – all combinations of all factors Minimal – only interactions between the base classes from each hierarchy are tested Random – tester haphazardly selects cases from several of the classes; not statistically random Representative – uniform sample ensures every class is tested to some level Weighted representative – adds cases to the representative approach based on relative importance or risk associated with the class