1. Software Testing Sudipto Ghosh CS 406 Fall 99 November 23, 1999

2. 11/23/99CS 406 Testing2 Learning Objectives Coverage Principle Saturation principle Mutation testing Test case generation and tool support Testing OO programs Testing distributed and real-time systems

3. 11/23/99CS 406 Testing3 Principle underlying test assessment Uniform principle that underlies test assessment throughout the testing process Coverage Principle Result of intensive research at Purdue and other research groups in software testing

4. 11/23/99CS 406 Testing4 The coverage principle To formulate and understand the coverage principle, we need to understand: coverage domains coverage elements Coverage domain: finite domain, related to the program under test Coverage element: individual elements in the domain

5. 11/23/99CS 406 Testing5 The coverage principle (contd) Requirements Functions Statements Branches : All uses All paths Coverage DomainsCoverage Elements

6. 11/23/99CS 406 Testing6 The coverage principle Measuring test adequacy and improving a test set against a set of well-defined, increasingly strong, coverage domains leads to improved confidence in the reliability of the system under test.

7. 11/23/99CS 406 Testing7 The coverage principle (contd) Note the following properties of a coverage domain: It is related to the program under test. It it finite. It may come from program requirements, related to the inputs and outputs. Exercise: Give examples. It may come from the program code. Exercise: Give examples. It aids in measuring test adequacy and the progress made in testing.

8. 11/23/99CS 406 Testing8 Using coverage criteria Improving coverage improves our confidence in the correct functioning of the program under test Given a program P, and a test set T, suppose that T is adequate w.r.t. a coverage criterion C. Does that mean that P is error free? Obviously …???

9. 11/23/99CS 406 Testing9 Test effort Several measures available Size of the test set T A test set with a larger number of test cases corresponds to higher effort than one with less number of test cases

10. 11/23/99CS 406 Testing10 Error detection effectiveness Each coverage criterion has its error detection ability. How do you measure effectiveness? 1Fraction f of faults guaranteed to be revealed by a test T that satisfies C. 2At least a fraction f of the faults in P will be revealed by test T that satisfies C. No absolute measure of effectiveness of any given coverage criterion for a general class of programs and arbitrary test sets.

11. 11/23/99CS 406 Testing11 Error detection effectiveness (contd) Empirical studies conducted by researchers give us an idea of the relative goodness of various coverage criteria. For a variety of criteria, we can make a statement like: Criterion C 1 is definitely better than C 2. Sometimes, we can say: Criterion C 1 is probably better than C 2.

12. 11/23/99CS 406 Testing12 Error detection effectiveness (contd) Such information allows us to construct a hierarchy of coverage criteria This hierarchy helps in organizing and managing testing.

13. 11/23/99CS 406 Testing13 The saturation effect The rate at which new faults are discovered reduces as test adequacy w.r.t. a finite coverage domain increases. The rate reduces to zero when the coverage domain is exhausted. Coverage Rate of fault discovery 01

14. 11/23/99CS 406 Testing14 Saturation effect: Fault view Testing Effort Remaining Faults 0 N M Functional tfstfs tfetfe tdstds t df e tmetme

15. 11/23/99CS 406 Testing15 Saturation effect: Reliability view FUNCTIONAL, DECISION, DATAFLOW AND MUTATION COVERAGE PROVIDE VARIOUS TEST EVALUATION CRITERIA Reliability Testing Effort True reliability (R) Estimated reliability (R’) Saturated region Mutation Dataflow Decision Functional RmRm R df RdRd RfRf R’ f R’ d R’ df R’ m tfstfs tfetfe tdstds tdetde t df s t df e tmstms tfetfe

16. 11/23/99CS 406 Testing16 Mutation testing Fundamentally different from previous approaches. 1Take a program. 2Create mutants by making simple changes. 3Goal of testing is to make sure that each mutant produces an output different from the output of the original program.

17. 11/23/99CS 406 Testing17 Mutant operators Makes a small unit change in the program Replace an arithmetic operator by another Replace a constant by another Change an array reference Change the label for a goto statement Replace a variable by a special value (like 0) First-order mutant obtained by making exactly one change. Need to model real faults.

18. 11/23/99CS 406 Testing18 Basis for mutation testing Competent programmer hypothesis Coupling effect

19. 11/23/99CS 406 Testing19 Competent programmer hypothesis Programmers are generally competent and don’t create programs at random. A programmer produces a program that is very close to a correct program. A correct program can be constructed from an incorrect program with minor changes in the program.

20. 11/23/99CS 406 Testing20 Coupling effect Test cases that distinguish programs with minor differences with each other are so sensitive that they will also distinguish programs with more complex differences.

21. 11/23/99CS 406 Testing21 Mutation testing You are building up your test set T with test cases. Generate mutants for P. Suppose that there are N mutants. Execute each mutant and P on each test case in T. Find how many mutants were distinguished. These mutants are called dead. Let there be D dead mutants.

22. 11/23/99CS 406 Testing22 Mutation testing For each mutant that was not distinguished (called, live), find out how many are equivalent to P, i.e. they will always produce the same output as P. Let E be the number of equivalent mutants. The mutation score is Add more test cases to T and continue testing until the mutation score is 1.

23. 11/23/99CS 406 Testing23 Using mutation testing to improve test sets Empirically shown to be stronger than c- uses and p-uses. Suppose that you have used some coverage criteria and you have a test set T adequate w.r.t. the criteria. You have removed all the errors found during testing and now the program passes the tests in T. You can use mutation testing to improve T.

24. 11/23/99CS 406 Testing24 Performance Serious problem with mutation testing Number of mutants generated with the first-order operators is large Fortran: For a program with L lines of code, the total number of mutants is L 2 All have to be compiled and executed Tester has to spend considerable time to decide if some mutants are equivalent. This is an undecidable problem.

25. 11/23/99CS 406 Testing25 Test case generation and tool support Once you have decided a coverage criterion, there are two problems: Decide if a test set is adequate w.r.t. the criterion Generate a set of test sets Difficult to do both tasks manually “Feasibility” of a path is an undecidable problem

26. 11/23/99CS 406 Testing26 Test generation Fully automated tool not possible Tools aid the tester Ways: Tool randomly generates tests until the criterion is satisfied Generates a lot of unnecessary tests

27. 11/23/99CS 406 Testing27 Test generation Testing is done manually by performing structural testing iteratively Tools gives feedback on what paths are left Static dataflow tools can be used to decide what input values should be given for that path to be executed Symbolic evaluation techniques can be used Ingenuity and creativity of the tester is important Often the criteria is weakened (< 100%)

28. 11/23/99CS 406 Testing28 Testing tools Usual steps: Instrument the program with probes Execute the program with test cases Analyze the results of the probe data

29. 11/23/99CS 406 Testing29 Testing object oriented programs Not really different if you are using black- box testing Why is white-box testing different for OO? White-box likely to be at unit level. Basic unit is “Class” Fundamental differences between testing classes and functions Testing methods individually and functions may be similar

30. 11/23/99CS 406 Testing30 Testing OO Programs Testing all methods separately is not same as testing the class Class cannot be tested directly; only an instance can be tested. How do you test abstract classes? Control flow characterized by message passing among objects, no sequential control flow within a class like in a function Different approach required.

31. 11/23/99CS 406 Testing31 Testing OO Programs State associated with object also influences the path of execution; methods of a class can communicate among themselves using the state. for function, arguments and global variables determine the path of execution Inheritance structure of inheritance hierarchy kind of inheritance

32. 11/23/99CS 406 Testing32 Problems with inheritance Structure (lattice or tree) Lattice - a class at the bottom may inherit some features more than once Classes are more complex and error-prone Kind (strict or redefining allowed) Dynamic binding of operations, we don’t know until run-time whether the operation is defined

33. 11/23/99CS 406 Testing33 Techniques Exercise the routines provided by an object with the goal of uncovering errors in the implementation of the routines or state of the object or both Find patterns of method invocation of the object under test with different arguments. State-based testing tests for interaction by changing the state of the object under which the methods are tested.

34. 11/23/99CS 406 Testing34 Testing distributed software Uniprocessor environment assumptions do not hold Global environment Deterministic execution Sequential execution of instructions Insertion of debugging statements do not alter execution of product Timing-dependent faults Need special techniques history files

35. 11/23/99CS 406 Testing35 Issues in testing real-time systems “Real-world” inputs No control over timing No control over order Similar problems as with distributed software Often no human operators More robustness required high fault-tolerance Difficulty in setting up test cases

36. 11/23/99CS 406 Testing36 Testing real-time systems Structure analysis Correctness proofs Systematic testing Statistical testing Simulation