1. Nov. 17, 2005 1 Verification and Validation F Verification: checks that the program conforms to its specification. –Are we building the product right? F Validation: checks that the program as implemented meets the expectations of the user. –Are we building the right product?

2. Nov. 17, 2005 2 The Test Plan F A ``living document''. It is born with the system and evolves as the system evolves. It is what the key decision makers use to evaluate the system. F User objectives. F System Description and traceability Matrices. F Special Risk Elements. F Required Characteristics-- operational and technical. F Critical Test Issues -- operational and technical.

3. Nov. 17, 2005 3 Management Plan F Integrated Schedule F Roles and Responsibilities F Resources and Sharing

4. Nov. 17, 2005 4 Verification Outline F Verification to Date F Previous Results F Testing Planned – Unresolved Issues – Issues arising during this phase – Scope of Planned tests – Test Objectives F Special Resources F Test Articles

5. Nov. 17, 2005 5 Validation Outline F Validation to Date F Previous Results F Testing Planned – Unresolved Issues – Issues arising during this phase – Scope of Planned tests – Test Objectives F Special Resources F Test Articles

6. Nov. 17, 2005 6 Test Results and Traceability F Test Procedures F Test Reporting F Development Folders

7. Nov. 17, 2005 7 Static Verification F Program inspection F Formal method

8. Nov. 17, 2005 8 Code Walkthroughs and Inspections F Code reviews lead to rapid and thorough fault detection –Up to 95 percent reduction in maintenance costs

9. Nov. 17, 2005 9 Verification and Proofs of Correctness F Formally specify the desired functionality, then verify that the program is a correct implementation of the specification.

10. Nov. 17, 2005 10 Hoare's Rules F Program fragments and assertions are composed into triples {P}S{Q} –where P is the precondition assertion, Q is the postcondition assertion, and S is program statements. – Interpretation: If P is true before S is executed, then when S terminates, Q is satisfied.

11. Nov. 17, 2005 11 Proofs of Correctness F Partial correctness: if the precondition is true, and the program terminates, then the postcondition is satisfied. F Total correctness: is partial correctness, plus a proof of termination.

12. Nov. 17, 2005 12 The Assignment Rule {P} x:=f {Q} F where P and Q are the same, except that all the occurrences of x in P have been replaced by f. F Backward substitution F {x=5} x = x + 1 { x = 6} F {z > y + 50} x = z – 43 { x > y + 7}

13. Nov. 17, 2005 13 Rule for Sequencing Statements {F 1 } S 1 ; S 2 {F 3 } {F1}S1{F2}, {F2}S2{F3} {F1}S1;S2{F3}

14. Nov. 17, 2005 14 Rule for Conditions and Loops {P & C} S1 {Q}, {P & ~C} S2 {Q} {P} if C then S1 else S2 endif {Q} {I & C} S { I } {I} while C do S od {I & ~C}

15. Nov. 17, 2005 15 Cleanroom (contd) F Prototype automated documentation system for the U.S. Naval Underwater Systems Center F 1820 lines of FoxBASE –18 faults were detected by “functional verification” –Informal proofs were used –19 faults were detected in walkthroughs before compilation –There were NO compilation errors –There were NO execution-time failures

16. Nov. 17, 2005 16 Software Testing F Software Requirements Specifications –Describes the expected runtime behaviors of the software. F A Test Plan –Describe how to test each behavior. F The software (source code or executable)

17. Nov. 17, 2005 17 Testing F Failure: –The departure of program operation from user requirements. F Fault: – A defect in a program that may cause a failure. F Error: –Human action that results in software containing a fault.

18. Nov. 17, 2005 18 Types of Faults F algorithmic faults F computation and precision faults F documentation faults F stress or overloaded faults F capacity or boundary faults F timing or coordination faults F throughput or performance faults

19. Nov. 17, 2005 19 IBM Orthogonal Defect Classification F Function : fault that affects capability, end-user interfaces, product interface with hardware architecture, or global data structure. F Interface : fault in interfacing with other components or drives via calls, macros, control blocks, or parameter lists. F Checking : fault in program login that fails to validate data and values properly before they are used. F Assignment : fault in data structure or code block initialization. F Timing/serialization : fault that involves timing of shared and real- time resources. F Build/package/merge : fault that occurs because of problems in repositories, management changes or version control. F Documentation : fault that affects publications and maintenance notes F Algorithm : fault involving efficiency or correctness of algorithm or data structure but not design.

20. Nov. 17, 2005 20 The Testing Process F Unit testing F Component testing F Integration testing F System testing F Acceptance testing

21. Nov. 17, 2005 21 Testing Strategies F Top-down testing F Bottom-up testing F Thread testing F Stress testing F Back-to-back testing

22. Nov. 17, 2005 22 Traditional Software Testing Techniques F Black box testing –program specifications : functional testing, graph-based testing –operational profile: random testing, partition testing –Boundary value analysis F White box testing –statement coverage – branch coverage – data flow coverage – path coverage F Others –Stress testing –Back-to back testing

23. Nov. 17, 2005 23 Graph-Based Testing F Transaction flow modeling F Finite state modeling F Data flow modeling F Timing modeling

24. Nov. 17, 2005 24 Partition Testing F If an input condition specifies a range, one valid and two invalid equivalence classes are defined. F If an input condition requires a specific value, one valid and two invalid equivalence classes are defined. F If an input condition specifies a member of a set, one valid and one invalid class are defined. F If an input condition is Boolean, one valid and one invalid class are defined.

25. Nov. 17, 2005 25 Boundary Value Analysis F Select test cases on or just to one side of the boundary of equivalence classes F If an input condition specifies a range bounded by values a and b, test cases should be designed with values a and b, just above and just below a and b, respectively. F If an input condition specifies a number of values, test cases should be developed that exercise the minimum and maximum numbers. Values just above and below minimum and maximum are also tested. F Apply guidelines 1 and 2 to output conditions. F If internal program data structures have prescribed boundaries, be certain to design a test case to exercise the data structure at its boundary

26. Nov. 17, 2005 26 Partition/Equivalence Testing F Any one member of an equivalence class is as good a test case as any other member of the equivalence class F Range (1..16,383) defines three different equivalence classes: –Equivalence Class 1: Fewer than 1 record –Equivalence Class 2: Between 1 and 16,383 records –Equivalence Class 3: More than 16,383 records

27. Nov. 17, 2005 27 Database Example (contd) F Test case 1: 0 recordsMember of equivalence class 1 and adjacent to boundary value F Test case 2: 1 recordBoundary value F Test case 3:2 recordsAdjacent to boundary value F Test case 4:723 recordsMember of equivalence class 2

28. Nov. 17, 2005 28 Database Example (contd) F Test case 5:16,382 recordsAdjacent to boundary value F Test case 6:16,383 recordsBoundary value F Test case 7:16,384 recordsMember of equivalence class 3 and adjacent to boundary value

29. Nov. 17, 2005 29 Equivalence Testing of Output Specifications F We also need to perform equivalence testing of the output specifications F Example: In 2003, the minimum Social Security (OASDI) deduction from any one paycheck was $0.00, and the maximum was $5,394.00 –Test cases must include input data that should result in deductions of exactly $0.00 and exactly $5,394.00 –Also, test data that might result in deductions of less than $0.00 or more than $5,394.00

30. Nov. 17, 2005 30 Overall Strategy F Equivalence classes together with boundary value analysis to test both input specifications and output specifications –This approach generates a small set of test data with the potential of uncovering a large number of faults

31. Nov. 17, 2005 31 White-Box Testing F Statement Coverage Criterion F Branch coverage criterion F Data flow coverage criterion F Path coverage criterion

32. Nov. 17, 2005 32 Statement Coverage while (…) { …. while (…) { ….. break; } ….. break;

33. Nov. 17, 2005 33 Branch Coverage if ( StdRec != null) StdRec.name = arg[1]; …… ……. write (StdRec.name)

34. Nov. 17, 2005 34 Path Coverage read (x, y); if x> 0 z = 1; else z = 0; if y < 0 write (z) else write (y / z); x = 1, y = 1 -> 1 x = -1, y = -1 -> 0 x = 1, y = -1 -> 1 x = -1, y = 1 -> failure X>0 Y < 0

35. Nov. 17, 2005 35 Data Flow F Data-flow graph: is defined on the control flow graph by defining sets of variables DEF(n), C-USE(n) and P- USE(n), for each node n. F Variable x is in DEF(n) if –n is the start node and x is a global, parameter, or static local variable. –x is declared in a basic block n with an initializer. –x is assigned to the basic block with the =, op=, ++, or --operator.

36. Nov. 17, 2005 36 Data flow Testing read (x, y); if x> 0 z = 1; else z = 0; if y < 0 write (z) else write (y / z); n2n2 n3n3 n4n4 n5n5 n6n6 n7n7 X>0 Y < 0 n1n1 DEF(n 1 ) = {x, y} P-USE(n 2 ) = {x} DEF(n 3 ) = {z} DEF(n 4 ) = {z} USE(n 5 ) = {y} USE(n 6 )= {z} C-USE(n 7 } = {y, z} x: (n 1, n 2 ) y: (n 1, n 5 ), (n 1, n 7 ) z: ( n 3, n 6 ), (n 3, n 7 ), (n 4, n 6 ), (n 4, n 7 )

37. Nov. 17, 2005 37 C-Use F Variable x is in C-USE(n) if x occurs as a C-USE expression in basic block as – a procedure argument, – an initializer in a declaration, – a return value in a return statement, – the second operand of “=“ – either operand of “op=“ – the operand of ++, --, * – the first operand of., =>

38. Nov. 17, 2005 38 P-Use F Variable x is in P-USE(n) if x occurs as a P- USE expression in basic block – as the conditional expression in a if, for, while, do, or switch statement. – as the first operand of the condition expression operator (?:), the logical and operator (&&), or the logical or operator (||)

39. Nov. 17, 2005 39 C-Use Coverage F A C-Use is a variable x and the set of all paths in the data-flow graph from node n a to n b such that –x is in DEF(n a ), and – x is not in DEF(n i ) for any other node n i on the paths (definition clear path), and – x is in C-USE(n b ). F A C-Use is covered by a set of tests if at least one of the paths in the C-Use is executed when the test is run.

40. Nov. 17, 2005 40 P-Use Coverage F A P-Use is a variable x and the set of all paths in the data-flow graph from node n a to n b such that –x is in DEF(n a ), and – x is not in DEF(n i ) for any other node ni on the paths (definition clear path), and – x is in P-USE(n b ). F A P-Use is covered by a set of tests if at least one of the paths in the P-Use is executed when the test is run.

41. Nov. 17, 2005 41 all-paths all-du Paths all-uses all-c-uses/some-p-uses all-c-uses all-p-uses/some-c-uses all-p-uses Branch Statement all-defs

42. Nov. 17, 2005 42 Complexity of White-Box Testing F Branch Coverage: –McCabe’s Cyclomatic complexity: CC(G) = #E - #V + 2P (G = (E, V)) F All-defs: M + I * V F All-p-uses, all-c-uses/some-p-uses, all-p-uses/some-c-uses, all-uses: N 2 F All-du, All-Path: 2 N: N is the number of conditional statements

43. Nov. 17, 2005 43 14.15 Comparison of Unit- Testing Techniques F Experiments comparing –Black-box testing –Glass-box testing –Reviews F [Myers, 1978] 59 highly experienced programmers –All three methods were equally effective in finding faults –Code inspections were less cost-effective F [Hwang, 1981] –All three methods were equally effective

44. Nov. 17, 2005 44 Comparison of Unit-Testing Techniques (contd) F [Basili and Selby, 1987] 42 advanced students in two groups, 32 professional programmers F Advanced students, group 1 –No significant difference between the three methods F Advanced students, group 2 –Code reading and black-box testing were equally good –Both outperformed glass-box testing F Professional programmers –Code reading detected more faults –Code reading had a faster fault detection rate

45. Nov. 17, 2005 45 Mutation Testing F A program under test is seeded with a single error to produce a ``mutant'' program. F A test covers a mutant if the output of the mutant and the program under test differ for that test input. F The mutation coverage measure for a test set is the ratio of mutants covered to total mutants

46. Nov. 17, 2005 46 Debugging Program Slicing Approach F Static slicing: decomposes a program by statically analyzing data-flow and control flow of the program. – A static program slice for a given variable at a given statement contains all the executable statements that could influence the value of that variable at the given statement. –The exact execution path for a given input is a subset of the static program slice with respect to the output variables at the given checkpoint. – Focus is an automatic debugging tool based on static program slicing to locate bugs.

47. Nov. 17, 2005 47 Dynamic Slicing F dynamic data slice: A dynamic data slice with respect to a given expression, location, and test case is a set of all assignments whose computations have propagated into the current value of the given expression at the given location. F dynamic control slice: A dynamic control slice with respect to a given location and test case is a set of all predicates that enclose the given location.

48. Nov. 17, 2005 48 Object-Oriented Testing Process F Unit testing - class F Integration testing - cluster F System testing - program

49. Nov. 17, 2005 49 Class Testing F Inheritance F Polymorphism F Sequence

50. Nov. 17, 2005 50 Class Testing Techniques F Testing inheritance F Testing polymorphism F State-oriented testing F Data Flow testing F Function dependence class testing

51. Nov. 17, 2005 51 Inheritance A subclass may re- define its inherited functions and other functions may be affected by the re- defined functions. When this subclass is tested, which functions need to be re-tested? Class foo { int local var;... int f1() { return 1; } int f2() { return 1/f1(); } } Class foo child :: Public foo { // child class of foo int f1() { return 0; } }

52. Nov. 17, 2005 52 Testing Inheritance ``Incremental testing of object-oriented class structures.'' Harrold et al., (1992) F New methods: complete testing F Recursive methods: limited testing F Redefined methods: reuse test scripts

53. Nov. 17, 2005 53 Polymorphism An object may be bound to different classes during the run time. Is it necessary to test all the possible bindings? // beginning of function foo {... P1 p; P2 c;... return(c.f1()/p1.f1()); // end of function foo

54. Nov. 17, 2005 54 Testing Polymorphism F `` Testing the Polymorphic Interactions between Classes.” McDaniel and McGregor (1994) Clemson University F “ Coupling-based Testing of OO Programs.” Roger Alexander and Jeff Offutt, Springer's Journal of Universal Computer Science: Special Issue on Breakthroughs and Challenges in Software Engineering (invited), 10(4):391- 427, April 2004. F “ Fragment Class Analysis for Testing of Polymorphism in Java Software.” Atanas Rountev, Ana Milanova, and Barbara G. Ryder, IEEE TRANSACTIONS ON SOFTWARE ENGINEERING, VOL. 30, NO. 6, JUNE 2004

55. Nov. 17, 2005 55 State-Oriented Testing F `` The state-based testing of object-oriented programs,'' 1992, C. D. Turner and D. J. Robson F ``On Object State Testing,'' 1993, Kung et al. F `` The testgraph methodology: Automated testing of collection classes,'' 1995, Hoffman and Strooper F The FREE approach: Binder http://www.rbsc.com/pages/Free.html

56. Nov. 17, 2005 56 Data Flow Testing F ``Performing Data Flow Testing on Class,'' 1994, Harrold and Rothermel. F ``Object-oriented data flow testing,'' 1995, Kung et al.

57. Nov. 17, 2005 57 Function Dependence Relationship F A function uses a variable means that the value of the variable is referenced in a computation expression or used to decide a predicate. F A function defines a variable means that the value of the variable is assigned when the function is invoked. F A variable x uses a variable y means the value of x is obtained from the value of y and others. x is affected when the value of y is changed.

58. Nov. 17, 2005 58 Function Dependence Relationship f 1 depends on f 2 F f 1 uses a variable x that is defined in f 2, F f 1 calls f 2 and uses the return value of f 2, F f 1 is called by f 2 and uses a parameter p that is defined in f 2. F f 1 uses a variable x and x uses a variable y which is defined in f 2.

59. Nov. 17, 2005 59 Object-Oriented Pitfalls F Type I faults –inheritance/polymorphism faults F Type II faults –object management faults F Type III faults –traditional faults

60. Nov. 17, 2005 60 Object-Oriented Pitfalls - Type I Class foo { int local_var. int f1() {return 1;} int f2() {return 1/f1( );} } class foo_derived{ int f1() {return 0;} } main() { int x, y; foo Obj; cin >> x, y; if ( x > 0 ) Obj = new foo( ); else Obj = new foo_derived( ); if ( y > 0 ) cout << Obj.f2( ); else cout << Obj.f1( ); }

61. Nov. 17, 2005 61 Object-Oriented Pitfalls - Type II class foo { Int *m_data; foo() { m_data = new Int; *m_data = 0; } ~foo() {delete m_data;} print() { cout << *m_data;} inc() {*m_data++;} main() { L1: foo *obj1 = new foo( ); L2: obj1->inc(); L3: foo *obj2 = new foo( ); L4: *obj2 = *obj1; if (P1) L5: obj2->inc( ); L6: obj2->print( ); if (P2) L7: obj2->~foo( ); if (P3) L8: obj1->print( ); L9: obj1->~foo( ); }

62. Nov. 17, 2005 62 Empirical Study - Applications F System A : GUI F System B : Data logging system F System C : Network communication program

63. Nov. 17, 2005 63 Fault Summary

64. Nov. 17, 2005 64 Functional Testing

65. Nov. 17, 2005 65 Statement Testing

66. Nov. 17, 2005 66 Branch Testing

67. Nov. 17, 2005 67 Code-Based Testing

68. Nov. 17, 2005 68 All-States

69. Nov. 17, 2005 69 All-Transitions

70. Nov. 17, 2005 70 State-Based Testing

71. Nov. 17, 2005 71 Object-Flow Based Testing F Object –an object is an instance of a class F Define –an object is defined if its state is initiated or changed F Use –an object is used if one of its data members is referenced

72. Nov. 17, 2005 72 Object-Flow Coverage Criteria F All-du-pairs –at lease one definition-clear path from every definition of every object to every use of that definition must be exercised under some test. F All-bindings –Every possible binding of every object must be exercised at least once when the object is defined or used.

73. Nov. 17, 2005 73 Weak Object-Flow Testing F An object is defined – the constructor of the object is invoked; –a data member is defined; –a method that initiates/modifies the data member(s) of the object is invoked

74. Nov. 17, 2005 74 Object-Oriented Pitfalls - Type I Class foo { int local_var. int f1() {return 1;} int f2() {return 1/f1();} } class foo_derived{ int f1() {return 0;} } main() { int x, y; foo Obj; cin >> x, y; if ( x > 0 ) Obj = new foo(); else Obj = new foo_derived(); if ( y > 0 ) cout << Obj.f2(); else cout << Obj.f1(); }

75. Nov. 17, 2005 75 Object-Oriented Pitfalls - Type II class foo { Int *m_data; foo() { m_data = new Int; *m_data = 0; } ~foo() {delete m_data;} print() { cout << *m_data;} inc() {*m_data++;} main() { L1: foo *obj1 = new foo(); L2: obj1->inc(); L3: foo *obj2 = new foo(); L4: *obj2 = *obj1; if (P1) L5: obj2->inc(); if (P2) L6: obj2->~foo(); if (P3) L7 obj1->print(); obj1->~foo(); }

76. Nov. 17, 2005 76 All-DU-Pairs

77. Nov. 17, 2005 77 All-Bindings

78. Nov. 17, 2005 78 Object-Flow Based Testing I

79. Nov. 17, 2005 79 Object-Flow Based Testing I’

80. Nov. 17, 2005 80 Object-Flow Based Testing II

81. Nov. 17, 2005 81 Object-Flow Based Testing II’

82. Nov. 17, 2005 82 Integrated Testing F Functional testing F Code-based testing F Object-flow based testing I F State-based testing

83. Nov. 17, 2005 83 Integrated Approach