1. 1 CHAPTER 17 SOFTWARE TESTING STRATEGIES

2. December 28, 1997 R. A. Volz -- Assistance - Nirmal Patil 2 TOPICS u A strategic approach to software testing u Unit Testing u Integration Testing u Validation Testing u System Testing u The ART of Debugging u Summary

3. December 28, 1997 R. A. Volz -- Assistance - Nirmal Patil 3 STRATEGIC APPROACH TO SOFTWARE TESTING Generic characteristics of software testing strategies:- u Testing begins at module level and works outward towards the of integration entire computer based system. u Different testing techniques are required at different points in time. u Testing is conducted by the s/w developer and ITG ( Independent Test Group ) for large projects. u Testing and Debugging are different and Debugging is essential in any testing strategy.

4. December 28, 1997 R. A. Volz -- Assistance - Nirmal Patil 4 Verification and Validation u Verification -- Does the product meet its specifications? u Validation -- Does the product perform as desired?

5. December 28, 1997 R. A. Volz -- Assistance - Nirmal Patil 5 Software Testing Strategy A Software Testing Strategy

6. December 28, 1997 R. A. Volz -- Assistance - Nirmal Patil 6 Software Testing Strategy

7. December 28, 1997 R. A. Volz -- Assistance - Nirmal Patil 7 Software Error Model u f(t) = cumulative remaining errors at time t  l 0 = initial failure rate u p = exponential reduction as errors repaired f(t) = (1/p)ln( l 0 pt + 1)

8. December 28, 1997 R. A. Volz -- Assistance - Nirmal Patil 8 STRATEGIC APPROACH u Issues to be addressed to develop a successful software testing strategy: u Specify product requirements in a quantifiable manner long before testing commences. u State testing objectives explicitly. u Understand the users of the software. u Develop testing plan that emphasizes “rapid cycle testing.”

9. December 28, 1997 R. A. Volz -- Assistance - Nirmal Patil 9 STRATEGIC APPROACH u Issues to be addressed to develop a successful software testing strategy: u Build robust software that is designed to test itself. u Use effective formal technical reviews as a filter to testing. u Conduct formal technical reviews to assess test strategy and test cases. u Develop continuous improvement approach

10. December 28, 1997 R. A. Volz -- Assistance - Nirmal Patil 10 UNIT TESTING u Unit testing -- focuses on the smallest element of software design viz. the module. u Corresponds to class testing in the OO context. u Makes heavy use of white-box testing.

11. December 28, 1997 R. A. Volz -- Assistance - Nirmal Patil 11 UNIT TESTING Unit Test Generation Considerations: Review Design information - develop unit test cases. Test cases interface local data structures boundary conditions independent paths error handling paths driver Module to be tested stub

12. December 28, 1997 R. A. Volz -- Assistance - Nirmal Patil 12 Unit Test Generation u Interface considerations u # of input parameters = # arguments? u Parameter and argument attributes match? u Parameter and argument units match? u Order correct (if important)? u Number and order of arguments for built-ins? u References to parms not associated with entry point? u Attempt to modify input-only arguments? u Global variable definitions consistent? u Constraints passed as arguments?

13. December 28, 1997 R. A. Volz -- Assistance - Nirmal Patil 13 Unit Test Generation u External I/O considerations u Files attributes correct? u OPEN/CLOSE correct? Format specification matches I/O statement? u Buffer size matches record size? u Files opened before use? u EOF handled correctly? I/O errors handled? u Textual errors in output?

14. December 28, 1997 R. A. Volz -- Assistance - Nirmal Patil 14 Unit Test Generation u Data structure considerations u Improper or inconsistent typing? u Erroneous initialization or default values? u Incorrect variable names? u Inconsistent data types? u Underflow, overflow and addressing exceptions?

15. December 28, 1997 R. A. Volz -- Assistance - Nirmal Patil 15 Unit Test Generation u Test cases must cover all execution paths u Common computational errors to be checked: u incorrect arithmetic u mixed mode operations u incorrect initialization u precision inaccuracy u incorrect symbolic representation of expression u Other tests needed u incompatible data types in comparisons u incorrect logical operators or precedence u comparison problems (e.g., == on floats) u loop problems

16. December 28, 1997 R. A. Volz -- Assistance - Nirmal Patil 16 Unit Test Generation u Error handling tests u Exception-handling is incorrect? u Error description is unintelligible, insufficient or incorrect? u Error condition causes system interrupt before error handling completed?

17. December 28, 1997 R. A. Volz -- Assistance - Nirmal Patil 17 INTEGRATION TESTING u A systematic approach for constructing program structure while conducting tests to uncover errors associated with interfacing. u Tendency for Non-Incremental integration.. Big Bang approach …. Chaos !! ( usually ). u Incremental integration - program is constructed and tested in small segments. u Top-Down Integration testing u Bottom-Up Integration testing

18. December 28, 1997 R. A. Volz -- Assistance - Nirmal Patil 18 INTEGRATION TESTING

19. December 28, 1997 R. A. Volz -- Assistance - Nirmal Patil 19 INTEGRATION TESTING Top-Down Approach u Begin construction and testing with main module. u Stubs are substituted for all subordinate modules. u Subordinate stubs are replaced one at a time by actual modules. u Tests are conducted as each module is integrated. u On completion of each set of tests, another stub is replaced with the real module. u Regression testing may be conducted to ensure that new errors have not been introduced.

20. December 28, 1997 R. A. Volz -- Assistance - Nirmal Patil 20 Top Down Approach - Use Stubs

21. December 28, 1997 R. A. Volz -- Assistance - Nirmal Patil 21 INTEGRATION TESTING Top-Down Approach : u Advantages: u Verifies major control or decision points early in the test process. u With the use of depth-first integration testing, a complete function of the software can be demonstrated. -- Confidence builder for developer/customer. u Disadvantages: u Since stubs replace lower level modules, no significant data can flow upwards to the main module.

22. December 28, 1997 R. A. Volz -- Assistance - Nirmal Patil 22 INTEGRATION TESTING Bottom Up Approach : u This approach begins construction and testing with modules at the lowest levels in the program structure. u Low-level modules are combined into clusters. u A driver is written to coordinate test case input and output. u The cluster is tested. u Drivers are removed and clusters are combined moving upward in the program hierarchy.

23. December 28, 1997 R. A. Volz -- Assistance - Nirmal Patil 23 Bottom Up Approach

24. December 28, 1997 R. A. Volz -- Assistance - Nirmal Patil 24 INTEGRATION TESTING Bottom Up Approach u Advantages: u Easier test case design and lack of stubs. u Disadvantages: u The program as an entity is does not exist until the last module is added. Sandwich Testing:- combined approach u Top down strategy for upper levels and Bottom up strategy for subordinate levels.

25. December 28, 1997 R. A. Volz -- Assistance - Nirmal Patil 25 INTEGRATION TESTING u Regression Testing u Re-execution of some subset of tests already conducted to ensure that the new changes do not have unintended side effects. u The Regression test suite should contain three different classes of test cases : u A representative sample of tests that will exercise all software functions u Additional tests that focus on functions that are likely to be affected by the change. u Tests that focus on software components that have changed.

26. December 28, 1997 R. A. Volz -- Assistance - Nirmal Patil 26 INTEGRATION TESTING Integration Test Documentation Order of Integration Scope of testing 1234 Test plan Test Procedure n Actual Test Results 5 Ref. & Appendix Test phases and builds Schedule Overhead software Environment / Resources Unit test Test environment Test case data Expected Results for build n

27. December 28, 1997 R. A. Volz -- Assistance - Nirmal Patil 27 VALIDATION TESTING u It provides final assurance that software meets all functional, behavioral, and performance requirements. -- Exclusive use of Black-box testing techniques. u After each validation test case either software conforms to specs or a deviation from specs is detected and a deficiency list needs to be worked. u Alpha and Beta testing u Alpha test -- At developer’s site by customer. u Beta test -- At customer’s site in a “live” environment.

28. December 28, 1997 R. A. Volz -- Assistance - Nirmal Patil 28 SYSTEM TESTING u A series of tests to verify that all system elements have been properly integrated. u Recovery Testing: u Forces software to fail in a variety of ways and verifies that recovery is properly performed. u Security Testing: u Attempts to verify the software’s protection mechanisms. The software designer tries to make penetration cost more than the value of information obtained by breaking in.

29. December 28, 1997 R. A. Volz -- Assistance - Nirmal Patil 29 SYSTEM TESTING u Stress Testing: u Executes the system in a manner that demands resources in abnormal quantity, frequency or volume. u Performance Testing: u To test the run time performance of a system within the context of an integrated system.

30. CLCS Test Approach Developers System Integration and Test Group Validation Group Application S/W IPT Users Operations Environment User Acceptance Tests Development Environment Unit Test Design Early User Eval Unit Integ Test Integration Environment System Test COTS H/W on Dock User Eval CSCI Int Test System Delivery Acceptance Test System S/W Validation Tests User App S/W Validation Tests

31. December 28, 1997 R. A. Volz -- Assistance - Nirmal Patil 31 THE ART OF DEBUGGING u Debugging is a consequence of successful testing -- when a test case uncovers an error, it is the debugging process that results in the removal of the error. u Debugging is an ART. The external manifestation of the error and the cause of the error normally do not share an obvious relationships.

32. December 28, 1997 R. A. Volz -- Assistance - Nirmal Patil 32 THE ART OF DEBUGGING The Debugging process Test cases Execution of test cases Results Debugging Suspected causes Additional tests Identified causes Corrections Regression tests

33. December 28, 1997 R. A. Volz -- Assistance - Nirmal Patil 33 THE ART OF DEBUGGING u Debugging Approaches u Brute force : - Take memory dumps, invoke run time traces. Least efficient and quite common. u Backtracking :- Once an error is uncovered, trace your way back to the cause of the error. u Cause Elimination : - Isolate potential causes, devise cause hypotheses tests to isolate bug. u Use of debugging tools

34. December 28, 1997 R. A. Volz -- Assistance - Nirmal Patil 34 COMMENTS u Should the software developer be involved with testing ? u Developer’s have a vested interest in demonstrating that their software is error-free. u Developer’s (psychologically) feel that testing is destructive. u When are we done with testing ? u “You are never done with testing, the burden shifts from you to the customer.”

35. December 28, 1997 R. A. Volz -- Assistance - Nirmal Patil 35 SUMMARY u Software Testing accounts for the largest percentage of technical effort in the software process. u Objective of Software testing -- To uncover errors, maintain software quality. u Steps : Unit, Integration, Validation, System. u Debugging is often an art and the most valuable resource is often the counsel of other software engineers.