CMSC 345, Version 11/07 SD Vick from S. Mitchell Software Testing

2 CMSC 345, Version 11/07 SD Vick from S. Mitchell Topics The testing process  unit testing  integration and system testing  acceptance testing Test case planning and tracking Testing types  black box vs. white box testing  coverage testing  regression testing  stress testing Test case prioritization

3 CMSC 345, Version 11/07 SD Vick from S. Mitchell Testing Goal is to discover defects The only validation technique for non- functional requirements Successful test: causes a program to behave in an anomalous way Testing shows the presence, not the absence, of defects.

4 CMSC 345, Version 11/07 SD Vick from S. Mitchell Unit Testing Integration and System Testing Acceptance Testing Testing Process Overview unit 1 unit 2 unit 3 unit n unit Subsystem 1 unit Subsystem 2 unit Subsystem m Final System

5 CMSC 345, Version 11/07 SD Vick from S. Mitchell unit Subsystem The Testing Process Elaborated Unit (component) testing  Testing of individual program components (e.g., functions, methods, or classes)  Usually the responsibility of the component’s developer  Must be structured  White box testing Integration and system testing  Testing of groups of components integrated to create a subsystem or the entire system  Sometimes the responsibility of an independent testing team  Tests are based on system specification (black box testing)  Is iterative (keeps building)  May include alpha and beta testing Acceptance testing  Run in the presence of customer or by customer  Used to validate all system requirements unit Final System $$$$!

6 CMSC 345, Version 11/07 SD Vick from S. Mitchell Unit (Component) Testing You may do this on your own, but always … have a plan keep records automate!  JUnit (for Java) or other tools  even #ifdef - #endif pairs (C/C++)

7 CMSC 345, Version 11/07 SD Vick from S. Mitchell Integration Testing Approaches – Procedural Programming Top-down testing  Start with high-level components and integrate from top down  Uses stubs Bottom-up testing  Start with low-level components and integrate from bottom up  Uses drivers Sandwich testing  Combination of top-down and bottom-up Thread testing  Tests one piece of functionality at a time

8 CMSC 345, Version 11/07 SD Vick from S. Mitchell n Level 0 1 Top-down Integration & Testing Being Tested Stub Stub: 1) says “I’ve been called!”, 2) confirms receipt of inputs, if any, and 3) returns hard-coded outputs, if necessary

9 CMSC 345, Version 11/07 SD Vick from S. Mitchell n Level 0 1 Top-down Integration & Testing Tested Being Tested Stub Stub: 1) says “I’ve been called!”, 2) confirms receipt of inputs, if any, and 3) returns hard-coded outputs, if necessary

10 CMSC 345, Version 11/07 SD Vick from S. Mitchell n Level 0 1 Top-down Integration & Testing Tested Being Tested Stub: 1) says “I’ve been called!”, 2) confirms receipt of inputs, if any, and 3) returns hard-coded outputs, if necessary

11 CMSC 345, Version 11/07 SD Vick from S. Mitchell n Level 0 1 Bottom-up Integration & Testing Being Tested Driver Driver: 1) Calls functions, 2) passes in inputs, if any, and 3) confirms receipt of outputs, if any

12 CMSC 345, Version 11/07 SD Vick from S. Mitchell n Level 0 1 Bottom-up Integration & Testing Being Tested Tested Driver: 1) Calls functions, 2) passes in inputs, if any, and 3) confirms receipt of outputs, if any

13 CMSC 345, Version 11/07 SD Vick from S. Mitchell n Level 0 1 Bottom-up Integration & Testing Tested Driver Driver: 1) Calls functions, 2) passes in inputs, if any, and 3) confirms receipt of outputs, if any

14 CMSC 345, Version 11/07 SD Vick from S. Mitchell n Level 0 1 Sandwich Integration & Testing Uses both stubs and drivers

15 CMSC 345, Version 11/07 SD Vick from S. Mitchell n Level 0 1 Thread Integration & Testing May be done top-down, bottom-up, or sandwich

16 CMSC 345, Version 11/07 SD Vick from S. Mitchell Exercise (on your own!) What are some of the benefits associated with each of the following integration and testing methods? Top-down Bottom-up Sandwich Thread

17 CMSC 345, Version 11/07 SD Vick from S. Mitchell Integration Testing Approaches – Object-oriented Programming Use based  Start with classes that have no dependencies  Move “out” to next “level” of classes, and so on  Similar to bottom-up for procedural programming Cluster  Test clusters of associated classes  Integrate the clusters  May be combined with use based testing (above) Scenario  Based on functional specifications (possibly expressed as use cases)  Similar to thread testing for procedural programming

18 CMSC 345, Version 11/07 SD Vick from S. Mitchell System Testing Alpha  Internal to development organization, but not the developers themselves Beta  External to development organization Acceptance  Performed in the presence of or possibly by the customer  Determines final “acceptance” by the customer

19 CMSC 345, Version 11/07 SD Vick from S. Mitchell Test Case Planning and Tracking Natural language description ID Specific values (if possible) Pass, Fail, other notes How to determine Pass or Fail

20 CMSC 345, Version 11/07 SD Vick from S. Mitchell A Sample Testing Form Before Testing

21 CMSC 345, Version 11/07 SD Vick from S. Mitchell A Sample Testing Form After Testing

22 CMSC 345, Version 11/07 SD Vick from S. Mitchell A Sample Testing Form Before Testing

23 CMSC 345, Version 11/07 SD Vick from S. Mitchell A Sample Testing Form After Testing

24 CMSC 345, Version 11/07 SD Vick from S. Mitchell White Box Testing Sometimes called structural testing Can “see inside” the code unit Objectives:  Exercise unique algorithms  Exercise all program statements at least once Typically tests small program units such as functions or class methods code code code code code input output

25 CMSC 345, Version 11/07 SD Vick from S. Mitchell Coverage Testing A technique for white box testing Ensures that each statement is executed at least once Use a program flow graph to derive paths Find minimum paths needed to “cover” all statements (there are tools to do this) Drive the code through all of these paths All branches are executed, but not all combinations of branches. Some paths may be impossible to test.

26 CMSC 345, Version 11/07 SD Vick from S. Mitchell Describes the program control flow. Each branch is shown as a separate path and loops are shown by arrows looping back to the loop condition node Used as a basis for computing the cyclomatic complexity Cyclomatic complexity = Number of edges - Number of nodes +2 Program flow graphs

27 CMSC 345, Version 11/07 SD Vick from S. Mitchell The number of tests to test all control statements equals the cyclomatic complexity Cyclomatic complexity equals number of conditions in a program Useful if used with care. Does not imply adequacy of testing. Although all paths are executed, all combinations of paths are not executed Cyclomatic complexity

28 CMSC 345, Version 11/07 SD Vick from S. Mitchell Binary search flow graph

29 CMSC 345, Version 11/07 SD Vick from S. Mitchell 1, 2, 3, 8, 9 1, 2, 3, 4, 6, 7, 2 1, 2, 3, 4, 5, 7, 2 1, 2, 3, 4, 6, 7, 2, 8, 9 Test cases should be derived so that all of these paths are executed A dynamic program analyser may be used to check that paths have been executed Independent paths

30 CMSC 345, Version 11/07 SD Vick from S. Mitchell Black Box Testing Cannot “see inside” code unit Test cases are:  based on the system specification and/or interfaces (preconditions, postconditions, invariants, and parameters) Use equivalence partitions when conducting black box testing input output

31 CMSC 345, Version 11/07 SD Vick from S. Mitchell Equivalence Partitions and Boundary Testing Equivalence Partition A group of input values that will produce equivalent behavior Example  Square root function the set of all numbers >= zero the set of all negative numbers Test cases should be chosen from each partition, especially at the boundaries.

32 CMSC 345, Version 11/07 SD Vick from S. Mitchell Equivalence Partitions and Boundary Testing Exercise A system accepts 4 to 10 inputs that are 5- digit integers greater than 10,000 Partition the system inputs into groups (partitions) that should cause equivalent behavior. Include both valid and invalid inputs. Ian Somerville, Software Engineering, 6 th editionc

33 CMSC 345, Version 11/07 SD Vick from S. Mitchell Ian Somerville, Software Engineering, 6 th edition The six partitions with corresponding boundary values

34 CMSC 345, Version 11/07 SD Vick from S. Mitchell Regression Testing Code A Test 1 Test 2 Code A Code B Code C Test 1 Test 2 Test 3 Test 4 Test 5 Code A Code B Code C Code D Test 2 Test 3 Test 4 Test 5 Test 1 Test 6 Test 7 Test 8 … and so on

35 CMSC 345, Version 11/07 SD Vick from S. Mitchell Stress Testing Example: A child’s car seat Exercises the system beyond its maximum design load. Examples:  Exceed string lengths  Store/manipulate more data than in specification  Load system with more users than in specification Often causes defects to come to light Systems should not fail catastrophically.  Stress testing checks for unacceptable loss of service or data.

36 CMSC 345, Version 11/07 SD Vick from S. Mitchell Test Case Prioritization Why? To locate defects quickly to:  begin debugging ASAP  debug and deliver in increments ASAP To locate the majority of errors quickly Other How? Can prioritize according to:  module or functional dependencies  module size  module complexity  module suspected to be the most error-prone  most “important” module or functionality  other

37 CMSC 345, Version 11/07 SD Vick from S. Mitchell References Somerville, I., Software Engineering, 6 th ed SourceForge.net, JUnit, accessed 11/6/07 JUnit.org, accessed 11/6/07http://