1. These courseware materials are to be used in conjunction with Software Engineering: A Practitioner’s Approach, 6/e and are provided with permission by R.S. Pressman & Associates, Inc., copyright © 1996, 2001, 2005 1 Software Engineering: A Practitioner’s Approach, 6/e Chapter 14 Software Testing Techniques Software Engineering: A Practitioner’s Approach, 6/e Chapter 14 Software Testing Techniques copyright © 1996, 2001, 2005 R.S. Pressman & Associates, Inc. For University Use Only May be reproduced ONLY for student use at the university level when used in conjunction with Software Engineering: A Practitioner's Approach. Any other reproduction or use is expressly prohibited.

2. These courseware materials are to be used in conjunction with Software Engineering: A Practitioner’s Approach, 6/e and are provided with permission by R.S. Pressman & Associates, Inc., copyright © 1996, 2001, 2005 2 Testability: how easily a software can be tested Operability— the better it works, the more efficiently it can be tested. - It operates cleanly Operability— the better it works, the more efficiently it can be tested. - It operates cleanly Observability—What you see is what you test. The results of each test case are readily observed (clear and visible variables and states, incorrect output, …) Observability—What you see is what you test. The results of each test case are readily observed (clear and visible variables and states, incorrect output, …) Controllability—the degree to which testing can be automated and optimized Controllability—the degree to which testing can be automated and optimized Decomposability—testing can be targeted to independent modules Decomposability—testing can be targeted to independent modules Simplicity—reduce complex architecture and logic to simplify tests Simplicity—reduce complex architecture and logic to simplify tests Stability—few changes are requested during testing Stability—few changes are requested during testing Understandability—of the design Understandability—of the design

3. These courseware materials are to be used in conjunction with Software Engineering: A Practitioner’s Approach, 6/e and are provided with permission by R.S. Pressman & Associates, Inc., copyright © 1996, 2001, 2005 3 What is a “Good” Test? A good test has a high probability of finding an error A good test has a high probability of finding an error A good test is not redundant. A good test is not redundant. A good test should be “best of breed” (best between those with similar intent) A good test should be “best of breed” (best between those with similar intent) A good test should be neither too simple nor too complex A good test should be neither too simple nor too complex

4. These courseware materials are to be used in conjunction with Software Engineering: A Practitioner’s Approach, 6/e and are provided with permission by R.S. Pressman & Associates, Inc., copyright © 1996, 2001, 2005 4 Test Case Design "Bugs lurk in corners and congregate at boundaries..." Boris Beizer OBJECTIVE CRITERIA CONSTRAINT to uncover errors in a complete manner with a minimum of effort and time

5. These courseware materials are to be used in conjunction with Software Engineering: A Practitioner’s Approach, 6/e and are provided with permission by R.S. Pressman & Associates, Inc., copyright © 1996, 2001, 2005 5 Exhaustive Testing loop < 20 X There are 10 possible paths! If we execute one test per millisecond, it would take 3,170 years to test this program!! 14

6. These courseware materials are to be used in conjunction with Software Engineering: A Practitioner’s Approach, 6/e and are provided with permission by R.S. Pressman & Associates, Inc., copyright © 1996, 2001, 2005 6 Selective Testing loop < 20 X Selected path

7. These courseware materials are to be used in conjunction with Software Engineering: A Practitioner’s Approach, 6/e and are provided with permission by R.S. Pressman & Associates, Inc., copyright © 1996, 2001, 2005 7 Software Testing Methods Strategies white-box methods black-box methods

8. These courseware materials are to be used in conjunction with Software Engineering: A Practitioner’s Approach, 6/e and are provided with permission by R.S. Pressman & Associates, Inc., copyright © 1996, 2001, 2005 8 White-Box Testing... our goal is to ensure that all statements and conditions have been executed at least once...

9. These courseware materials are to be used in conjunction with Software Engineering: A Practitioner’s Approach, 6/e and are provided with permission by R.S. Pressman & Associates, Inc., copyright © 1996, 2001, 2005 9 Why Cover? logic errors and incorrect assumptions are inversely proportional to a path's execution probability we often believe that a path is not that a path is not likely to be executed; in fact, reality is often counter intuitive typographical errors are random; it's likely that untested paths will contain some

10. Basic Path Testing Define a basic set of execution paths Define a basic set of execution paths Guarantee that every statement is executed at least once Guarantee that every statement is executed at least once Using a logical complexity measure of a procedural design and use it as a guide for defining a basic set of execution path Using a logical complexity measure of a procedural design and use it as a guide for defining a basic set of execution path These courseware materials are to be used in conjunction with Software Engineering: A Practitioner’s Approach, 6/e and are provided with permission by R.S. Pressman & Associates, Inc., copyright © 1996, 2001, 2005 10

11. Flow Graph Notation A representation for control flow of program A representation for control flow of program Each structured construct -> a flow graph symbol Each structured construct -> a flow graph symbol These courseware materials are to be used in conjunction with Software Engineering: A Practitioner’s Approach, 6/e and are provided with permission by R.S. Pressman & Associates, Inc., copyright © 1996, 2001, 2005 11

12. Independent program path Any path through the program that introduces at least one new set of processing statements or a new condition. Any path through the program that introduces at least one new set of processing statements or a new condition. An independent path must move along at least one edge that has not been traversed before the path is defined. An independent path must move along at least one edge that has not been traversed before the path is defined. The basic set is not unique. The basic set is not unique. How do we know how many paths to look for? How do we know how many paths to look for? Cyclomatic complexity Cyclomatic complexity These courseware materials are to be used in conjunction with Software Engineering: A Practitioner’s Approach, 6/e and are provided with permission by R.S. Pressman & Associates, Inc., copyright © 1996, 2001, 2005 12

13. 13 Basis Path Testing: Cyclomatic Complexity First, we compute the cyclomatic complexity: number of simple decisions + 1 (simple decisions: predicate node) or number of enclosed areas + 1 or E-N+2 (E number of edges, N number of nodes) In this case, V(G) = 4

14. These courseware materials are to be used in conjunction with Software Engineering: A Practitioner’s Approach, 6/e and are provided with permission by R.S. Pressman & Associates, Inc., copyright © 1996, 2001, 2005 14 Cyclomatic Complexity A number of industry studies have indicated that the higher V(G), the higher the probability or errors. V(G) modules modules in this range are more error prone

15. These courseware materials are to be used in conjunction with Software Engineering: A Practitioner’s Approach, 6/e and are provided with permission by R.S. Pressman & Associates, Inc., copyright © 1996, 2001, 2005 15 Basis Path Testing Next, we derive the independent paths: Since V(G) = 4, there are four paths Path 1: 1,2,3,6,7,8 Path 2: 1,2,3,5,7,8 Path 3: 1,2,4,7,8 Path 4: 1,2,4,7,2,4,...7,8 Finally, we derive test cases to exercise these paths. 1 2 3 4 56 7 8

16. These courseware materials are to be used in conjunction with Software Engineering: A Practitioner’s Approach, 6/e and are provided with permission by R.S. Pressman & Associates, Inc., copyright © 1996, 2001, 2005 16 Basis Path Testing Notes you don't need a flow chart, but the picture will help when you trace program paths count each simple logical test, compound tests count as 2 or more basis path testing should be applied to critical modules

17. Deriving test cases Using the design or code, draw a flow diagram Using the design or code, draw a flow diagram Determine the cyclomatic complexity of the flow graph Determine the cyclomatic complexity of the flow graph Determine a basis set of independent paths Determine a basis set of independent paths Prepare test cases to force the execution of each path Prepare test cases to force the execution of each path These courseware materials are to be used in conjunction with Software Engineering: A Practitioner’s Approach, 6/e and are provided with permission by R.S. Pressman & Associates, Inc., copyright © 1996, 2001, 2005 17

18. These courseware materials are to be used in conjunction with Software Engineering: A Practitioner’s Approach, 6/e and are provided with permission by R.S. Pressman & Associates, Inc., copyright © 1996, 2001, 2005 18 Graph Matrices A tabular representation of a flow graph A tabular representation of a flow graph A graph matrix is a square matrix whose size (i.e., number of rows and columns) is equal to the number of nodes on a flow graph A graph matrix is a square matrix whose size (i.e., number of rows and columns) is equal to the number of nodes on a flow graph Each row and column corresponds to an identified node, and Each row and column corresponds to an identified node, and matrix entries correspond to connections (an edge) between nodes. matrix entries correspond to connections (an edge) between nodes. By adding a link weight to each matrix entry, the graph matrix can become a powerful tool for evaluating program control structure during testing By adding a link weight to each matrix entry, the graph matrix can become a powerful tool for evaluating program control structure during testing

19. Link weights can be Link weights can be The probability of execution of that edge The probability of execution of that edge The processing time for execution that edge The processing time for execution that edge The memory required The memory required The resource required The resource required Help to design test cases Help to design test cases These courseware materials are to be used in conjunction with Software Engineering: A Practitioner’s Approach, 6/e and are provided with permission by R.S. Pressman & Associates, Inc., copyright © 1996, 2001, 2005 19

20. These courseware materials are to be used in conjunction with Software Engineering: A Practitioner’s Approach, 6/e and are provided with permission by R.S. Pressman & Associates, Inc., copyright © 1996, 2001, 2005 20 Other Control Structure Testing Condition testing — a test case design method that exercises the logical conditions contained in a program module Condition testing — a test case design method that exercises the logical conditions contained in a program module Data flow testing — selects test paths of a program according to the locations of definitions (D) and uses (U) of variables in the program (every DU chain be covered at least once) Data flow testing — selects test paths of a program according to the locations of definitions (D) and uses (U) of variables in the program (every DU chain be covered at least once) Loop testing – focus on validity of loop constructs Loop testing – focus on validity of loop constructs

21. These courseware materials are to be used in conjunction with Software Engineering: A Practitioner’s Approach, 6/e and are provided with permission by R.S. Pressman & Associates, Inc., copyright © 1996, 2001, 2005 21 Loop Testing four different classes of loop NestedLoops Concatenated Loops Loops Unstructured Loops Simpleloop

22. These courseware materials are to be used in conjunction with Software Engineering: A Practitioner’s Approach, 6/e and are provided with permission by R.S. Pressman & Associates, Inc., copyright © 1996, 2001, 2005 22 Loop Testing: Simple Loops Minimum conditions—Simple Loops 1. skip the loop entirely 2. only one pass through the loop 3. two passes through the loop 4. m passes through the loop m < n 5. (n-1), n, and (n+1) passes through the loop where n is the maximum number of allowable passes

23. 23 Loop Testing: Nested Loops Start at the innermost loop. Set all outer loops to their minimum iteration parameter values. Test the min+1, typical, max-1 and max for the innermost loop, while holding the outer loops at their minimum values. Move out one loop and set it up as in step 2, holding all other loops at typical values. Continue this step until the outermost loop has been tested. Nested Loops

24. 24 Loop Testing: Concatenated Loops If the loops are independent of one another then treat each as a simple loop then treat each as a simple loop else* treat as nested loops else* treat as nested loops endif* for example, the final loop counter value of loop 1 is used to initialize loop 2. Concatenated Loops

25. These courseware materials are to be used in conjunction with Software Engineering: A Practitioner’s Approach, 6/e and are provided with permission by R.S. Pressman & Associates, Inc., copyright © 1996, 2001, 2005 25 Black-Box Testing requirements events input output

26. Incorrect or missing functions Incorrect or missing functions Interface errors Interface errors Errors in Data structures or external data base access Errors in Data structures or external data base access Behavior or performance errors Behavior or performance errors initialization and termination errors initialization and termination errors These courseware materials are to be used in conjunction with Software Engineering: A Practitioner’s Approach, 6/e and are provided with permission by R.S. Pressman & Associates, Inc., copyright © 1996, 2001, 2005 26

27. These courseware materials are to be used in conjunction with Software Engineering: A Practitioner’s Approach, 6/e and are provided with permission by R.S. Pressman & Associates, Inc., copyright © 1996, 2001, 2005 27 Black-Box Testing How is functional validity tested? How is functional validity tested? How is system behavior and performance tested? How is system behavior and performance tested? What classes of input will make good test cases? What classes of input will make good test cases? Is the system particularly sensitive to certain input values? Is the system particularly sensitive to certain input values? How are the boundaries of a data class isolated? How are the boundaries of a data class isolated? What data rates and data volume can the system tolerate? What data rates and data volume can the system tolerate? What effect will specific combinations of data have on system operation? What effect will specific combinations of data have on system operation?

28. These courseware materials are to be used in conjunction with Software Engineering: A Practitioner’s Approach, 6/e and are provided with permission by R.S. Pressman & Associates, Inc., copyright © 1996, 2001, 2005 28 Graph-Based Methods To understand the objects that are modeled in software and the relationships that connect these objects In this context, we consider the term “objects” in the broadest possible context. It encompasses data objects, traditional components (modules), and object-oriented elements of computer software.

29. Graph based testing Transaction flow modeling Transaction flow modeling Finite state modeling Finite state modeling Data flow modeling Data flow modeling Timing modeling (sequence of execution and its time) Timing modeling (sequence of execution and its time) These courseware materials are to be used in conjunction with Software Engineering: A Practitioner’s Approach, 6/e and are provided with permission by R.S. Pressman & Associates, Inc., copyright © 1996, 2001, 2005 29

30. Equivalence Partitioning Divides the input domain of a program into classes of data from which test cases can be derived. Divides the input domain of a program into classes of data from which test cases can be derived. Define test cases that uncovers classes of errors Define test cases that uncovers classes of errors These courseware materials are to be used in conjunction with Software Engineering: A Practitioner’s Approach, 6/e and are provided with permission by R.S. Pressman & Associates, Inc., copyright © 1996, 2001, 2005 30

31. If an Input condition is If an Input condition is A range A range One valid and two invalid equivalence classes One valid and two invalid equivalence classes A specific value A specific value One valid and two invalid equivalence classes One valid and two invalid equivalence classes A member of a set A member of a set One valid and one invalid equivalence classes One valid and one invalid equivalence classes Boolean Boolean One valid and one invalid equivalence classes One valid and one invalid equivalence classes These courseware materials are to be used in conjunction with Software Engineering: A Practitioner’s Approach, 6/e and are provided with permission by R.S. Pressman & Associates, Inc., copyright © 1996, 2001, 2005 31

32. Boundary value checking A greater number of errors occurs at the boundaries of the input domain A greater number of errors occurs at the boundaries of the input domain These courseware materials are to be used in conjunction with Software Engineering: A Practitioner’s Approach, 6/e and are provided with permission by R.S. Pressman & Associates, Inc., copyright © 1996, 2001, 2005 32

33. These courseware materials are to be used in conjunction with Software Engineering: A Practitioner’s Approach, 6/e and are provided with permission by R.S. Pressman & Associates, Inc., copyright © 1996, 2001, 2005 33 Comparison Testing (N-versioning) Used only in situations in which the reliability of software is absolutely critical (e.g., human-rated systems) Used only in situations in which the reliability of software is absolutely critical (e.g., human-rated systems) Separate software engineering teams develop independent versions of an application using the same specification Separate software engineering teams develop independent versions of an application using the same specification Each version can be tested with the same test data to ensure that all provide identical output Each version can be tested with the same test data to ensure that all provide identical output Then all versions are executed in parallel with real-time comparison of results to ensure consistency Then all versions are executed in parallel with real-time comparison of results to ensure consistency

34. 34 Orthogonal Array Testing Used when the number of input parameters is small and the values that each of the parameters may take are clearly bounded Dispersed uniformly throughout the test domain (balancing property)

35. These courseware materials are to be used in conjunction with Software Engineering: A Practitioner’s Approach, 6/e and are provided with permission by R.S. Pressman & Associates, Inc., copyright © 1996, 2001, 2005 35 OOT—Test Case Design Berard [BER93] proposes the following approach: 1.Each test case should be uniquely identified and should be explicitly associated with the class to be tested, 2.The purpose of the test should be stated, 3.A list of testing steps should be developed for each test and should contain [BER94]: a.a list of specified states for the object that is to be tested b.a list of messages and operations that will be exercised as a consequence of the test c.a list of exceptions that may occur as the object is tested d.a list of external conditions (i.e., changes in the environment external to the software that must exist in order to properly conduct the test) e.supplementary information that will aid in understanding or implementing the test.

36. 36 Testing Methods Fault-based testing (same for conventional) Fault-based testing (same for conventional) The tester looks for plausible faults The tester looks for plausible faults aspects of the implementation of the system that may result in defects. aspects of the implementation of the system that may result in defects. To determine whether these faults exist, test cases are designed to exercise the design or code. To determine whether these faults exist, test cases are designed to exercise the design or code. Class Testing and the Class Hierarchy Class Testing and the Class Hierarchy Inheritance does not obviate the need for thorough testing of all derived classes. Inheritance does not obviate the need for thorough testing of all derived classes. In fact, it can actually complicate the testing process. In fact, it can actually complicate the testing process. Scenario-Based Test Design Scenario-Based Test Design Scenario-based testing concentrates on what the user does, not what the product does. Scenario-based testing concentrates on what the user does, not what the product does. This means capturing the tasks (via use-cases) that the user has to perform, then applying them and their variants as tests. This means capturing the tasks (via use-cases) that the user has to perform, then applying them and their variants as tests.

37. 37 OOT Methods: Random Testing class based Random testing: check class instance life histories Random testing: check class instance life histories identify operations applicable to a class identify operations applicable to a class define constraints on their use define constraints on their use identify a minimum test sequence identify a minimum test sequence an operation sequence that defines the minimum life history of the class (object) an operation sequence that defines the minimum life history of the class (object) generate a variety of random (but valid) test sequences generate a variety of random (but valid) test sequences exercise other (more complex) class instance life histories exercise other (more complex) class instance life histories Like for an instance of Account class: open, setup, deposit, withdraw, close open, setup, deposit, withdraw, closeAnd open, setup, deposit,[deposit, withdraw, balance,creditlimit], withdraw, close open, setup, deposit,[deposit, withdraw, balance,creditlimit], withdraw, close

38. 38 OOT Methods: Partition Testing class based Partition Testing Partition Testing reduces the number of test cases required to test a class in much the same way as equivalence partitioning for conventional software reduces the number of test cases required to test a class in much the same way as equivalence partitioning for conventional software state-based partitioning state-based partitioning categorize and test operations based on their ability to change the state of a class categorize and test operations based on their ability to change the state of a class attribute-based partitioning attribute-based partitioning categorize and test operations based on the attributes that they use categorize and test operations based on the attributes that they use For example: partition the operations according to their use/modify/don’t use or modify an attribute For example: partition the operations according to their use/modify/don’t use or modify an attribute category-based partitioning category-based partitioning categorize and test operations based on the generic function each performs categorize and test operations based on the generic function each performs For example: operations can be categorized as initialization/computation/queries/termination For example: operations can be categorized as initialization/computation/queries/termination

39. 39 OOT Methods: Inter-Class Testing (During integration) Random Random Like in class testing Like in class testing Partitioning Partitioning In (intra-)class testing: state-based, attribute-based, category-based In (intra-)class testing: state-based, attribute-based, category-basedHere: Partition according to messages which serve different classes Partition according to messages which serve different classes State-based State-based Behavioral Behavioral State-based State-based

40. 40 OOT Methods: Inter-Class Testing (During integration) Random Random For each client class, use the list of class operators to generate a series of random test sequences. The operators will send messages to other server classes. For each client class, use the list of class operators to generate a series of random test sequences. The operators will send messages to other server classes. For each message that is generated, determine the collaborator class and the corresponding operator in the server object. For each message that is generated, determine the collaborator class and the corresponding operator in the server object. For each operator in the server object (that has been invoked by messages sent from the client object), determine the messages that it transmits. For each operator in the server object (that has been invoked by messages sent from the client object), determine the messages that it transmits. For each of the messages, determine the next level of operators that are invoked and incorporate these into the test sequence For each of the messages, determine the next level of operators that are invoked and incorporate these into the test sequence

41. These courseware materials are to be used in conjunction with Software Engineering: A Practitioner’s Approach, 6/e and are provided with permission by R.S. Pressman & Associates, Inc., copyright © 1996, 2001, 2005 41 OOT Methods: Behavior Testing The tests to be designed should achieve all state coverage [KIR94]. That is, the operation sequences should cause the Account class to make transition through all allowable states