1. SENG521 (Fall 2002)far@enel.ucalgary.ca1 SENG 521 Software Reliability & Testing Preparing for Test (Part 6a) Department of Electrical & Computer Engineering, University of Calgary B.H. Far （ far@enel.ucalgary.ca ） http://www.enel.ucalgary.ca/~far/Lectures/SENG521/06a/

2. SENG521 (Fall 2002)far@enel.ucalgary.ca2 Concepts /1 Run: Run: The smallest division of work that can be initiated by external intervention. Run is associated with input state (set of input variables) and runs with identical input state are of the same run type. Operation: Operation: An operation is a group of run types as conceived at the development stage. Test procedure: Test procedure: A controller that sets up environmental conditions and invokes randomly selected test cases at random times.

3. SENG521 (Fall 2002)far@enel.ucalgary.ca3 Concepts /2 Types of test Feature Test: Feature Test: A single execution of an operation with interaction between operations minimized. Load Test: Load Test: Testing with field use data and accounting for interactions Regression Test: Regression Test: Feature tests after every build involving significant change, i.e., check whether a bug fix worked.

4. SENG521 (Fall 2002)far@enel.ucalgary.ca4 Concepts /3 Direct input variable: Direct input variable: is a variables that controls the operation directly Example: arguments, selection menu, entered data field. Indirect input variable: Indirect input variable: is a variable that only influences the operations or its effects are propagated to the operation Example: traffic load, environmental variable

5. SENG521 (Fall 2002)far@enel.ucalgary.ca5 Example Example of direct and indirect input variables for an operation

6. SENG521 (Fall 2002)far@enel.ucalgary.ca6 What is a Test Case? A test case is a partial specification of a run through the naming of its direct input variables and their values. Test case is independent of operational mode The same test case can execute in different operational modes. Thus a test case can generate multiple runs each with different potential failure behavior

7. SENG521 (Fall 2002)far@enel.ucalgary.ca7 Properties A “good” test case satisfies: It has a reasonable probability to catch an error It is not redundant It is the best of its breed It is neither too simple nor too complex It makes program failure obvious

8. SENG521 (Fall 2002)far@enel.ucalgary.ca8 Example Test case is specified with its direct input variables In theory, it is possible to record all the input variables needed to initiate the runs that make up the whole execution space of the software

9. SENG521 (Fall 2002)far@enel.ucalgary.ca9 Test Case & Run Specification of the indirect input variable gives a test case the necessary context so it can become a run. During feature and regression test, the influence of indirect input variables should be kept to minimum to ensure that the operation is reliable. Indirect input variables are effective during load test. The load test is divided into a number of operation modes each derived by a test procedure.

10. SENG521 (Fall 2002)far@enel.ucalgary.ca10 Procedure The procedure involves two steps: 1)Preparing test cases (Test case management) Using either field data recording or operational profiles 2)Preparing test procedures Test procedure is a controller that sets up environmental conditions and invokes randomly selected test cases at random times. Prepare one test procedure for each operational mode using operational profile and the operation occurrence rate

11. SENG521 (Fall 2002)far@enel.ucalgary.ca11 Test Cases Management The procedure for preparing test cases involve: 1.Estimate the number of new test cases needed for the current release 2.Allocate the number of new test cases among the systems to be tested 3.Allocate the number of new test cases for each system among its new operations 4.Specify the new test cases 5.Adding the new test cases to the ones already available (may be from a previous release)

12. SENG521 (Fall 2002)far@enel.ucalgary.ca12 1. Estimate New Test Cases Affected by two factors: time and cost Method: Method: Compute the number of test cases for the given Time: (available time X available staff)/ (average time to prepare a test case) Cost: (available budget) / (average preparation cost per test case) Select the minimum number of the two

13. SENG521 (Fall 2002)far@enel.ucalgary.ca13 2. Test Case Allocation: System Allocate the bulk of the test cases to the product itself. Give weight to the differing operations that have high occurrence probabilities. Do not allocate test cases to acquired components unless the newly acquired components are substituted for previous ones. Allocate test cases to acquired components based on their size relative to the product and the estimated risk. In the case of Fone Follower (example in Musa’s Book), the operating system represented a substantial part of the product and its reliability was unknown. Hence 500 test cases were allocated to the product and 200 to the operating system.

14. SENG521 (Fall 2002)far@enel.ucalgary.ca14 3. Test Case Allocation: Operations 1.Convert graphical representation of the operational profile to the tabular representation by walking through all the paths, obtain an occurrence probability for each operation (path) by multiplying the branch probabilities together. 2.Identify the rarely occurring critical new operations and determine how many test cases to pre-assign to each. 3.Determine the allocation probabilities for the other new operations. 4.Pre-assign one test case to each infrequent other new operation. 5.Assign the remaining test cases to the remaining other new operations in accordance with the allocation probabilities.

15. SENG521 (Fall 2002)far@enel.ucalgary.ca15 Critical Operations A critical operation is one for which successful execution adds a great deal of extra value and failure causes a great deal of impact with respect to human life, cost, or system capability. An example of a critical operation is the SCRAM operation in nuclear power plants that shuts down a reactor when it starts to overheat. The operation that handles this condition is very rarely used, but it is extremely critical. Identify only the rarely occurring critical operations because sufficient test cases will be allocated to the frequently occurring critical operations because of their substantial occurrence probabilities.

16. SENG521 (Fall 2002)far@enel.ucalgary.ca16 New Operations In the case of a new release, set the allocation probabilities for the new operations equal to the occurrence probabilities of the system operational profile. This is a satisfactory approximation; the sum of the occurrence probabilities for the other new operations will be very close to 1. For subsequent releases, divide the system occurrence probabilities for the other new operations by the total of the system occurrence probabilities for the other new operations.

17. SENG521 (Fall 2002)far@enel.ucalgary.ca17 Infrequent Operations Infrequent operations are those that would not normally be assigned test cases by virtue of their occurrence probabilities. Ensure that we assign each operation at least one test case. The occurrence probability of an operation is often such that, because of the fact that runs usually substantially outnumber test cases, the operation might well invoke a run even though the occurrence probability might not normally result in test case selection. Thus determine the threshold occurrence probability below which you would not normally assign a test case to an operation. This is 0.5 divided by the number of new test cases. We then pre-assign one test case to each other new operation whose occurrence probability falls below the threshold.