1. Validating and Improving Test- Case Effectiveness Yuri Chernak Presented by Michelle Straughan

2. Overview Introduction Introduction Test-Case Effectiveness Metric (TCE) Test-Case Effectiveness Metric (TCE) Improving Test-Case Effectiveness Improving Test-Case Effectiveness 5 steps to improving TCE 5 steps to improving TCE Summary Summary

3. Introduction Software testing is crucial to the success of product Software testing is crucial to the success of product Evaluation certifies product usage Evaluation certifies product usage Verification of test specifications Verification of test specifications Validation of test cases Validation of test cases New metric for evaluating test case New metric for evaluating test case Framework for test-case effectiveness Framework for test-case effectiveness

4. Improving Test-Case Effectiveness – Test-Case Effectiveness Metric Formulation of metric to measure the effectiveness of testing Formulation of metric to measure the effectiveness of testing Metric Definition: TCE = N tc /N tot * 100% Metric Definition: TCE = N tc /N tot * 100% TCE value compared to a baseline to determine effectiveness (eg 75% baseline) TCE value compared to a baseline to determine effectiveness (eg 75% baseline)

5. Improving Test-Case Effectiveness Analyze causes and areas improvements are needed Analyze causes and areas improvements are needed Proposed improvement framework based on test-escapes ( defects test case suite did not find but were side effects found in the same test cycle ) Proposed improvement framework based on test-escapes ( defects test case suite did not find but were side effects found in the same test cycle ) Approach relies on analysis of missed defects by test cases Approach relies on analysis of missed defects by test cases

6. 5 steps to Improvement 1. Thoroughly understanding and properly documenting the test process 2. Make Assumptions 3. Gathering defect data and performing causal analysis 4. Identifying main factors 5. Implementing corrective actions

7. Step 1: Thoroughly understanding and properly documenting the test process Stages in Test Process DescriptionDeliverables Test Planning defines scope, objective and test approach Test Plan Test Design designs test cases Test-Case Specifications Test preparation and execution- preparation and setup of test environment, execution of test cases, finding and reporting defects Defect Report Test evaluation and improvement analyzing testing results Summary Report

8. Step 2: Make Assumptions After completing step 1, project team makes assumptions about factors affecting TCE After completing step 1, project team makes assumptions about factors affecting TCE Test Planning Test Planning Test objectives defined in test plan should be traceable to functional specifications Test objectives defined in test plan should be traceable to functional specifications Incomplete specifications lead to an incomplete test plan and leads to reduced TCE Incomplete specifications lead to an incomplete test plan and leads to reduced TCE

9. Step 2: Make Assumptions Test Design Test Design Necessary test cases to test logic of functional features need to be defined, if not adequate coverage will be lacking Necessary test cases to test logic of functional features need to be defined, if not adequate coverage will be lacking Test case specifications should correspond to business rules Test case specifications should correspond to business rules Deficiencies could be caused by: Deficiencies could be caused by: Incomplete test suite (all possible test cases are not defined) Incomplete test suite (all possible test cases are not defined) Incomplete test cases – proper logic is not tested – negative test cases Incomplete test cases – proper logic is not tested – negative test cases Incorrect test-specifications Incorrect test-specifications Test Preparation and Execution Test Preparation and Execution Test cases not executed Test cases not executed Test cases improperly executed Test cases improperly executed

10. Factors affecting test-case effectiveness Figure 1: Factors affecting test-case effectiveness

11. Step 3: Gathering defect data and performing causal analysis Figure 2: Factors based on causal analysis logic

12. Step 3: Gathering defect data and performing causal analysis Testers need to track defects and categorize defects. Test-escapes are further categorized according to (fig. 2) Testers need to track defects and categorize defects. Test-escapes are further categorized according to (fig. 2) Understand reasons for missed defects Understand reasons for missed defects Verify completeness of test specifications Verify completeness of test specifications Verify coverage of test-cases Verify coverage of test-cases Check for test deficiencies Check for test deficiencies Verify the correctness of test cases Verify the correctness of test cases

13. Step 4: Identifying main factors Identifying factors that are causing major defects Identifying factors that are causing major defects Analyze these defects and focus on these as most needing corrective measures Analyze these defects and focus on these as most needing corrective measures

14. Step 4: Identifying main factors Figure 3: Pareto chart

15. Step 5: Implementing corrective actions Depending on project at hand, corrective measures could vary. These may include: Depending on project at hand, corrective measures could vary. These may include: Revisiting and revising functional specifications, which results in revising test case specifications Revisiting and revising functional specifications, which results in revising test case specifications Use of traceability matrix to meet coverage needs of the project – ensures that test cases meet business rule Use of traceability matrix to meet coverage needs of the project – ensures that test cases meet business rule Training for testers on proper design techniques, using checklists to help better design test case specifications (involves revisiting and revising test case specifications) Training for testers on proper design techniques, using checklists to help better design test case specifications (involves revisiting and revising test case specifications) Training for testers on proper way to execute tests and verifying test results Training for testers on proper way to execute tests and verifying test results

16. Summary New framework is to aid in better effectiveness of test-cases, however its not applicable to some projects as it requires large number of defects New framework is to aid in better effectiveness of test-cases, however its not applicable to some projects as it requires large number of defects Framework would theoretically help to reduce defects present in a product when released to production, since it first checks the test process and if its not effective then provides steps for achieving effectiveness Framework would theoretically help to reduce defects present in a product when released to production, since it first checks the test process and if its not effective then provides steps for achieving effectiveness Formalized way of achieving effectiveness, other ways for improving effectiveness could include the use of other testing techniques Formalized way of achieving effectiveness, other ways for improving effectiveness could include the use of other testing techniques