1. Validating and Improving Test-Case Effectiveness Author: Yuri Chernak Presenter: Lam, Man Tat

2. Agenda Some definition that need to be understand How verification and validation is done Overview of how in process validation of test cases is done Case-studyConclusion

3. Functional testing By focusing on a system's functionality Looking for as many defects as possible Test plan documents and test case specifications are important deliverables

4. Test Cases Critical for effective testing Testers use test-case specifications does not guarantee that systems are sufficiently tested

5. Verification First verify test case specifications at the end of the test design phase Reviews or inspections – evaluate test case specifications for their correctness and completeness Test cases that passed verification could have weak failure-detecting ability

6. Validation Can process as soon as testers have executed all test cases Determines whether the test cases were sufficiently effective in finding defects If test cases effectiveness has not proved satisfactory, it is not too late to analyze the reasons and correct the test process Test case effectiveness metric introduced

7. Test case effectiveness metric Mainly use for online client server systems Certain number of defects are always found as a side effect Assume that the more defects test cases find, the more effective they are Define as the ratio of defects found by test cases (Ntc) to the total number of defects (Ntot) reportes during the test cycle

8. Test case effectiveness metric Cont’ TCE = Ntc / Ntot *100% TCE metric evaluates test cases from the test cycle perspective, which provides in process feedback to the project team on how well a test suite has worked for testers Compare the actual TCE for a given project with a baseline value (such as previous successful project, author experience suggests 75 %)

9. Test case effectiveness metric Cont’ If the TCE value is at baseline or above, we can conclude that the test cases have been sufficiently effective in test cycle If the TCE value falls below the baseline, the higher is the risk of user dissatisfaction

10. Improving test case effectiveness If in process validation finds test case effectiveness to be less than acceptable, the project team should analyze the cause and identify areas that need to be improve Author’s proposed improvement framework was based on IBM’s “Test case escape”

11. IBM’s “Test case escape” Define as “product defects that a particular test failed to find, but which were found in a later test, or by a customer [in production]

12. Author’s proposed improvement framework Define as software defects that a given suite of test cases failed to find but that were found as a side effect in the same test cycle Rather than compare the Ntot with the whole process

13. Author’s proposed improvement framework 1- Understand and document the test process used by the project team 2- Make assumptions about factors that affect test cases 3- Gather defect data 4- Identify main factors 5- Implement corrective actions

14. Understand, document the test process Test planning – definition of the scope, objectives, and approach to testing Test design – the design of the test cases Test preparation and execution – preparation of the test environment executing test cases and finding defects Test evaluation and improvement – analyzing the results of testing

15. Make assumption Once understands and documents the test process, the project team should analyze each phase and identify factors that can affect test case effectiveness Lets look at some example

16. Example of making assumption Test planning – if the functional specifications do not completely define functional features, the test plan document will not be complete either. So, the test cases will not completely cover a system’s functionality

17. Example of making assumption 2 Test design – a common example could be lack of negative test cases (such as abnormal conditions by using either invalid data input or the wrong user action Test design – test case specifications could be simply incorrect

18. Factors affecting test case effectiveness

19. Gather defect data and perform analysis Defect tracking system Able to identify which defects were found as result of test cases and which were found as a side effect Once selected, defects should be classified according to one of the factors based on the analysis

20. Test Case escape classification logic

21. Gather defect data and perform analysis cont’ Analysis is used to evaluate each test case and understand why the test suite missed the corresponding defect during the test execution Example: A lack of negative test cases in is a common cause of missed defects. Can you see which category it will be under?

22. Identify the main factors At this point, all test case escapes have been classified according to their respective causes Build a “Pareto chart” which displays frequency bars in descending order, convenient for analyzing (will be an example later)

23. Implement corrective actions After identification of the main causes of test case escapes, the project team should implement corrective actions and repeat the test execution

24. Example of corrective actions Rework functonal specifications and test case specification Use a traceability matrix to ensure complete coverage of business rules Use checklists to design test case specification Implement training of testers on test execution

25. Case study Banking application intended for external clients Project team consisted 10 developers and 3 testers 183 defects reported 71 found by test case escapes 112 found by test case execution

26. Case study cont’ TCE = 61% Lower than the acceptable level of 75% Project team performed the test process improvement according to the framework Pareto chart created for analysis

27. Example of Pareto chart

28. Case study cont’ Analysis of the distribution showed incomplete test design and incomplete functional specification to be the main factors causing missed defects. Project team correcting and completing the test design and functional specification

29. Case study cont’ As a result, 48 additional defects found before releasing the product By the end of the second month the number of production, only 23 defects was found Indicated sufficient effectiveness of test progress, user should be satisfied with the system quality

30. Conclusion This technique intended to give project teams better visibility into test process effectiveness before their system are released into production Help evaluate each test case and understand why the test suite missed the corresponding defect during test execution

31. Conclusion Help testers revise and improve the test suite In other word, it helps testers find additional defects and deliver a better software product

32. If my presentation did not put you into sleep and you have a question… Now its time!!!