1. CS4723 Software Validation and Quality Assurance Lecture 15 Advanced Topics Test Plans and Management

2. Test Plan & Management Something you are going to think about when you are a project manager instead of a tester / developer About handle the testing progress of a whole software project About quality assurance at highest level Approach: risk based testing

3. Presentation Outline Introduction - The ChallengeWhy? Risk Analysis and Testing Risk Based Test Approach What? Test Metrics Risk Based Test Metrics Risk Analysis in Risk Based Testing Risk Analysis in Practice Plan and Manage - Metrics How? Quality and Progress Monitoring Warning! Organization and Process...

4. The Challenges Time Constraints Resource Constraints Quality Requirements Risk Factors: New technology Lack of knowledge Lack of experience Take Control!

5. Presentation Outline Introduction - The Challenge Risk Analysis and TestingWhat? Risk Based Test Approach Test Metrics Risk Based Test Metrics Risk Analysis in Risk Based Testing Theory and Practice Plan and Manage Quality and Progress Monitoring Warning! Organization and Process...

6. Risk Analysis and Testing Risk Identfication Risk Strategy Risk Assessment Risk Mitigation Risk Reporting Risk Prediction Testing, Inspection etc. Test Plan Metrics: Cost and Probability Test Item Tree Test Metrics

7. Risk Based Testing - Approach Risk Identification / Test Items Identify the items to be tested Functions, classes, features, transactions, … Risk Strategy / Test Plan: Setup approaches to generating test cases Automatic / Manual Original / Regression Load / Security / Usability /,,, Risk Assessment / Metrics Assess risk of each test items

8. Risk Based Testing - Approach Risk Mitigation / Testing Generate and execute the concrete test cases Risk Reporting Measure how much have we mitigated the risk Test coverage Found and fixed bugs Risk Prediction Based on the risk reports, predict risks of test items for new round of testing

9. Risk Based Testing - Metrics Identify Areas with “High Risk Exposure” Probability and Cost All functions/modules should be tested to a “minimum level” “Extra Testing” in areas with high risk exposure Establish Test Plan and Schedule Monitor Quality Number of Faults per function and time Monitor Progress Number of hours in test and fix -> ETC

10. The Formula Re(f) - Risk Exposure of function f P(f) - Probability of a fault in function f C(f) - Cost related to a fault in function f Risk Based Testing - Theory

11. Simple Test Metrics Test Planning Number of test cases per function Number of hours testing per function Progress Tracking Number of tests planned, executed and completed Number of faults per function Number of hours used for test and fix Estimated to Complete

12. Simple Test Metrics Probability of faults - Indicators New functionality Size Complexity Quality of previous phases and documents Cost of Faults

13. Presentation Outline Introduction - The Challenge Risk Analysis and Testing Risk Based Test Approach Test Metrics Risk Based Test Metrics Risk Analysis in Risk Based TestingHow? Theory and Practice Plan and Manage Quality and Progress Monitoring Warning! Organization and Process...

14. Risk Based Testing - Example Other Probability Factors might include: Function Points, Frequency of Use etc. Ranking the functions based on Risk Exposure The Probability of a Fault The Cost of a Fault Example:

15. Consequence Probability Low High 1 2 3 4 Risk Based Testing - Reporting LowMedium High Consequence Probability

16. Prior to test execution: identify critical transactions 1 Test Execution identifies “bad” transactions 2 Extra Testing: - Additional testing by product specialist - Automated regression testing 3 Risk Based Testing - Practice

17. Presentation Outline Introduction - The Challenge Risk Analysis and Testing Risk Based Test Approach Test Metrics Risk Based Test Metrics Risk Analysis in Risk Based Testing Theory and Practice Plan and ManageHow? Quality and Progress Monitoring Warning! Organization and Process...

18. Two Strategies On-line Risk Analysis Usually used for first time testing Cold start with little data Minimal testing and a small list of critical features provide by customers Batch Risk Analysis Assume a large amount of data at the beginning Different metrics

19. Planned Executed QAed Planning and Progress Tracking

20. Progress Indicators - “To be vs. Actual” “To be fixed” vs. “Actually fixed” To be retestedAct. retested Rejected To Be Restested, Actually Retested and Rejected Number of Faults “To be Retested” vs. “Actually Retested” Actually fixed To Be Fixed and Actually Fixed Number of Faults To Be Fixed

21. Progress Indicators - Hours Used Batch Number of hours for finding one fault and for fixing one Online Number of hours for finding one fault and for fixing one Hours per Fault for Test and Fix Date Hours per Fault Test Fix Test Fix Hours per Fault for Test and Fix Hours per Fault

22. “Estimated to Complete” ETC for system test based on: Number of hours testing per fault found Number of hours fixing per fault Number of faults found per function Number of fixes being rejected Number of remaining tests (functions to be tested) Estimated to Complete at Time t Actual to Complete at Time t

23. Benefits of Risk Based Testing Improved Quality? all critical functions tested Reduced Time and Money in Testing effort not wasted on non critical or low risk functions Improved customer confidence due to customer involvement and good reporting and progress tracking

24. Presentation Outline Introduction - The Challenge Risk Analysis and Testing Risk Based Test Approach Test Metrics Risk Based Test Metrics Risk Analysis in Risk Based TestingHow? Theory and Practice Plan and Manage Quality and Progress Monitoring Warning! Organization and Process...

25. WARNING! Require a focused Organization… Strong Test Management Testers to “drive” the Programmers Responsibility …and a well defined Process Documentation Planning and Execution Procedures Control Procedures Planning and Progress Tracking

26. LD PD Test Exec PTDs Raised Fix CR Fix Procedure Problem Mngmnt. Procedure Test Exec. Procedure Case Build Procedure Case Quality Standards Change Mngmnt. Procedure Re- test Good/ Bad Test Completed Regression Test Good Bad ProAte QC / QA Basic Test Data Test Case Risk Identification Risk Assessment Risk Mitigation Risk Reporting Risk Prediction Test Process Work Flow

27. Summary Risk Based Test Approach Focused Testing Reduced Resources Improved Quality Metrics are fundamental Process and Organization must support the new strategy Metrics must support the organization and process