Optimize Quality for Business Outcomes Charlie Li, Andreas Golze, Shel Prince

First Published Book from Mercury!

Understanding Defects The majority of defects are introduced during the requirements and design phase However, the majority of defects are actually detected during user acceptance testing and in production Introduction of Defects Detection of Defects Unmanaged risks results in defects. A majority of defects are introduced during the requirements and design phase. This is due to unclear or ambiguous requirements and design specifications that often result in problematic implementations. Most of the defects are actually detected during the user acceptance testing phase. By addressing defects earlier during the requirement and design stages, many of the defects can be addressed earlier rather than later. Source: NIST 2002 RTI Project 7007.011

Defect Correction Costs 1000x in Production! Addressing defects earlier during the requirements and design phase ensures lower total cost of correcting defects. As defects are detected in later phases such as testing and maintenance the cost for remediation is exponentially higher. This industry average is used as a baseline for arriving at cost savings Industry References: 3 B. Boehm and V. Basili, "Software Defect Reduction Top 10 List," IEEE Computer, IEEE Computer Society, Vol. 34, No. 1, January 2001, pp. 135-137.

A Risk-Mitigated Quality Approach Align testing with business requirements Mitigate implementation risk with effective quality process Project Prep Blueprint Design Build Test Go-Live Support Assess Risk Test Test Manage Quality Process Risk Mitigated Approach Traditional Cost Quality Effort and Cost Time Solve the problems of time and risk by shifting quality management earlier in the lifecycle by prioritizing quality tasks based on risk assessment. Establish an effective and efficient quality process and Test according to business requirements. Optimize test automation in order to reduce the QA lifecycle.

The Missing link Who’s testing for the business outcomes? How do you structure requirements? Requirements -> behavioral model -> Test Cases What to test? How to Test? How much to test? What test strategies to use and how can automation be leveraged? What do you measure, assess, and continuously improve?

Behavioral Modeling

What is it? A graphical representation… ...of the functional behavior… …of the software article.

What does it buy us? Allows the behavior to be specified at any level of detail. Allows different people to work on different sections at the same time. Limited “vocabulary” enforces a high level of consistency in the models. Allows understanding (and review) by people with a wide variety of experience.

Vocabulary Beginning or End Process Decision Case Result Displayed Result Manual Action Representative Sample Straight Line Connector Note Curved Line Connector

How do we do it? There are only two questions -- What does it do? “Do” in the active sense What influences it?

Sample Problem 1 Externals only Example – Bigger: Two numbers in One number out +1 if 1st number bigger than 2nd -1 if 1st number smaller than 2nd 0 if the numbers are equal

Sample Problem 2 FAST The Functionally Advanced Sidewalk Teller Deposit Withdraw Transfer Check Balance

Attributes of Good Test Suites Effective Efficient

Test Case Design Sample

Business Impact Testing

The Specification

The User Acceptance Test

Improvements

The Retest

Tracking Risk Ratio Baseline the risk ratio and set the risk value to 1; current risk/initial risk Adding new requirements post-analysis increases the risk ratio; enables risk-based discussions Risk analysis acts as a verification of the requirements Risk hopefully reduces over the lifecycle Analyze Build Design Test Deploy Test Activities R I S K A T O 1 Risk Ratio Analysis An effective way to normalizing and tracking risk, at both a requirement and application level, is to get the ratio of current risk to baseline risk. The ratio may be calculated as follows: For a single requirements: ratio = (current / baseline). For entire application: ratio = average of (current / baseline) across all in-scope requirements. The ratio always starts at 1 (where current = baseline risk), and hopefully diminishes as the requirements are designed, built, and tested. The goal is to reduce the risk ratio to zero or acceptable levels. Adding new requirements post-analyze will always increases the risk ratio. This enables the business, development, and QA teams to engage in risk-based discussions and decision making. It formalizes ad-hoc processes to determine options to adjust test plans if schedule or resource constraints occur.

Test Requirements Structure Organization of Test Requirements Applications support business activities/processes and functions Structure is uniform and enforced, and naming convention is informative Example 1 Example 2 35

Business Impact Driven Test Strategy Analysing Probability Assess the business impact of each activity Assess the Probability for failure Determine the Risk based on Impact and probability Determining the Risk 36

The Risk determines the Test Strategy Procedure for High Risk A Procedure Systematic testing using business component testing strategy, and root-cause analysis Approach Automated: 30% Manual: 70% Systematic testing with or without business component testing strategy, and root-cause analysis Procedure for Medium Risk B Procedure Approach Automated: 20% Manual: 80% Procedure for Low Risk C Procedure Systematic or Ad-hoc testing Approach Automated: 5% Manual: 95% 37

Functional Complexity determines Test effort Structuring Test Requirements 38

Prioritizing Automation Effort Prioritization Strategy Automation ROI Matrix Assuming test cases for all risks and complexities are in-scope for each test cycle. Testing Scope Defined by risk-levels to be tested (ie. Higher risk items tested first) Prioritization Order for Automation: Test cases with lowest # of test cycles for ROI Test cases with highest risk levels first Test cases with lowest complexity first # of Test Cycles for Beginning of Return on Automation Complexity Risk High Medium Low 5.0 5.9 5.5 6.6 7.3 8.1 1 4 3 5 6 8 2 7 9 In order to maximize ROI from automation, it is best to invest in test cases that will yield ROI quicker. The following guidelines for prioritization should be used, based on the scope of testing (ie. risk levels to be tested) 1st) test cases with the lowest # of test cycles for ROI should be automated first 2nd) test cases with highest risk levels – as higher risk items are more important to be covered first in any testing cycle 3rd) test cases with lowest complexity – as lower complexity items require much less maintenance than higher complexity items. In the ROI Matrix example, the ordering in which automation is performed assuming every test case is in scope is shown. # - Priority # / Order

Utilize the best testing strategies Start Project Application Available Map Requirements Test Plans Define Use Cases Manual Testing Create Test Lab Run Tests Modify Tests Analyze Tests Run Tests Modify Tests Map Requirements Test Plans Define Use Cases Test Automation Automate Tests Run Tests Analyze Tests Modify Tests Run Tests Customize Pre-Defined Content Map Requirements Test Plans Define Use Cases Test Acceleration Run Tests Analyze Tests Modify Tests Run Tests Customize Pre-Defined Content Map Requirements Test Plans Define Use Cases Change Impact Testing Run Tests Analyze Tests Modify Tests Run Tests Test What Matters!!! Time Line

Wrapping up

10 SUCCESSFUL Steps to Measure

More Book information Books will be available for purchase by 12/15 from Mercury’s Education site www.merc-training.com Send Questions to getanswers@mercury.com

Questions?