© USC-CSE1 Determine How Much Dependability is Enough: A Value-Based Approach LiGuo Huang, Barry Boehm University of Southern California

© USC-CSE2 Software Dependability Business Case Software dependability in a competitive world –Software dependability requirements often conflict with schedule/cost requirements How much dependability is enough? –When to stop testing and release the product – Determining a risk-balanced “sweet spot” operating point

© USC-CSE3 Competing on Schedule and Dependability – A risk analysis approach Risk Exposure RE = Prob (Loss) * Size (Loss) –“Loss” – financial; reputation; future prospects, … For multiple sources of loss: RE =  [Prob (Loss) * Size (Loss)] source sources

© USC-CSE4 Example RE Profile: Time to Ship – Loss due to unacceptable dependability Time to Ship (amount of testing) RE = P(L) * S(L) Many defects: high P(L) Critical defects: high S(L) Few defects: low P(L) Minor defects: low S(L)

© USC-CSE5 Example RE Profile: Time to Ship - Loss due to unacceptable dependability - Loss due to market share erosion Time to Ship (amount of testing) RE = P(L) * S(L) Few rivals: low P(L) Weak rivals: low S(L) Many rivals: high P(L) Strong rivals: high S(L) Many defects: high P(L) Critical defects: high S(L) Few defects: low P(L) Minor defects: low S(L)

© USC-CSE6 Example RE Profile: Time to Ship - Sum of Risk Exposures Sweet Spot Time to Ship (amount of testing) RE = P(L) * S(L) Few rivals: low P(L) Weak rivals: low S(L) Many rivals: high P(L) Strong rivals: high S(L) Many defects: high P(L) Critical defects: high S(L) Few defects: low P(L) Minor defects: low S(L)

© USC-CSE Slight inconvenience (1 hour) Low, easily recoverable loss Moderate recoverable loss High Financial Loss Loss of Human Life 300 hours 10 hours 10K hours 300K hours Defect RiskRough MTBF(mean time between failures) Commercial quality leader 1.10 In-house support software 1.0 Commercial cost leader Startup demo Safety-critical 1.26 Relative Cost/Source Instruction High RELY Rating Very High Nominal Low Very Low Software Development Cost/Reliability Tradeoff - COCOMO II calibration to 161 projects

© USC-CSE8 Current COQUALMO System COCOMO II COQUALMO Defect Introduction Model Defect Removal Model Software platform, Project, product and personnel attributes Software Size Estimate Defect removal profile levels Automation, Reviews, Testing Software development effort, cost and schedule estimate Number of residual defects Defect density per unit of size

© USC-CSE9 Highly advanced tools, model- based test More advance test tools, preparation. Dist-monitoring Well-defined test seq. and basic test coverage tool system Basic test Test criteria based on checklist Ad-hoc test and debug No testing Execution Testing and Tools Extensive review checklist Statistical control Root cause analysis, formal follow Using historical data Formal review roles and Well- trained people and basic checklist Well-defined preparation, review, minimal follow-up Ad-hoc informal walk-through No peer review Peer Reviews Formalized specification, verification. Advanced dist- processing More elaborate req./design Basic dist- processing Intermediate- level module Simple req./design Compiler extension Basic req. and design consistency Basic compiler capabilities Simple compiler syntax checking Automated Analysis Extra HighVery HighHighNominalLowVery Low COCOMO II p.263 Defect Removal Rating Scales

© USC-CSE10 Defect Removal Estimates - Nominal Defect Introduction Rates (60 defects/KSLOC) Delivered Defects / KSLOC Composite Defect Removal Rating (1.0) (.475) (.24) (.125) (.06) (.03) Prob. of Loss P(L)

© USC-CSE11 Relations Between COCOMO II and COQUALMO COQUALMO rating scales for levels of investment in defect removal via automated analysis, peer reviews, and execution testing and tools have been aligned with the COCOMO II RELY rating levels.

© USC-CSE12 Typical Marketplace Competition Value Estimating Relationships Market Share Loss VL(T d ) System Delivery Time T d Fixed-schedule Event Support: Value of On-time System Delivery Market Share Loss VL(T d ) System Delivery Time T d Off-line Data Processing: Value Loss vs. System Delivery Market Share Loss VL(T d ) System Delivery Time T d Internet Services, Wireless Infrastructure: Value Loss vs. System Delivery Time

© USC-CSE13 How much Dependability is Enough? - Early Startup: Risk due to low dependability - Commercial: Risk due to low dependability - High Finance: Risk due to low dependability - Risk due to market share erosion COCOMO II: Added % test time COQUALMO: P(L) Early Startup: S(L) Commercial: S(L) High Finance: S(L) Market Risk: RE m Sweet Spot

© USC-CSE14 20% of Features Provide 80% of Value: Focus Testing on These (Bullock, 2000) Customer Type % of Value for Correct Customer Billing Automated test generation tool - all tests have equal value

© USC-CSE15 Value-Based vs. Value-Neutral Testing – High Finance COCOMO II: Added % test time COQUALMO: P(L) Value-based: S(L): Exponential Value-Neutral: S(L): Linear Market Risk: RE m Sweet Spot

© USC-CSE16 Reasoning about the Value of Dependability – iDAVE iDAVE: Information Dependability Attribute Value Estimator Extend iDAVE model to enable risk analyses – Determine how much dependability is enough

© USC-CSE17 iDAVE Model Framework

© USC-CSE18 Usage Scenario of iDAVE Combined Risk Analyses - How much Dependability is Enough? 1.Estimate software size in terms of value-adding capabilities. 2.Enter the project size and cost drivers into iDAVE to obtain project delivered defect density (= (defects introduced – defects removed)/KSLOC) for the range of dependability driver (RELY) ratings from Very Low to Very High. 3.Involve stakeholders in determining the sizes of loss based on the value estimating relationships for software dependability attributes. 4.Involve stakeholders in determining the risk exposures of market erosion based on the delivery time of the product. 5.Apply the iDAVE to assess the probability of losses for the range of dependability driver (RELY) ratings from Very Low to Very High. 6.Apply the iDAVE to combine the dependability risk exposures and market erosion risk exposures to find the sweet spot.

© USC-CSE19 Conclusions Increasing need for value-based approaches to software dependability achievement and evaluation –Methods and models emerging to address needs Value-based dependability cost/quality models such as COCOMO II, COQUALMO can be combined with Value Estimating Relationships (VERs) to –Perform risk analyses to determine “how much dependability is enough?” –Perform sensitivity analyses for the most appropriate dependability investment level for different cost-of-delay situations –Determine relative payoff of value-based vs. value-neutral testing