1. University of Southern California Center for Systems and Software Engineering Improving Affordability via Value-Based Testing 27th International Forum on COCOMO® and Systems/Software Cost Modeling USC-CSSE/ISCAS Qi Li, Barry Boehm, Qing Wang, Ye Yang, Song Wang October 16, 2012

2. University of Southern California Center for Systems and Software Engineering Outline Research Motivation Research Method Case Studies Future Work Conclusion 10/16/2012COCOMO Forum 20122

3. University of Southern California Center for Systems and Software Engineering Research Motivation Value-neutral SE methods are increasingly risky [Boehm, 2003] –Every requirement, use case, object, test case, and defect is equally important –System value-domain problems are the chief sources of software project failures –Existing prioritization is intuitional, not systematic and comprehensive –“Earned Value” Systems don’t track business value Testing & Inspection resources are expensive and scarce –30%-50%, even higher for high reliability projects [Ramler, 2005] –Time-to-market [Boehm, Huang, 2005] Empirical Findings [Bullock 2000, Boehm & Basili 2001 ] –About 20 percent of the features provide 80 percent of business value –About 80 percent of the defects come from 20 percent of the modules –… Value-based Software Engineering 4+1 theorem [Boehm, 2005] 10/16/2012COCOMO Forum 20123

4. University of Southern California Center for Systems and Software Engineering Outline Research Motivation Research Method Case Studies Future Work Conclusion 10/16/2012COCOMO Forum 20124

5. University of Southern California Center for Systems and Software Engineering Value-Based Software Test Prioritization What to be prioritized? Testing items: Testing Scenarios, Testing Features, Test Cases How to prioritize? Value-Based (Business Importance, Risk, Cost) Dependency Aware How to Measure? Average Percentage of Business Importance Earned (APBIE)

6. University of Southern California Center for Systems and Software Engineering Research Method: Value-Based Risk Exposure (RE) –Where Size (Loss) is the risk impact size of loss if the outcome is unsatisfactory, Pro (Loss) is the probability of an unsatisfactory outcome Risk Reduction Leverage (RRL) –Where RE before is the RE before initiating the risk reduction effort and RE after is the RE afterwards. –RRL is a measure of the cost-benefit ratio of performing a candidate risk reduction or defect removal activity 610/16/2012COCOMO Forum 2012

7. University of Southern California Center for Systems and Software Engineering Research Method: Value-Based –Business Case Analysis –Stakeholder Prioritization –Impact of Defect –Experience Base 10/16/2012COCOMO Forum 20127 –Business Value Defect Criticality –Defect-prone Components, Performers Value-Based Prioritization Drivers: –Size of Loss –Probability of Loss Risk Exposure Testing items are to be ranked by how well they can reduce RE

8. University of Southern California Center for Systems and Software Engineering Research Method: Value-Based Combining with the testing items’ relative costs =>Priority Trigger: This proposed strategy enables them to be prioritized in terms of Risk Reduction Leverage (RRL) or ROI Supposed to improve the lifecycle cost-effectiveness of defect removal techniques 10/16/2012COCOMO Forum 20128

9. University of Southern California Center for Systems and Software Engineering Animated displays Secondary application functions User amenities Main application functions Natural speech input Tertairy application functions Basic application functions Data management system Operating System InvestmentHigh-payoffDiminishing returns Cost of software product [Boehm, 1981] Value of software product to organization Research Method: Dependency Aware 10/16/20129COCOMO Forum 2012

10. University of Southern California Center for Systems and Software Engineering Research Method: Dependency Aware Dependency: –Example: dependencies among test cases to be executed –Solution: Prioritization Algorithm (greedy alg) Select the one with the highest RRL Check dependency 10/16/2012COCOMO Forum 201210 9->3->9->5->9->4->7

11. University of Southern California Center for Systems and Software Engineering Research Method: Metrics Testing Cost Effectiveness –Average Percentage of Business Importance Earned (APBIE) 10/16/201211COCOMO Forum 2012

12. University of Southern California Center for Systems and Software Engineering Outline Research Motivation Research Method Case Studies Future Work Conclusion 10/16/2012COCOMO Forum 201212

13. University of Southern California Center for Systems and Software Engineering Case Studies Results Exercise Test Prioritization based on Risk Reduction Level (RRL) software testing scenarios to be walked through in Galorath.Inc software features to be tested in a Chinese company software test cases to be executed in USC SE course projects All of them show positive results 10/16/2012COCOMO Forum 201213

14. University of Southern California Center for Systems and Software Engineering Case Studies Results (Galorath Inc.) Prioritize testing scenarios to be walked through Case Study: –Galorath Inc. (2011 Summer) –Project: Installation Process Automation –Challenge: Testing all scenarios is impossible under limited testing resources (69 scenarios) 10/4/2012Qi Li _Defense14

15. University of Southern California Center for Systems and Software Engineering Case Studies Results (Galorath Inc.) Prioritize testing scenarios to be walked through 10/16/2012COCOMO Forum 201215 APBIE-170.99% APBIE-210.08% APBIE-332.10% –Value-based prioritization can improve the cost-effectiveness of testing Value-based Value-neutral Value-inverse (worst case)

16. University of Southern California Center for Systems and Software Engineering Case Studies Results (USC Course Projects) Prioritize software test cases to be executed Experiment –USC-CSCI 577ab –18 teams (5 2011 Spring teams + 13 Fall teams) –Acceptance testing phase 10/16/2012COCOMO Forum 201216

17. University of Southern California Center for Systems and Software Engineering Case Studies Results (USC Course Projects) Prioritize software test cases to be executed Experiment Results (Quantitative) 10/16/2012COCOMO Forum 201217 APBIE-252% APBIE-346% APBIE-181.9% –Project 1 as an example, –Value-based prioritization can improve the cost- effectiveness of testing Value-based Value-neutral Value-based

18. University of Southern California Center for Systems and Software Engineering Prioritize software test cases to be executed An automatic tool for facilitating prioritization 10/16/2012COCOMO Forum 201218 http://greenbay.usc.edu/dacs/vbt/testlink/index.php Case Studies Results (USC Course Projects)

19. University of Southern California Center for Systems and Software Engineering Case Studies Results (USC Course Projects) Prioritize software test cases to be executed (Quantitative) 19 APBIE# of TCsValue-Based Value- NeutralImprovement 2011S_T012856.41%46.38%10.03% 2011S_T022954.94%53.80%1.14% 2011S_T032251.76%50.75%1.01% 2011S_T053154.36%51.87%2.49% 2011S_T063953.07%50.40%2.67% 2011F_T011951.93%45.98%5.95% 2011F_T031452.15%50.33%1.82% 2011F_T042461.95%53.62%8.33% 2011F_T057763.21%42.07%21.14% 2011F_T063159.22%53.31%5.91% 2011F_T071057.25%56.25%1.00% 2011F_T08755.71%54.76%0.95% 2011F_T091057.27%51.51%5.76% 2011F_T101862.08%57.23%4.85% 2011F_T112553.16%51.39%1.77% 2011F_T12658.33% 0.00% 2011F_T133153.64%53.25%0.39% 2011F_T142957.24%48.17%9.07% Average 56.32%51.63%4.68% F-test0.5745 T-test0.000661 PBIE1/2 # of TCsValue-BasedValue-NeutralImprovement 2011S_T011460%40%20.00% 2011S_T021561%58%3.00% 2011S_T031152%50%2.00% 2011S_T051656%50%6.00% 2011S_T062059%51%8.00% 2011F_T011060%45%15.00% 2011F_T03750% 0.00% 2011F_T041270%50%20.00% 2011F_T053970%40%30.00% 2011F_T061665%50%15.00% 2011F_T07553%52%1.00% 2011F_T08460%50%10.00% 2011F_T09558%45%13.00% 2011F_T10963%55%8.00% 2011F_T111355%50%5.00% 2011F_T12350% 0.00% 2011F_T131651%50%1.00% 2011F_T141560%40%20.00% Average 58.50%48.67%9.83% F-test0.3822 T-test0.000083 Delivered Value Comparison when Cost is fixedAPBIE

20. University of Southern California Center for Systems and Software Engineering 20 # of TCs when gaining 50% BIValue-BasedValue-Neutral# of TCsValue-Based Cost%Value-Neutral Cost%Cost saving % 2011S_T0112172842.86%60.71%17.86% 2011S_T0213 2944.83% 0.00% 2011S_T0311 2250.00% 0.00% 2011S_T0513163141.94%51.61%9.68% 2011S_T0618213946.15%53.85%7.69% 2011F_T019111947.37%57.89%10.53% 2011F_T03771450.00% 0.00% 2011F_T048142433.33%58.33%25.00% 2011F_T0521517727.27%66.23%38.96% 2011F_T0611163135.48%51.61%16.13% 2011F_T07551050.00% 0.00% 2011F_T0844757.14% 0.00% 2011F_T09461040.00%60.00%20.00% 2011F_T10791838.89%50.00%11.11% 2011F_T1111132544.00%52.00%8.00% 2011F_T1233650.00% 0.00% 2011F_T1316 3151.61% 0.00% 2011F_T1412182941.38%62.07%20.69% Average 44.01%54.33%10.31% F-test0.2616 T-test0.000517 Cost Comparison when Delivered Value is fixed Prioritize software test cases to be executed (Quantitative) Case Studies Results (USC Course Projects)

21. University of Southern California Center for Systems and Software Engineering Prioritize software test cases to be executed Experiment Results (Qualitative) 10/16/2012COCOMO Forum 201221 “Before doing the prioritization, I had a vague idea of which test cases are important to clients. But after going through the Value-Based testing, I had a better picture as to which ones are of critical importance to the client.” “I prioritized test cases mainly based on the sequence of the system work flow, which is performing test cases with lower dependencies at first before using value-based testing. I like the value-based process because it can save time by letting me focus on more valuable test cases or risky ones. Therefore, it improves testing efficiency.” “Value-based testing is very useful in complex systems with hundreds or thousands of test-cases. However in 577 it should not be difficult to run every test-case in every test iteration, making the prioritization less useful. The impact of value-based testing and automated test management on software quality is entirely dependent on the complexity of the project. If complete test coverage is possible in the time given, the benefit of VBST to software quality is minimal.” Case Studies Results (USC Course Projects)

22. University of Southern California Center for Systems and Software Engineering Prioritize software test cases to be executed Some lessons learned from this case study –Differentiating the priority factor levels is the prerequisite of value-based prioritization –Small size project might have difficulties to prioritize those equally important core capabilities –Strong positive correlation between the size of test cases and the improvements –Even small percentage of improvement might result saving lots of budget in reality 10/4/2012Qi Li _Defense22 Case Studies Results (USC Course Projects)

23. University of Southern California Center for Systems and Software Engineering Outline Research Motivation Research Method Case Studies Future Work Conclusion 10/16/2012COCOMO Forum 201223

24. University of Southern California Center for Systems and Software Engineering Outline Research Motivation Research Method Case Studies Future Work Conclusion 10/16/2012COCOMO Forum 201224

25. University of Southern California Center for Systems and Software Engineering Conclusion Propose a Real “Earned Value” System to Track Business Value of Testing and Measure Testing Efficiency in terms of APBIE Propose a Systematic Strategy for Value-based, Dependency Aware Test Processes Apply This Strategy to a Series of Empirical Studies with different granularities of Prioritizations Elaborate Decision Criteria of Testing Priorities Per Project Contexts, Which are Helpful for Real Industry Practices Implement an automatic tool for its application on large-scale industrial projects 10/16/2012COCOMO Forum 201225

26. University of Southern California Center for Systems and Software Engineering Question and Answer 10/16/2012COCOMO Forum 201226