University of Southern California Center for Systems and Software Engineering Reducing Estimation Uncertainty with Continuous Assessment Framework Pongtip Aroonvatanaporn Annual Research Review March 8, 2011

University of Southern California Center for Systems and Software Engineering Outline Introduction and Motivation Framework Model Experiment Results Conclusion and future work Extending the framework 03/08/20112© USC-CSSE

University of Southern California Center for Systems and Software Engineering The Cone of Uncertainty 03/08/20113© USC-CSSE Inexperienced teams Experienced teams Also applies to project estimation accuracy

University of Southern California Center for Systems and Software Engineering Definition Inexperience –Inexperienced in general –Experienced, but in a new domain –Anything that is new with little knowledge or experience 03/08/2011© USC-CSSE4

University of Southern California Center for Systems and Software Engineering The Problem Experienced teams can produce better estimates –Use “yesterday’s weather” –Past projects of comparable size –Past data of team’s productivity –Knowledge of accumulated problems and solutions Inexperienced teams do not have this luxury No tools or data that monitors project’s progression within the cone of uncertainty 03/08/20115© USC-CSSE

University of Southern California Center for Systems and Software Engineering The Problem Imprecise project scoping –Overestimate vs. underestimate Manual assessments are tedious –Complex and discouraging Project estimation not revisited –Insufficient data to perform predictions –Project’s uncertainties not adjusted Limitations in software cost estimation –Models cannot fully compensate for lack of knowledge and understanding 03/08/20116© USC-CSSE

University of Southern California Center for Systems and Software Engineering The Goal Develop a framework to address mentioned issues Help unprecedented projects track project progression Reduce the uncertainties in estimation –Achieve eventual convergence of estimate and actual Must be quick and easy to use 03/08/20117© USC-CSSE

University of Southern California Center for Systems and Software Engineering Benefits Improve project planning and management –Resources and goals Improved product quality control Actual project progress tracking –Better understanding of project status –Actual progress reports 03/08/20118© USC-CSSE

University of Southern California Center for Systems and Software Engineering Outline Introduction and Motivation Framework Model Experiment Results Conclusion and future work Extending the framework 03/08/20119© USC-CSSE

University of Southern California Center for Systems and Software Engineering Estimation Model Integration of the Unified Code Count tool and COCOMO II estimation model 03/08/201110© USC-CSSE Adjusted with REVL

University of Southern California Center for Systems and Software Engineering Outline Introduction and Motivation Framework Model Experiment Results Conclusion and future work Extending the framework 03/08/201111© USC-CSSE

University of Southern California Center for Systems and Software Engineering Experiment Setup Performed simulation on 2 projects from USC software engineering course Project similarities –Real-client –SAIV: 24-weeks –Architected agile process, 8-member team –Size, type, and complexities –Product E-services Web content management system JSP, MySQL, Tomcat 03/08/201112© USC-CSSE

University of Southern California Center for Systems and Software Engineering Obtaining Data Source code files retrieved from Subversion server Simulation of assessment done weekly Both teams were closely involved –Provide estimation of module completion –Rationale 03/08/201113© USC-CSSE

University of Southern California Center for Systems and Software Engineering Outline Introduction and Motivation Framework Model Experiment Results Conclusion and future work Extending the framework 03/08/201114© USC-CSSE

University of Southern California Center for Systems and Software Engineering Results Accumulated effort Initial estimate Adjusted estimate ~50 % ~18% 03/08/201115© USC-CSSE

University of Southern California Center for Systems and Software Engineering Results Project progress reaches 100% –Reflects reality Estimation errors reduced to 0% 03/08/201116© USC-CSSE

University of Southern California Center for Systems and Software Engineering Outline Introduction and Motivation Framework Model Experiment Results Conclusion and future work Extending the framework 03/08/201117© USC-CSSE

University of Southern California Center for Systems and Software Engineering Conclusion Both teams demonstrated the same phenomenon –Gaps in estimation errors decrease –Representation of “cone of uncertainty” Estimation framework reflects the reality of project’s progress Assessment process was quick and simple –Requires few inputs –Little analysis needed Assessment framework help inexperienced team improve project tracking and estimation 03/08/201118© USC-CSSE

University of Southern California Center for Systems and Software Engineering Future Work Tool development currently in progress Determine the frequencies of assessments required –The sweet spot Observe prediction accuracies –Experiment on projects of larger scale –Experiment on projects of different types Use concept of value-based –Apply weights to calculation of each software module based on priorities and criticalities –How to adjust COCOMO parameters 03/08/201119© USC-CSSE

University of Southern California Center for Systems and Software Engineering Outline Introduction and Motivation Framework Model Experiment Results Conclusion and future work Extending to the Framework 03/08/201120© USC-CSSE

University of Southern California Center for Systems and Software Engineering Problem and Motivation No effective way to improve COCOMO estimation accuracy –How to adjust the parameters? –Often done by guessing, not accurate, not realistic Need an approach to assess team performance –Good performance is essential in achieving successful project –Aid in stabilization and synchronization of the team 03/08/2011© USC-CSSE21

University of Southern California Center for Systems and Software Engineering Approach Develop an assessment framework to aid COCOMO estimation Determine the criteria for adjusting COCOMO parameters Develop a survey-based approach to implement the framework –Questions correlate to COCOMO cost drivers and scale factors 03/08/2011© USC-CSSE22

University of Southern California Center for Systems and Software Engineering Survey Question Development Team members assess and evaluate their own team –operational concept engineering, requirements gathering, business analysis, collaboration, etc. Identify strengths, weaknesses, issues. –Things done right and things done wrong Survey questions derived based on these assessment data 03/08/2011© USC-CSSE23 Team’s assessment performed by each team member Strengths, weaknesses, issues, etc. Survey questions

University of Southern California Center for Systems and Software Engineering Question  COCOMO How to determine the effects? –Which COCOMO parameters are affected by the question? –How much? Two approaches: –Finding correlation –Expert advice 03/08/2011© USC-CSSE24

University of Southern California Center for Systems and Software Engineering Team Performance Correlation 03/08/2011© USC-CSSE25 Issues filed by IV&Vers Team grades Client satisfaction Estimated effort from COCOMO Actual effort reported by team Survey questions Survey Compare Correlation

University of Southern California Center for Systems and Software Engineering Expert Advice Survey experts for opinions Provide ratings for weights on effects –Relationship between question and COCOMO parameters –Some questions affect COCOMO drivers more than others Ratings can be averaged to determine actual weights 03/08/2011© USC-CSSE26

University of Southern California Center for Systems and Software Engineering Expert Advice SubjectQuestionProductPlatformPersonnelProject RELYDATADOCUCPLXRUSETIMESTORPVOLACAPAPEXPCAPPLEXLTEXPCONTOOLSCEDSITE Prototyping Do you prototype regularly to verify the validity of requirements among all the stakeholders?0.5NA NA NA0.20.4NA Prototyping Do the prototypes address the critical use- cases, GUI issues, and risks and evolve over time as requirements change?0.9NA NA NA 0.5 Business process Do you understand the current and proposed business process or workflow? This includes the current system that is being utilized and what needs to be improved0.3NA NA /08/2011© USC-CSSE27 Relationship between question/answer and COCOMO parameter it affects

University of Southern California Center for Systems and Software Engineering Benefit Improved process to adjusting project estimates –Adjustments made based on data, not guesses Faster convergence of “Cone of Uncertainty” –Less range of possible product outcomes –More accurate project tracking 03/08/2011© USC-CSSE28

University of Southern California Center for Systems and Software Engineering Tool Support Utilize IBM Jazz Server technology –Focus on distributed environment support –Provides services for user management, team management, project management Integration of Unified Code Count tool and COCOMO model –Automatic code counting –Effort calculation –COCOMO parameter adjustments 03/08/2011© USC-CSSE29

University of Southern California Center for Systems and Software Engineering Conclusion A semi-automated framework to reduce estimation uncertainty throughout project life cycle Integration of Code Count and COCOMO –Helps track project progress –Shows improved estimates Survey-based assessment –Aid in team synchronization and stabilization –Automatically adjust COCOMO parameters to reflect reality of project 03/08/2011© USC-CSSE30

University of Southern California Center for Systems and Software Engineering Questions? 03/08/2011© USC-CSSE31

University of Southern California Center for Systems and Software Engineering Backup Slides 03/08/2011© USC-CSSE32

University of Southern California Center for Systems and Software Engineering 03/08/2011© USC-CSSE33

University of Southern California Center for Systems and Software Engineering 03/08/2011© USC-CSSE34

University of Southern California Center for Systems and Software Engineering 03/08/2011© USC-CSSE35

University of Southern California Center for Systems and Software Engineering 03/08/2011© USC-CSSE36

University of Southern California Center for Systems and Software Engineering 03/08/2011© USC-CSSE37

University of Southern California Center for Systems and Software Engineering References 1.Boehm, B., Abts, C., Brown, A. W., Chulani, S., Clark, B. K., Horowitz, E., Madachy, R., Reifer, D. J., and Steece, B. Software Cost Estimation with COCOMO II, Prentice-Hall, Cohn, M. Agile Estimating and Planning, Prentice-Hall, DeMarco, T. Controlling Software Projects: Management, Measurement, and Estimation, Yourdon Press, Fleming, Q. W. and Koppelman, J. M. Earned Value Project Management, 2 nd edition, Project Management Institute, Jorgensen, M. and Boehm, B. “Software Development Effort Estimation: Formal Models or Expert Judgment?” IEEE Software, March-April 2009, pp Nguyen, V., Deeds-Rubin, S., Tan, T., and Boehm, B. "A SLOC Counting Standard," COCOMO II Forum Stutzke, R. D. Estimating Software-Intensive Systems, Pearson Education, Inc, /08/201138© USC-CSSE

University of Southern California Center for Systems and Software Engineering Related Work Software estimation methods –Estimating Software-Intensive Systems [Stutzke, 2005] –Expert-judgement vs. parametric-model [Jorgensen, 2007] –Agile estimation [Cohn, 2005] Software estimation uncertainty –PERT sizing methods [Nguyen, 2007] –Wideband Delphi estimate distributions [Boehm, 2000] Software project tracking methods –Controlling Software Projects [DeMarco, 1982] –Earned Value Management [Fleming, 2000] 03/08/201139© USC-CSSE