Local Calibration: How Many Data Points are Best? Presented by Barry Boehm on behalf of Vu Nguyen, Thuy Huynh University of Science Vietnam National University - Ho Chi Minh city, Vietnam

23/1/2016 COCOMO Forum 2015 Outline Motivation and Objectives Methods Data set Results Conclusions

33/1/2016 COCOMO Forum 2015 Motivation Importance of local calibration for adapting estimation model in organizations Projects used for calibration affect model performance Small organizations lack of data while large ones have abundance for calibration Old data may become irrelevant for training models to estimate future projects

43/1/2016 COCOMO Forum 2015 Objectives Our studies attempt to address the following questions:  How many data points are best for calibrating COCOMO models?  How much old past data can be used for calibrating COCOMO models?

53/1/2016 COCOMO Forum 2015 Moving windows A technique to select training sets, previously investigated in some studies [1][2][3] All data points/projects within a window are used as a training set A window has a size, either the number of projects or time duration Time Training set Estimating period Window moving direction

63/1/2016 COCOMO Forum 2015 COCOMO calibration COCOMO II effort formula  EM and SF are effort multipliers and scale factors, respectively  A and B are constants This study calibrates only A and B constants

73/1/2016 COCOMO Forum 2015 Outline Motivation and Objectives Methods Data set Results Conclusions

83/1/2016 COCOMO Forum 2015 Applying moving windows All projects within a windows are used to calibrate COCOMO constants A and B Only projects within one year succeeding the window are estimated (estimating period) Variable window size: different number of projects and years Time Estimating period: 1 year Window moving direction Window 1 Window 2 Window n …

93/1/2016 COCOMO Forum 2015 Applying moving windows – 2 For each window, calibrate COCOMO using projects in the window  Use the calibrated model to estimate projects in the estimating period  Compute MRE’s for estimated projects Increase window size and repeat above steps Move window one year forward Compute Magnitude of Relative Errors (MRE)

103/1/2016 COCOMO Forum 2015 Data Set Total of 341 projects completed between 1970 and 2009  including 161 projects used to calibrate COCOMO II.2000  from 25 organizations Number of projects each year from 1970 to 2009

113/1/2016 COCOMO Forum 2015 Outline Motivation and Objectives Methods Data set Results Conclusions

123/1/2016 COCOMO Forum 2015 How many data points are best for calibrating COCOMO models? Lowest mean MRE’s obtained with window of 10 – 25 data points More data points for calibration do not necessarily result in best calibrated models

133/1/2016 COCOMO Forum 2015 Best window sizes (project) Best window sizes with lowest MRE’s vary by year In most years, best window sizes are below 50 projects

143/1/2016 COCOMO Forum 2015 How much old past data can be used for calibrating COCOMO models? Mean MRE’s increase when using older past data Best model performance can be achieved with past data within 5 years

153/1/2016 COCOMO Forum 2015 Best window sizes (year) Best sizes with lowest MRE’s vary by year Recent years ( ), best sizes are less than 5 years

163/1/2016 COCOMO Forum 2015 Outline Motivation and Objectives Methods Data set Results Conclusions

173/1/2016 COCOMO Forum 2015 Conclusions Best numbers of projects and years to select data for calibrating COCOMO vary by year But, generally, calibrating between 10 and 25 data points and within 5 years for COCOMO models is best Counter-intuitively, more data points used for calibration do not necessarily result in high model accuracy Legacy data may become irrelevant for calibrating models to estimate future projects

183/1/2016 COCOMO Forum 2015 Future study Analyze the issue of why best window sizes vary significantly by year Take into account organizations in the analysis of best window sizes Apply different calibration methods in answering the objective questions

Thank You

203/1/2016 COCOMO Forum 2015 References [1] C. Lokan, E. Mendes, “Applying moving windows to software effort estimation”, in: Proceedings of the rd International Symposium on Empirical Software Engineering and Measurement, IEEE Computer Society, 2009, pp. 111–122. [2] S. Amasaki, C. Lokan, “The effects of moving windows to software estimation: comparative study on linear regression and estimation by analogy”, in: IWSM/Mensura’12, [3] C. Lokan, E. Mendes, “Investigating the use of duration-based moving windows to improve software effort prediction”, in: K. R. P. H. Leung, P. Muenchaisri (Eds.), APSEC, IEEE, 2012, pp. 818–827.