Local Bias and its Impacts on the Performance of Parametric Estimation Models Accepted by PROMISE2011 (Best paper award) Ye Yang, Lang Xie, Zhimin He (iTechs) Qi Li, Vu Nguyen, Barry Boehm (USC) Ricardo Valerdi (MIT)

Agenda  Background  Research questions  Measuring local bias  Measuring the impacts of local bias  Handling Local Bias  Conclusions and future work 2

Background  COCOMO II model Proposed by Dr. Barry Boehm; one of the most accurate cost estimation models; widely adopted by industry. Typical parametric estimation model, need tune parameters against local data (local calibration) Organization 1 Organization 2 General Model 3

Background (Cont.)  Model usage circle Local calibration relies on local historical data and domain knowledge, i.e. with local assumptions. In most cases, such local assumptions vary from the general model assumptions. It is possible that the mismatches between “general assumptions” and “local assumptions” will result in surprising calibration results. E.g., counter-intuitive calibration results: negative values of regression coefficients for level of programmer capability (PCAP), indicating higher PCAP leads to higher effort. 4

Research questions  Research questions: Is there a way to measure the local bias introduced in the model localization (local calibration) stage? As the historical data accumulates from multiple companies, how will the associated local bias impact the performance of the general parametric estimation model? Are there any correlation patterns between local bias and model performance variation after incorporating local dataset into the calibration dataset?  Assumptions: The general parametric model follows a similar structure as the COCOMO II. In model localization stage, constant A and constant B are tuned with local data. In model usage stage, locally calibrated A and B are used for project estimation. 5

Measuring local bias  Definition of local bias: where A’and B’are model parameters calibrated from local data of each organization, A and B are default constant values of COCOMO II model (A=2.94, B=0.91), and in our study we set Size=100KLOC. 6

Measuring local bias (cont.)  Data sets CII 2010 data set; contains two subsets: the CII2000 subset (161 data points from 16 organizations) and the After2000 subset (92 additional data points newly collected from 10 different organizations since year 2000) 7

Measuring local bias (cont.)  Analysis procedure Divide After2000 subset into 10 groups according to their corresponding organization. For each group, we conduct a representative local calibration using data in that group only and produce its local A’ and B’. Calculate the corresponding local bias value of each group. Compare local bias values among all groups. CII 2000 SubsetAfter2000 Subset Subset 1 … A, B A 1 ’, B 1 ’ A 2 ’, B 2 ’ A n ’, B n ’ local_bias 1 local_bias 2 local_bias n CII 2010 Dataset Subset 2 Subset n Group by Organization_ID Default Constants: A, B 8

Measuring local bias (cont.) Parameters of local models:Local bias of each group: Different local A and B in each group, indicating local bias introduced when adopting local calibration; Local bias varies in different group, ranging from 0.06 to 2.25; the local bias measures how much relative error the corresponding local model will produce. 9

Measuring the impacts of local bias  Analysis procedure First, for each group ss i in the After2000 subset: 1.combine ss i with CII 2000 data set to produce a new data set ds i ; 2.Assessing model performance on data set ds i, record values of performance indicators; Then conduct correlation analysis between local bias and model performance CII 2000 subsetI SS1 Performance Local bias CII 2000 subsetI SS2 Performance Local bias …… Correlation analysis 10

Measuring the impacts of local bias  Performance assessment Basic performance indicators: MMRE (mean MRE), stdMRE (the variance of MRE) Assessment procedure: In our study, we employ Average MMRE, Range of MMRE, Average stdMRE, and Range of stdMRE to assess the performance of an estimation model. Spliting data set into training set and test set Tuning model parameters on training set Evaluating model performance on test set MMRE, stdMRE Average MMRE Range of MMRE Average stdMRE Range of stdMRE Repeat the above steps for 2000 times 2000 (MMRE, stdMRE) pairs 11

Measuring the impacts of local bias(cont.)  Model performance 12

Measuring the impacts of local bias(cont.)  Spearman correlation coefficients between local bias and model performance: At the significant level of p-value less than 0.05, the range of stdMRE is significantly positive correlated with local bias and local_bias*num. Both the average stdMRE and the average MMRE are significantly positive correlated with local_bias*num. Range of stdMRE reflects the uncertainty of model performance. Hence, the bigger the local bias is, the weaker the performance is. 13

Handling Local Bias  Motivation Performance of the general COCOMO II model seriously decrease on the After2000 subset! Need to calibrate a new version of COCOMO II model on the CII 2010 data set. 14

Handling Local Bias (cont.)  Local bias handling approach Assumption ： local historical data set with higher local bias presents more different pattern for cost estimation, and it should be assigned a lower weight when being used for model calibration. Constraints for weight distribution function Weight=F ( LocalBias ) IF LocalBias =0, THEN Weight =1; IF LocalBias → +∞, THEN Weight → 0; The F should be a decreasing function on interval [0, +∞). Three functions 15

Handling Local Bias (cont.) 16  Weight assigned to each organization

Handling Local Bias (cont.)  Model performance on the CII2000 subset 17  Model calibrated with equal weights performs worst on the CII2000 subset;  The general COCOMO II model performs best;

Handling Local Bias (cont.)  Model performance on the After2000 subset 18  The general COCOMO II model performans worst on the After 2000 subset  Models calibrated with weights exhibit better performance than models calibrated without weights.

Handling Local Bias (cont.)  Model performance on the whole CII 2010 data set 19  The general COCOMO II model works better on the whole CII 2010 data set than calibrated models;  Models calibrated with weights exhibit better performance than models calibrated without weights.

Conclusions  The proposed LocalBias measure can be used to quantitatively measure and analyze potential local bias associated with individual organization data subset in the overall dataset.  As historical data accumulates from multiple companies, the associated local bias will cause the range of stdMRE increase.  The correlation analysis verifies that the model performance is significantly correlated by the degree of local bias and the number of data points associated with each additional group.  Weight calibration helps to reduce impact of local bias and thus improve the usability of cross-company data for model calibration. 20

Future work  More empirical studies on other public dataset to future validate and refine results.  Develop more effective methods for reducing local bias and improving general calibration outcomes. 21

Thanks! Q&A