Presentation on theme: "Nearest Neighbor Sampling for Better Defect Prediction Gary D. Boetticher Department of Software Engineering University of Houston - Clear Lake Houston,"— Presentation transcript:
Nearest Neighbor Sampling for Better Defect Prediction Gary D. Boetticher Department of Software Engineering University of Houston - Clear Lake Houston, Texas, USA
The Problem: Why is there not more ML in Software Engineering? Human-Based 62 to 86% [Jørgensen 2004] Algorithmic Machine Learning 7 to 16%
Key Idea More ML in SE through a more defined experimental process.
Agenda A better defined process for better predicting (quality) Experiments: Nearest Neighbor Sampling on PROMISE Defect data sets Extending the approach Discussion Conclusions
A Better Defined Process Emphasis of ML approaches Emphasis on Measuring Success – PRED(X) – Accuracy – MARE Prediction success depends upon the relationship between training and test data.
PROMISE Defect Data (from NASA) 21 Inputs – Size (SLOC, Comments) – Complexity (McCabe Cyclomatic Comp.) – Vocabulary (Halstead Operators, Operands) 1 Output: Number of Defects
Assessing Experiment Difficulty Exp_Difficulty = 1 - Matches / Total_Test_Instances Match = Test vector’s nearest neighbor is from the same class instance in the training set. Experimental Difficulty = 1 Experimental Difficulty = 0 Hard experiment Easy experiment
Assessing Overall Data Difficulty Overall Data Difficulty = 1 - Matches / Total_Data_Instances Match = A data vector’s nearest neighbor is from the same class instance as another vector in the data set. Overall Data Difficulty = 1 Overall Data Difficulty = 0 Difficult Data Easy Data
Discussion: Anticipated Benefits Method for characterizing difficulty of experiment More realistic models Easy to implement Can be integrated into N-Way Cross Validation Can apply to various types of SE data sets: – Defect Prediction – Effort Estimation Can be extended beyond SE to other domains
Discussion: Potential Problems More work needs to be done Agreement on how to measure Experimental Difficulty Extra overhead Implicitly or Explicitly Data Staved Domain
How to get more ML in SE? Conclusions Assess experiments/data for their difficulty Benefits: More credibility to the modeling process More reliable predictors More realistic models
Thanks to the reviewers for their comments! Acknowledgements
1) M. Jørgensen, A Review of Studies on Expert Estimation of Software Development Effort, Journal Systems and Software, Vol 70, Issues 1-2, 2004, Pp. 37-60. References