1. THE IRISH SOFTWARE ENGINEERING RESEARCH CENTRELERO© 2006 1 What we currently know about software fault prediction: A systematic review of the fault prediction literature Presentation by Sarah Beecham 27 January 2010 – 3 rd CREST Open Workshop – King’s College London

2. THE IRISH SOFTWARE ENGINEERING RESEARCH CENTRELERO© 2006 2 Talk will cover 1. Introduction – why are we interested in fault prediction? 2. Our research questions 3. Methodology – how we derived our 111 studies 4. The results – the context (e.g. type of system, programming lang.) emerging themes (qualitative analysis) 5. Quality assessment & Conclusions 6. Future work – unresolved questions

3. THE IRISH SOFTWARE ENGINEERING RESEARCH CENTRELERO© 2006 3 Why do we need to know about Fault Prediction? Fixing faults major cost driver in software development (Tomaszewski et al. 2006). Preventing & removing faults costs $50 - $78 billion per year in US alone (Levinson 2001; Runeson and Andrews 2003). Software testing and debugging phases take up most of this spend (Di Fatta et al. 2006). Identifying faults early in lifecycle with high degree of accuracy results in.. –higher quality software –better use of resources ( Koru and Liu 2005; Bezerra et al. 2007; Oral and Bener 2007)

4. THE IRISH SOFTWARE ENGINEERING RESEARCH CENTRELERO© 2006 4 Our research questions RQ1: What is the context of the fault prediction model? RQ2: What variables have been used in fault prediction models? 2.1: predictor or independent variables, e.g. size; McCabe 2.2: dependent variables –faults RQ3:What modelling approaches have been used in the development of fault prediction models? RQ4: How do studies measure performance of their models? RQ5: How well do fault prediction models predict faults in code?

5. THE IRISH SOFTWARE ENGINEERING RESEARCH CENTRELERO© 2006 5 Methodology (1)– Search Terms and Resources; selection criteria Search terms and string (Fault* OR bug* OR corrections OR corrective OR fix* OR defect*) in title AND (Software) anywhere Sources/ databases: ACM Digital library; IEEEXplore key conferences; e.g. (ICSM); (SCAM); (ISSTA) ; key authors; +.. Include study if: Empirical; Focused on fault prediction; predictor variable (input)linked to code as output; faults in code main output (dependent variable) Exclude study if: testing, fault injection, inspections, reliability modelling, aspects, effort estimation, nano-computing.. List added to when conducting the review

6. THE IRISH SOFTWARE ENGINEERING RESEARCH CENTRELERO© 2006 6 Methodology (2) Study Acceptance – 111 papers Selection Process# of papers Papers extracted from databases, conferences & author names 1,316 Sift based on Title and Abstract-1154 rejected Papers considered for review (full papers downloaded & reviewed) 162 primary 80 secondary Papers accepted for the review (qualitative meta-analysis performed) 111 papers accepted papers [TH1] Need to comment on imbalance here[TH1] Each stage independently validated – 3 researchers involved Meta-analysis/themes underpinned by previous SLRs

7. THE IRISH SOFTWARE ENGINEERING RESEARCH CENTRE Approaches to Fault prediction components: LERO© 2006 7 CONTEXT E.g. Type of System (NASA project, Telecomms,..); Programming Language (C++, Java, …); Development process (Waterfall, Agile,..) Programmer expertise. Input Variable (Independent Variable) e.g. complexity metrics, module size, even faults themselves Modelling method e.g. regression, machine learning, correlation, BBN Output Variable (Dependent Variable) Fault prone (fp) /not fault prone (nfp) unit OR Number of faults; fault density, ranking model predicts where fault will occur in the code with a % of accuracy.. – however this alone has little meaning – we need to know when the model predicts a fault that IS NOT there, and when a model doesn’t pick up a fault that IS there.

8. THE IRISH SOFTWARE ENGINEERING RESEARCH CENTRE Results … context first (RQ1) LERO© 2006 8 Origin of data, a large proportion from NASA promise repository

9. THE IRISH SOFTWARE ENGINEERING RESEARCH CENTRE C/C++, 54% Java, 26% Various, 4% Other, 7% Not given,9% Key: ‘Other’ languages include Assembly, Fortran and Protel Context (2)–Programming Language LERO© 2006 9

10. THE IRISH SOFTWARE ENGINEERING RESEARCH CENTRE Independent or ‘predictor’ Variables (RQ2.1) LERO© 2006 10 Miscellaneous, 16% Static Code Metrics, 43% Dynamic Code Metrics, 2% Change Data, 20% Previous Fault Information, 19% Key: Miscellaneous category includes code patterns, testing and inspection metrics, qualitative data, and function points...

11. THE IRISH SOFTWARE ENGINEERING RESEARCH CENTRE Code Granularity LERO© 2006 11

12. THE IRISH SOFTWARE ENGINEERING RESEARCH CENTRE Dependent Variables – Faults (RQ2.2) LERO© 2006 12

13. THE IRISH SOFTWARE ENGINEERING RESEARCH CENTRE Modelling approaches used to develop fault prediction models (RQ3) LERO© 2006 13

14. THE IRISH SOFTWARE ENGINEERING RESEARCH CENTRE Positive Performances (RQ5) LERO© 2006 14 Positive results reported in 86 out of 107 studies

15. THE IRISH SOFTWARE ENGINEERING RESEARCH CENTRELERO© 2006 15 Quality Assessment Does the study report whether results are significant? Does study recognize/address imbalanced dataset problem? How has the performance of the model been assessed? How has the model been validated?

16. THE IRISH SOFTWARE ENGINEERING RESEARCH CENTRELERO© 2006 16 Significance Test results.. 111 studies – 47% conducted significance tests some selected metrics based on their significance 53% of studies did not report such a test. No certainty that model outputs are statistically significant in over half the papers in this study

17. THE IRISH SOFTWARE ENGINEERING RESEARCH CENTRE Few studies balance data LERO© 2006 17

18. THE IRISH SOFTWARE ENGINEERING RESEARCH CENTRELERO© 2006 18 Performance measurements MeasureConstructs and Definitions Confusion Matrix constructs True Positive (TP), True Negative (TN), False Positive (FP), False Negative (FN), Type I Error, Type II Error Confusion Matrix composite measures Recall, Precision, Specificity, Accuracy, Sensitivity, Balance, F-measure Based on concept of code being faulty or not faulty Other measures: Descriptive statistics (e.g. frequencies, means, ratios); Regression coefficients, significance tests

19. THE IRISH SOFTWARE ENGINEERING RESEARCH CENTRE Performance indicator frequency LERO© 2006 19

20. THE IRISH SOFTWARE ENGINEERING RESEARCH CENTRE Generalisability of the models – Validation LERO© 2006 20

21. THE IRISH SOFTWARE ENGINEERING RESEARCH CENTRE Features identified as helpful to fault prediction… LERO© 2006 21

22. THE IRISH SOFTWARE ENGINEERING RESEARCH CENTRE Final Remarks… To allow cross comparison of models we need: A standard format to report model performance To report false positives and false negatives Understand that reporting one measure can be misleading –trade-off between accuracy, precision and recall (composite measures) Access to reliable quality data (next talk..) LERO© 2006 22

23. THE IRISH SOFTWARE ENGINEERING RESEARCH CENTRELERO© 2006 23 thank you

24. THE IRISH SOFTWARE ENGINEERING RESEARCH CENTRELERO© 2006 24 REFERENCES