1 Meta-analysis issues Carolyn Mair and Martin Shepperd Brunel University, UK

2 Outline What is meta-analysis? What is systematic review? Is there a difference? Why the need for meta-analysis MeLLow

3 What is the difference? Exploit your expertise and knowledge to reach consensus Record responses Discuss further discussed in reverse panel Draft paper disseminated to all for amendment as necessary Details of common understanding of definitions and processes involved will be submitted for publication.

4 Over to you… What is a meta-analysis? What is a systematic review? What stages are involved? Is there a difference? Summarise your responses please

5 What is meta-analysis? Combines results of several studies that address a set of related research hypotheses Attempt to overcome problem of reduced statistical power in studies with small sample sizes (Pearson, 1904) Educational research meta-analysis (1970s) e.g. Glass, Schmidt, and Hunter “Meta-analysis is a statistical procedure for synthesising quantitative results from different studies” (Kitchenham, 2004)

6 What is systematic review? “Summary of literature that uses explicit methods to perform thorough literature search and critical appraisal of individual studies to identify valid and applicable evidence, and then uses appropriate techniques to combine them” (Wikipaedia, 2006) A prerequisite for quantitative meta-analysis (Kitchenham, 2004) Majority based on explicit quantitative meta-analysis of available data, but some are qualitative Cochrane Collaboration Database of Systematic Reviews –2,785 complete reviews; 1,625 protocols (September, 2006)

7 Another definition 5 major sections of systematic review –Introduction –Searching –Critical Appraisal –Meta-analysis –Discussion Systematic Review Study Group of Students’ Scientific Research Centre (Tehran University of Medical Sciences)

8 7 steps for preparing and maintaining systematic review Formulating problem Locating and selecting studies Critical appraisal of studies Collecting data Analyzing and presenting results Interpreting results Improving and updating reviews

9 Why do a review / meta-analysis? Summarise existing evidence Identify gaps in current research Provide framework/background to position new research activities …what differentiates a literature review from a systematic review? …transparency, fairness and completeness

10 Meta-analysis improves understanding e.g. Basili et al.,1999; Hayes, 1999; Kitchenham et al., 2002, Miller, 1999; Pickard, 1998 Large literature of empirical studies in software engineering cost estimation Have evidence …but it’s not consistent Need to build ‘body of knowledge’ …but heterogeneity of studies, format and provision of data cause problems

11 …but Data from individual studies need to be made available in standardised format that enables cross comparison Need individual data points from all studies

12 What guidelines are exist? 1.The Cochrane Reviewer’s Handbook 2.Guidelines prepared by the Australian National Health and Medical Research Council 3.ANYTHING ELSE???????????

13 The MeLLow project Aim –To develop tool to enable practitioners to select, a priori, the optimal prediction technique for particular circumstances Objectives –To identify empirical studies of software engineering prediction techniques –Categorise salient project features that impinge on effort –Identify biases and effects within the studies Problems –Application of range of research methodologies Outcome –Efforts turned to promoting more rigorous, structured and uniform approach to research in software engineering prediction techniques

14 Heterogeneity Accuracy measures –20 studies (1-16 datasets, 1-6 accuracy measures) (Mair and Shepperd, 2004) Response variables Range of reporting techniques and content –Raw data, cleaned data, summary data Different imputation methods –ignored or imputed (various imputation methods) Different holdout strategies training set (used to develop prediction system) + validation set (data points to assess prediction system accuracy). e.g. 15%, 22% or 0%

15 …and Pet techniques (Spiegelhalter, 1995) –e.g. CBR, GA, RI Publication bias and the ‘file drawer problem’ (Rosenthal, 1979)...differences also in understanding what a meta-analysis is!

16 Advantages of a systematic approach Provide information about effects across range of settings and methods Examine evidential support or contradiction of theoretical hypotheses  generate new hypotheses Consistent study results = robust, transferable phenomenon Inconsistent study results can be investigated Furthermore, meta-analysis can try to find effects and biases not evident in individual smaller studies

17 Terminology and process What? How? Are there common standards?