RMF – Meta-analysis workshop (Marsh, O’Mara, Malmberg) 1 NCRM Research Methods Festival University of Oxford Prof. Herb MarshMs. Alison O’MaraDr. Lars-Erik Malmberg Dept of Education, University of Oxford

Traditionally, social science researchers collect and analyse their own data (referred to as primary data). Secondary data analysis is based on data collected by someone else (or, perhaps, re-analysis of your own published data). There are at least four logical perspectives to this issue: 1. Meta-analysis -- systematic, quantitative review of published research in a particular field, the focus of this presentation. 2. Systematic review -- systematic, qualitative review of published research in a particular field 3. Secondary Data Analyses -- using large (typically public) databases 4. Re-analyses of published studies -- often in ways critical of the original study. 2

 Systematic synthesis of various studies on a particular research question Do boys or girls have higher self-concepts?  Collect all studies relevant to a topic Find all published journal articles on the topic  An effect size (the ‘dependent variable’) is calculated for each outcome Determine the size/direction of gender difference for each study  “Content analysis” code characteristics of the study; age, setting, ethnicity, self- concept domain (math, physical, social), etc.  Effect sizes with similar features are grouped together and compared; tests moderator variables Do gender differences vary with age, setting, ethnicity, self-concept, domain, etc. 3

 Coding: the process of extracting the information from the literature included in the meta-analysis. Involves noting the characteristics of the studies in relation to a priori variables of interest (qualitative)  Effect size: the numerical outcome to be analysed in a meta-analysis; a summary statistic of the data in each study included in the meta-analysis (quantitative)  Summarise effect sizes: central tendency, variability, relations to study characteristics (quantitative) 4

 Compared to traditional literature reviews:  (1) there is a definite methodology employed in the research analysis; and  (2) the results of the included studies are quantified to a standard metric thus allowing for statistical techniques for further analysis.  Therefore less biased and more replicable 5

 Increased power: increases the chance of detecting a true treatment effect  Improved precision: with more information than a single study, the treatment effect estimate is improved  When study-to-study variation in results (which is typical) can evaluate differences in relation to study characteristics. Can delve into research questions not explored by the individual studies  Easy to interpret summary statistics (useful if communicating findings to a non-academic audience) 6

 The essence of good science is replicable and generalisable results.  Do we get the same answer to important research questions when we run the study again?  The primary aims of meta-analysis is to test the generalisability of results across a set of studies designed to answer the same research question.  Are the results consistent? If not, what are the differences in the studies that explain the lack of consistency? 7

 Meta-analysis is an increasingly popular tool for summarising research findings; literature review method of choice in many disciplines  Widely-cited. If there is a good meta-analysis relevant to your study, you have to cite it  Relied upon by policymakers  Important that we understand the method, whether we conduct or consume meta-analytic research  Should be one of the topics covered in all introductory research methodology courses 8

 There exists a critical mass of comparable studies designed to address a common research question.  Data are presented in a form that allows the meta- analyst to compute an effect size for each study.  Characteristics of each study are described in sufficient detail to allow meta-analysts to compare characteristics of different studies and to judge the quality of each study. 9

10 The number of meta- analyses is increasing at a rapid rate.

11

12 Psychology: Citations Psychology: Articles

 Amato, P. R., & Keith, B. (1991). Parental divorce and the well-being of children: A meta-analysis. Psychological Bulletin, 110, Times Cited: 471  Linn, M. C., & Petersen, A. C. (1985). Emergence and characterization of sex differences in spatial ability: A meta-analysis. Child Development, 56, Times Cited: 570  Johnson, D. W., & et al (1981). Effects of cooperative, competitive, and individualistic goal structures on achievement: A meta-analysis. Psychological Bulletin, 89, Times Cited: 426  Tett, R. P., Jackson, D. N., & Rothstein, M. (1991). Personality measures as predictors of job performance: A meta-analytic review. Personnel Psychology, 44, Times Cited: 387  Hyde, J. S., & Linn, M. C. (1988). Gender differences in verbal ability: A meta-analysis. Psychological Bulletin, 104, Times Cited: 316  Iaffaldano, M. T., & Muchinsky, P. M. (1985). Job satisfaction and job performance: A meta-analysis. Psychological Bulletin, 97, Times Cited:

 De Wolff, M., & van IJzendoorn, M. H. (1997). Sensitivity and attachment: A meta-analysis on parental antecedents of infant attachment. Child Development, 68, Times Cited: 340  Wellman, H. M., Cross, D., & Watson, J. (2001). Meta-analysis of theory- of-mind development: The truth about false belief. Child Development, 72, Times Cited: 276  Cohen, E. G. (1994). Restructuring the classroom: Conditions for productive small groups. Review of Educational Research, 64, Times Cited: 235  Hansen, W. B. (1992). School-based substance abuse prevention: A review of the state of the art in curriculum, Health Education Research, 7, Times Cited: 207  Kulik, J. A., Kulik, C-L., Cohen, P. A. (1980). Effectiveness of Computer- Based College Teaching: A Meta-Analysis of Findings. Review of Educational Research, 50, Times Cited:

 Sheppard, B. H., Hartwick, J., & Warshaw, P. R. (1988). The theory of reasoned action: A meta-analysis of past research with recommendations for modifications and future research. Journal of Consumer Research, 15, Times Cited: 515  Jackson, S. E., & Schuler, R. S. (1985). A meta-analysis and conceptual critique of research on role ambiguity and role conflict in work settings. Organizational Behavior and Human Decision Processes, 36, Times Cited: 401  Tornatzky Lg, Klein Kj. (1994). Innovation characteristics and innovation adoption-implementation - A meta-analysis of findings. IEEE Transactions On Engineering Management, 29, Times Cited: 269.  Lowe KB, Kroeck KG, Sivasubramaniam N. (1996). Effectiveness correlates of transformational and transactional leadership: A meta- analytic review of the MLQ literature. Leadership Quarterly, 7, Times Cited: 203.  Churchill GA, Ford NM, Hartley SW, et al. (1985). Title: The determinants of salesperson performance - A meta-analysis. Journal Of Marketing Research, 22, Times Cited:

 Jadad AR, Moore RA, Carroll D, et al. (1996). Assessing the quality of reports of randomized clinical trials: Is blinding necessary? Controlled Clinical Trials, 17, Times Cited:2008  Boushey Cj, Beresford Saa, Omenn Gs, Et. Al. (1995). A quantitative assessment of plasma homocysteine as a risk factor for vascular-disease - Probable benefits of increasing folic-acid intakes. JAMA-journal Of The American Medical Assoc, 274, Times Cited: 2,128  Alberti W, Anderson G, Bartolucci A, et al. (1995). Chemotherapy in non-small-cell lung-cancer - A metaanalysis using updated data on individual patients from 52 randomized clinical-trials. British Medical Journal, 311, Times Cited:1,591  Block G, Patterson B, Subar A (1992). Fruit, vegetables, and cancer prevention - A review of the epidemiologic evidence. Nutrition And Cancer-an International Journal, 18, Times Cited: 1,422 16

Gene Glass coined the phrase meta-analysis in classic study of the effects of psychotherapy. Because most individual studies had small sample sizes, the effects typically were not statistically significant.  Results of 375 controlled evaluations of psychotherapy and counselling were coded and integrated statistically. The findings provide convincing evidence of the efficacy of psychotherapy.  On the average, the typical therapy client is better off than 75% of untreated individuals.  Few important differences in effectiveness could be established among many quite different types of psychotherapy (e.g., behavioral and non-behavioral). 17 ESRC RDI One Day Meta-analysis workshop (Marsh, O’Mara, Malmberg) 17

Establish research question Define relevant studies Develop code materials Locate and collate studies Pilot coding; coding Data entry and effect size calculation Main analyses Supplementary analyses 18

 Need to have explicit inclusion and exclusion criteria  The broader the research domain, the more detailed they tend to become  Refine criteria as you interact with the literature  Components of a detailed criteria  distinguishing features  research respondents  key variables  research methods  cultural and linguistic range  time frame  publication types 19

 Search electronic databases (e.g., ISI, Psychological Abstracts, Expanded Academic ASAP, Social Sciences Index, PsycINFO, and ERIC)  Examine the reference lists of included studies to find other relevant studies  If including unpublished data, researchers in your discipline, take advantage of Listservs, and search Dissertation Abstracts International 20

 Random selection of papers coded by both coders  Meet to compare code sheets  Where there is discrepancy, discuss to reach agreement  Amend code materials/definitions in code book if necessary  May need to do several rounds of piloting, each time using different papers 21

__Study ID _ _Year of publication __Publication type (1-5) __Geographical region (1-7) _ _ _ _Total sample size _ _ _Total number of males _ _ _Total number of females 22 Publication type (1-5) 1.Journal article 2.Book/book chapter 3.Thesis or doctoral dissertation 4.Technical report 5.Conference paper Code Sheet Code Book/manual ESRC RDI One Day Meta-analysis workshop (Marsh, O’Mara, Malmberg)

 The effect size makes meta-analysis possible  It is the “dependent variable”  It standardizes findings across studies such that they can be directly compared  Any standardized index can be an “effect size” (e.g., standardized mean difference, correlation coefficient, odds- ratio), but must  be comparable across studies (generally requires standardization)  represent the magnitude and direction of the relationship of interest  be independent of sample size 23

 Represents a standardized group contrast on an inherently continuous measure  Uses the pooled standard deviation (some situations use control group standard deviation)  Commonly called “d” In a gender difference study, the effect size might be: In an intervention study with experimental and control groups, the effect size might be: 24

Means and standard deviations Correlations P-values F -statistics d t -statistics “other” test statistics Almost all test statistics can be transformed into an standardized effect size “d” ESRC RDI One Day Meta-analysis workshop (Marsh, O’Mara, Malmberg) 25 Lipsey & Wilson (2001) present formulae for calculating effect sizes from different information

26 Each study is one line in the data base Effect sizeDurationSample sizes Reliability of the instrument Variance of the effect size

 There are various ways of analysing meta-analytic data  Three main methods based on different statistical assumptions:  Fixed effects models  Random effects models  Multilevel models  These will be discussed in the afternoon workshop 27

 Meta-analysis is a method for synthesising and analysing the research literature on a particular topic  The essence of good science is replicable and generalisable results.  Increasingly sophisticated  Continuously evolving  For more information about the meta-analysis training courses that we offer, please see 28