Presentation is loading. Please wait.

Presentation is loading. Please wait.

The Campbell Collaborationwww.campbellcollaboration.org Computing Random Effects Models in Meta-analysis Terri Pigott, C2 Methods Editor & co-Chair Professor,

Similar presentations


Presentation on theme: "The Campbell Collaborationwww.campbellcollaboration.org Computing Random Effects Models in Meta-analysis Terri Pigott, C2 Methods Editor & co-Chair Professor,"— Presentation transcript:

1 The Campbell Collaborationwww.campbellcollaboration.org Computing Random Effects Models in Meta-analysis Terri Pigott, C2 Methods Editor & co-Chair Professor, Loyola University Chicago

2 Campbell Collaboration Colloquium – May 2012www.campbellcollaboration.org As I am going through the beginning slides: Please download RevMan if you do not already have it: – Data is here: https://my.vanderbilt.edu/emilytannersmith/training-materials/

3 Campbell Collaboration Colloquium – May 2012www.campbellcollaboration.org Fixed and Random Effects Models in Meta-analysis How do we choose among fixed and random effects models when conducting a meta-analysis? Common question asked by reviewers working on systematic reviews that include a meta-analysis

4 Campbell Collaboration Colloquium – May 2012www.campbellcollaboration.org Our goal today Provide a brief description of fixed and of random effects models Discuss how to estimate key parameters in the random effects model Do sample computations to illustrate how to obtain the variance component when computing the mean effect size Illustrate how to compute the random effects model in RevMan Share resources for these methods for reviewers

5 Campbell Collaboration Colloquium – May 2012www.campbellcollaboration.org An advertisement Much of my own thinking on these issues has been clarified by the following sources: Borenstein, M., Hedges, L. V., Higgins, J. P. T., & Rothstein, H. R. (2010). A basic introduction to fixed effect and random effects models for meta-analysis. Research Synthesis Methods, 1, Borenstein, M., Hedges, L. V., Higgins, J. P. T., & Rothstein, H. R. (2009). Introduction to meta-analysis. West Sussex, UK: John Wiley.

6 Campbell Collaboration Colloquium – May 2012www.campbellcollaboration.org Our choice between the two models depends on: Our assumption about how the effect sizes vary in our meta- analysis The two models are based on different assumptions about the nature of the variation among effect sizes in our research synthesis

7 Campbell Collaboration Colloquium – May 2012www.campbellcollaboration.org Fixed-Effect model Borenstein et al. adopt the wording of fixed effect (no “s”) here because in a fixed effect model, we assume that the effect sizes in our meta-analysis differ only because of sampling error and they all share a common mean Our effect sizes differ from each other because each study used a different sample of participants – and that is the only reason for the differences among our estimates

8 Campbell Collaboration Colloquium – May 2012www.campbellcollaboration.org Picture from Borenstein et al. (2010)

9 Campbell Collaboration Colloquium – May 2012www.campbellcollaboration.org In a fixed-effect model Note that the effect size from each study estimate a single common mean – the fixed-effect We know that each study will give us a different effect size, but each effect size is an estimate of a common mean, designated in the prior picture as θ

10 Campbell Collaboration Colloquium – May 2012www.campbellcollaboration.org In a random effects model We assume two components of variation: – Sampling variation as in our fixed-effect model assumption – Random variation because the effect sizes themselves are sampled from a population of effect sizes In a random effects model, we know that our effect sizes will differ because they are sampled from an unknown distribution Our goal in the analysis will be to estimate the mean and variance of the underlying population of effect sizes

11 Campbell Collaboration Colloquium – May 2012www.campbellcollaboration.org Another picture from Borenstein et al. (2010)

12 Campbell Collaboration Colloquium – May 2012www.campbellcollaboration.org In a random effects model We see in the picture that each distribution has its own mean that is sampled from the underlying population distribution of effect sizes That underlying population distribution also has its own variance, τ 2, commonly called the variance component Thus, each effect size has two components of variation, one due to sampling error, and one from the underlying distribution

13 Campbell Collaboration Colloquium – May 2012www.campbellcollaboration.org Some notation for shorthand

14 Campbell Collaboration Colloquium – May 2012www.campbellcollaboration.org In fixed effects, we can write our model as So, each effect size estimates a single mean effect, θ, and differs from this mean effect by sampling error

15 Campbell Collaboration Colloquium – May 2012www.campbellcollaboration.org In random effects, we can write our model as Each effect size differs from the underlying population mean, μ, due to both sampling error and the underlying population variance

16 Campbell Collaboration Colloquium – May 2012www.campbellcollaboration.org Given the prior slides, how do we choose? The fixed-effect model assumes only sampling error as the source of variation among effect sizes This assumption is plausible when our studies are close replications of one another, using the same procedures, measures, etc.

17 Campbell Collaboration Colloquium – May 2012www.campbellcollaboration.org Choosing a model (continued) In the random effects model, we assume that our effect sizes are sampled from an underlying population of effect sizes We already assume that our studies will differ not only because there are different participants, but also because of differences in the way studies were conducted Thus, we often choose random effects models because we anticipate variation among our studies for a number of reasons

18 Campbell Collaboration Colloquium – May 2012www.campbellcollaboration.org Implications of the choice Recall that all our analyses in a meta-analysis are weighted by the inverse of the variance of the effect size, i.e., by the precision of the effect size estimate For a random effects model, the variance for each effect size is equal to v i + τ 2 In other words, in a fixed effect model, we will more heavily weight larger studies. In a random effects model, the larger studies will not be weighted as heavily

19 Campbell Collaboration Colloquium – May 2012www.campbellcollaboration.org Example of a fixed effect analysis from RevMan

20 Campbell Collaboration Colloquium – May 2012www.campbellcollaboration.org Same data, random effects analysis

21 Campbell Collaboration Colloquium – May 2012www.campbellcollaboration.org So, where are we now? We have discussed the underlying assumptions of fixed effect and random effects models We have also talked about how we choose between the two models Now we will talk about how to estimate the parameters of a random effects model

22 Campbell Collaboration Colloquium – May 2012www.campbellcollaboration.org Review of steps for fixed effect basic meta-analysis Compute the effect size and its variance for each study Compute the study weight – the inverse of the variance (1/var) Compute the weighted mean effect size and its standard error (inverse of the sum of the weights) Compute the 95% confidence interval for the weighted mean Compute the test of homogeneity

23 Campbell Collaboration Colloquium – May 2012www.campbellcollaboration.org Goals in a basic meta-analysis using a random effects model We want to estimate a mean effect size, μ, the mean effect size from the underlying population We also want to estimate the variance of the underlying effect size distribution, τ 2 Biggest difficulty: how to estimate τ 2

24 Campbell Collaboration Colloquium – May 2012www.campbellcollaboration.org Estimating the variance component, τ 2 There are two main methods for computing τ 2 – Variously called the method of moments, the DerSimonian/Laird estimate – Restricted maximum likelihood The method of moments estimator is easy to compute and is based on the value of Q, the homogeneity statistic Restricted maximum likelihood requires an iterative solution

25 Campbell Collaboration Colloquium – May 2012www.campbellcollaboration.org Method of moments estimator This is the estimator used in many meta-analysis programs such as RevMan and CMA

26 Campbell Collaboration Colloquium – May 2012www.campbellcollaboration.org An example So before we do anything in our basic random effects meta- analysis, we need to compute Q and the fixed effects weights, w i = 1/v i, to get the variance component, τ 2,. Once we have our τ 2, we can continue with our computation of the mean effect size and its variance. We will use an example of a meta-analysis of gender differences in transformational leadership

27 Campbell Collaboration Colloquium – May 2012www.campbellcollaboration.org Eagly, Johannesen-Schmidt & van Engen (2003) This synthesis examines the standardized mean difference estimated in primary studies for the difference between men and women in their use of transformational leadership. Transformational leadership involves “establishing onself as a role model by gaining the trust and confidence of followers” (Eagly et al. 2003, p. 570). The sample data is a subset of the studies in the full meta- analysis, a set of 24 studies that compare men and women in their use of transformational leadership

28 Campbell Collaboration Colloquium – May 2012www.campbellcollaboration.org Data Study nameEffect sizeSE eff sizeN menN women Bass Bono & Judge Boomer Carless Church & Waclawski Church & Waclawski Church et al Cuadrado Daughtry & Finch Davidson Gillespie & Mann Hill Jantzi & Leithwood Judge & Bono Komives Kuchinke Leithwood & Jantzi Manning McGrattan Rosen Rozier & Hersh-C Sosik & Megerian Spreitzer et al Wipf

29 Campbell Collaboration Colloquium – May 2012www.campbellcollaboration.org Back to our Method of moments estimator So we need c, and Q

30 Campbell Collaboration Colloquium – May 2012www.campbellcollaboration.org To get c: ∑ wt - ∑ wt 2 / ∑ wt = – ( )/ = To get Q: ( ) 2 / = To get τ 2 : (Q – (k-1))/c = [ – (24-1)]/ = 0.079

31 Campbell Collaboration Colloquium – May 2012www.campbellcollaboration.org Now that we have τ 2, we need to get the mean and its variance We first need to add τ 2 to each study’s variance to get the random effects variance Then we get the new weights to use for the computation of the mean effect size and its variance

32 Campbell Collaboration Colloquium – May 2012www.campbellcollaboration.org Data The random effects mean is found from (-35.76)/ = The se for the mean is found by 1/sqrt(223.00) = 0.067

33 Campbell Collaboration Colloquium – May 2012www.campbellcollaboration.org RevMan for random effects models We can also get the results in RevMan without having to do the computations. Open the file called “Gender differences for transformational leadership.rm5” – click on open a review from a file

34 Campbell Collaboration Colloquium – May 2012www.campbellcollaboration.org Click on data and analyses (I already put in data)

35 Campbell Collaboration Colloquium – May 2012www.campbellcollaboration.org Click on 1.3 Transformational leadership score

36 Campbell Collaboration Colloquium – May 2012www.campbellcollaboration.org Click on Forest plot

37 Campbell Collaboration Colloquium – May 2012www.campbellcollaboration.org These are the fixed effects CI’s for each study. This is the random effects mean and CI Here is our tau and test that tau is zero.

38 Campbell Collaboration Colloquium – May 2012www.campbellcollaboration.org Once we have the random effects mean and CI We will want to test whether the variance component is different from zero. Testing whether the variance component is different from zero is equivalent to asking whether the variation among studies is greater than what we would expect with only sampling error The test: the fixed effects test of homogeneity

39 Campbell Collaboration Colloquium – May 2012www.campbellcollaboration.org Restricted maximum likelihood estimator Many statisticians do not like the method of moments estimator though RevMan and Comprehensive Meta- analysis use this estimator Can estimate τ 2 using HLM, SAS Proc Mixed, or R I can give you sample programs for these estimators

40 Campbell Collaboration Colloquium – May 2012www.campbellcollaboration.org Reporting guidelines for random effects models These are the reporting guidelines according to me You should report the following when using a random effects model for estimating a mean effect size: – Rationale for choosing the random effects model – The method for computing the variance component, τ 2 (and it’s okay to say you used RevMan or CMA, for example) – The estimate of the variance component – The mean effect size and its variance or standard error – The confidence interval around the effect size

41 Campbell Collaboration Colloquium – May 2012www.campbellcollaboration.org Some concluding remarks Given the anticipated differences among studies in a typical social science systematic review, we usually choose random effects models From the C2 Methods Editor (me): make sure to provide a rationale for your choice in the protocol, based on your substantive knowledge of the area of the review Many other issues not touched on in this talk and see the references following this slide

42 Campbell Collaboration Colloquium – May 2012www.campbellcollaboration.org References Borenstein, M., Hedges, L. V., Higgins, J. P. T., & Rothstein, H. R. (2010). A basic introduction to fixed effect and random effects models for meta-analysis. Research Synthesis Methods, 1, Borenstein, M., Hedges, L. V., Higgins, J. P. T., & Rothstein, H. R. (2009). Introduction to meta-analysis. West Sussex, UK: John Wiley. Eagly, A. H., Johannesen-Schmidt, M. C. & van Engen, M. L. (2003). Transformational, transactional, and laissez-faire leadership styles: A meta-analysis comparing women and men. Psychological Bulletin, 129, Raudenbush, S. W. (2009). Analyzing effect sizes: Random- effects models. In Cooper, H., Hedges, L. V. & Valentine, J. C. (Eds.). The handbook of research synthesis and meta-analysis (2 nd ed). New York: Russell Sage Foundation.

43 The Campbell Collaborationwww.campbellcollaboration.org P.O. Box 7004 St. Olavs plass 0130 Oslo, Norway


Download ppt "The Campbell Collaborationwww.campbellcollaboration.org Computing Random Effects Models in Meta-analysis Terri Pigott, C2 Methods Editor & co-Chair Professor,"

Similar presentations


Ads by Google