1. Practical Meta-Analysis for the Social Sciences Evan J. Ringquist School of Public and Environmental Affairs Indiana University Workshop in Methods Presentation January 11, 2013 Bloomington, IN 47405 1

2. 2 “Meta-Analysis is not a fad. It is rooted in the fundamental values of the scientific enterprise: replicability, quantification, causal and correlational analysis. Valuable information is needlessly scattered in individual studies. The ability of social scientists to deliver generalizable answers to basic questions of policy is too serious a concern to allow us to treat research integration lightly. The potential benefits of meta-analysis method seem enormous.” (Bangert-Drowns 1986: 398)

3. Meta-Analysis Defined Meta-analysis is a systematic, quantitative, replicable process of synthesizing numerous and sometimes conflicting results from a body of original studies. Meta-analysis, then, provides a powerful set of tools for aggregating knowledge. 3

4. Number of Articles Published Each Year in the “Meta-Analysis” Topic Category as Referenced by the Social Sciences Citation Index 4

5. Outline of Presentation 1.Motivation for Using Meta-Analysis in the Social Sciences 2.Quick Introduction to Meta-Analysis 3.Meta-Regression Models: The Basics 4.Meta-Regression Analysis for the Social Sciences –CRVE, GEE, and if we have time, HLM 5

6. Section I: Motivation 6

7. Cumulative Knowledge and the Scientific Enterprise A Central Goal of Science is the Accumulation and Aggregation of Knowledge. Yet the Social Sciences in General and Researchers in Public Management and Policy in Particular Have Found Cumulative Knowledge to be an Elusive Goal. 7

8. “Cumulative Knowledge” in Environmental Justice “Overwhelming evidence” of significant environmental inequities (Goldman 1993; see also Mohai and Bryant 1994) “even a reasonably generous reading of the empirical research alleging environment inequity... must leave room for profound skepticism” (Foreman 1998; see also Bowen 2001) 8

9. “Cumulative Knowledge” in Education Policy “Extensive research has been conducted on the academic success of students enrolled in school choice program nationwide. Rigorous studies show strong gains for voucher and scholarship tax credit recipients...” (Alliance for School Choice, 2010) “The reason that vouchers had subsided as a point of advocacy is because they don’t work.” (Randi Weingarten, President, American Federation of Teachers; Associated Press 2011). “Few contemporary questions in American education have produced such wide-spread controversy as that regarding the potential of school vouchers to reduce inequality in student outcomes... these efforts notwithstanding, answers to the voucher question still appear uncertain.” (Cowen 2008) 9

10. Cumulative Knowledge in Public Management the more knowledge we create and diffuse, “the less we know, because our research is not aggregated or accumulated into substantial bodies of knowledge” (Van Slyke, O’Leary, and Kim 2010: 290. The first step in making Public Administration a stronger and more robust field is the need to aggregate knowledge in the sense of making it cumulative. (Rosenbloom 2010) 10

11. Cumulative Knowledge in Policy Studies “political scientists have spent (almost literally) countless articles and books proposing something like ‘laws’ or theories that, taken collectively... have produced infinitely more confusion than clarity” (DeLeon 1998: 150). 11

12. Cumulative Knowledge in Policy Evaluation Social scientists identify new and important questions having policy relevance. Initial studies examining these questions provide clear answers regarding, for example, the effectiveness of social policy interventions. Subsequent studies cast doubt on these initial conclusions, however, and with the proliferation of studies comes a proliferation of conclusions. The best efforts of social and behavioral researchers generate confusion and uncertainty rather than clarity. In the end, researchers conclude that the phenomena being studied are ‘hopelessly complex’ and move on to other questions. After several repetitions of this cycle, social and behavioral scientists themselves become cynical about their own work and express doubts about whether behavioral and social science in general is capable of generating cumulative knowledge or answers to socially important questions (Hunter and Schmidt 1996: 325-6). 12

13. Why Do Social Scientists Struggle with Knowledge Accumulation? 1.Epistemological Differences: Social Sciences ought not aim to generate generalizable “scientific” knowledge ala Chemistry. 2.Subject Matter Differences: Social scientists study organizations and institutions devised and populated by strategic actors, not molecules or organizms without agency. 3.Social Scientists Use The Wrong Tools for Aggregating Knowledge. 13

14. Section II: Introduction to Meta-Analysis 14

15. Meta-Analysis Defined (again) Meta-analysis is a systematic, quantitative, replicable process of synthesizing numerous and sometimes conflicting results from a body of original studies. Meta-analysis, then, provides a powerful set of tools for aggregating knowledge. 15

16. The Language of Meta-Analysis Original Study: A piece of original research, published or unpublished, that aims to test a hypothesis and/or estimate a quantity of interest. Focal Predictor: The independent variable in an original study that measures the key exogenous concept associated with the research question of interest. 16

17. The Language of Meta-Analysis Effect Size: A standardized measure of the relationship between the focal predictor and the dependent variable in an original study. Effect sizes are the unit of analysis and the quantity of interest in a meta- analysis, often designated Θ i. Without effect sizes there can be no meta-analysis. 17

18. The Language of Meta-Analysis Effect Size Variance: A measure of the uncertainty associated with a particular effect size. Fixed and Random Effects Models: Methods of weighting effect sizes that embody different assumptions about the effect size variance. 18

19. How Do We Calculate Effect Sizes? Three Families of Effect Sizes 1.D-based effect sizes (standardized mean differences, common in education and psychology) 2.Odds-based effect sizes (e.g. log odds of an event, common in medicine) 3.R-based effect sizes (partial correlation coefficients, unusual in the literature but most useful for social scientists) 19

20. Calculating R-based Effect Sizes from Original Studies r = √[t 2 / (t 2 + df)][1] r = √[Z 2 / n][2] r = √[Χ 2 1 / n][3] V[r] = (1-r 2 ) 2 / (n-1)[4] 20

21. Fisher’s Corrections for R-based Effect Sizes and Variances Z r = 0.5 ln[(1+r) / (1-r)][5] V[Z r ] = 1/(n-3)[6] 21

22. Example: Are Pollution Emissions Higher in Black Neighborhoods? Ringquist 1997: t=3.06, df=29202 –r=.02 –Zr=.02 –V[Zr] =.000034 Downey 1998: t=3.38, df=112 –r=.30 –Zr=.31 –V[Zr] =.0072 22

23. What Can We Do With Effect Sizes Statistically? 1.Communicate results from original research in a more meaningful fashion (not meta-analysis). 2.Combine (average) effect sizes to estimate the population effect size 3.Test null hypothesis that population effect size equals zero 4.Explain variation in effect sizes across original studies. We use “Meta-Regression” to account for this variation. 23

24. Combining Effect Sizes We calculate average effect sizes, or our estimate of the population effect size, by calculating a weighted average of all effect sizes from original studies where the weights are inverse variances Θbar= Σw i Θ i / Σw i [7] W i = 1/v i 24

25. Combining Effect Sizes: Fixed Effects Models Fixed Effects models assume that effect sizes vary across original studies only due to sampling error: –Θ i = Θ + e i [8], so –E[Θ i ]= Θ [9], and –e i ~ N(0,v i )[10] 25

26. Combining Effect Sizes: Fixed Effects Models Fixed effects meta-regression assumes that effect sizes conditional upon moderator variables differ only due to sample size Fixed effects estimates apply only to sample of original studies in hand 26

27. Combining Effect Sizes: Random Effects Models Random Effects models assume that effect sizes are normally distributed random variables –Θ ~ N(µ Θ, τ 2 ) [11] –Θ i = µ Θ + e i [12] –E[Θ i ] = µ Θ [13], and –e i ~ N(0,v i + τ 2 )[14] –Always use random effects in the social sciences! 27

28. Estimating the Random Effects Variance Component Using Restricted Maximum Likelihood –L e (τ 2 ) = -.5 * Σ [ln(v i + τ 2 ) + ((e i 2 / (v i + τ 2 ) -.5 * ln |X’v -1 X| [15] Using Method of Moments (MOM) –tr(M) = Σv i -1 – tr [(Σv i -1 XX’) -1 (Σv i -2 XX’)][16] Using MOM Approximation –τ 2 = [SSE ols / (m-k-1)] – v bar [17] 28

29. What Can We Do With Effect Sizes Substantively? Estimate severity of problems –E.g., environmental inequities –E.g., effects of climate change Measure important quantities of interest –E.g., statistical value of a life –E.g., hedonic pricing of environmental amenities 29

30. What Can We Do With Effect Sizes Substantively? Program Evaluation –E.g., effectiveness of educational vouchers –E.g., effectiveness of job training programs Theory Testing and Development –E.g., Top-down vs. bottom-up implementation –E.g., effects of negative campaign ads –E.g., Ricardian equivalence 30

31. Section III: Introduction to Meta-Regression 31

32. “exploration of the pattern of variation in effect sizes among studies is a far more important goal of meta-analysis than the construction of powerful tests of null hypotheses.” (Osenberg et al. 1999: 1105) 32

33. Moving Meta-Analysis to the Social Sciences Techniques for calculating and combining effect sizes developed for synthesizing the results from experiments. Can these same techniques really be used to synthesize results from the non-experimental multivariate models common in the social sciences? 33

34. Three Critiques of Meta- Analysis in the Social Sciences 1.Parameter estimates from the general linear model are not comparable. 2.Effect sizes cannot be calculated from multivariate models. 3.Parameter estimates from different models estimate different population parameters, and therefore cannot be combined. 34

35. Response to Critique #1: While regression parameters are not comparable, meta-analysis does not combine parameter estimates. Rather, meta-analysis combines effect sizes which are standardized with respect to scale. This is not a valid critique. 35

36. Response to Critique #2 In fact, the same formulas that can be used to calculate r-based effect sizes from experimental studies can be used to calculate partial correlations from regression models, probit models, etc. (see Greene 1993: 180) 36

37. Response to Critique #2 Example 1: Panel regression model, Y=continuous, K=33, N=307,538 –Pcorr =.0099, Zr =.0096 Example 2: Probit model, Y=dichotomous, K=8, N=14131 –Pcorr =.1081, Zr =.1062 Example 3: OLS regression model, Y=continuous, K=24, N=285 –Pcorr =.1619, Zr =.1565 37

38. Response to Critique #3 E.g., Study 1 –Y = b 0 + b 1 X 1 + b 2 X 2 + b 3 X 3 + e E.g., Study 2 –Y = b 0 + b 1 X 1 + b 2 X 2 + b 4 X 4 + e E[b 11 ] ≠ E[b 12 ], β 11 ≠ β 12, and Θ 1 ≠ Θ 2 Therefore dependent variable Θ i measures fundamentally different quantities 38

39. Response to Critique #3 Critique #3 is a valid critique, and this is why calculating average effect sizes is of little value in the social sciences – these average effect sizes combine estimates of different population parameters, and therefore are not valid estimates of any useful quantity of interest. Thankfully, we can address Critique #3 using Meta-Regression. 39

40. What Do We Need to Conduct a Meta-Regression? Measure of effect size that is comparable across original studies employing different measures, models, and samples Measure of (un)certainty or effect size variance Moderator variables that account for variability in effect sizes 40

41. Introduction to Meta-Regression Θ i = b 0 + b 1 X 1i + b 2 X 2i + b 3 X 3 + e i : e i ~ N (0, v i ) –Θ i is the effect size –X represents moderator variables accounting for differences in effect sizes within and across studies Scientifically Interesting Moderators (Rubin 1992) –e.g., differences attributable to target characteristics –e.g., differences attributable to program or policy design Scientifically Uninteresting Moderators –e.g., differences attributable to estimation technique or research design 41

42. Number of Publications Using Phrase “Meta- Regression” in Google Scholar, 1990-2011 42

43. Example: Synthesizing Research on Educational Vouchers Θ i = b 0 + b 1 X 1i + b 2 X 2i + b 3 X 3 + e i : e i ~ N (0, v i ) –Θ i is the estimated point biserial correlation (effect size Zr) between the use of an educational voucher and student standardized test scores from a particular statistical model in a particular original study. 43

44. Example: Educational Vouchers Θ i = b 0 + b 1 X 1i + b 2 X 2i + b 3 X 3 + e i : e i ~ N (0, v i ) –X 1 moderator variable identifying effect sizes from models limiting sample to black children –X 2 moderator variable identifying effect sizes from voucher programs available to religious schools –X 3 moderator variable identifying effect sizes from models not using random assignment to treatment and control groups. 44

45. Example: Educational Vouchers Θ i = b 0 + b 1 X 1i + b 2 X 2i + b 3 X 3 + e i : e i ~ N (0, v i ) –b 0 average effect of vouchers on student test scores for all students in experimental studies where vouchers use is limited to secular schools –b 1 differential effect of vouchers for black students –b 2 differential effect from voucher programs that include religious schools –b 3 differential voucher effect from quasi-experimental studies 45

46. 46 Independent Variables CRVE Parameter Estimates (St.Err.) CRVE 95% CI GEE Parameter Estimates (St.Err.) GEE 95% CI Black Student0.0296**0.00300.05620.0370***0.01430.0598 (0.0130)(0.0116) Hispanic Student-0.0470***-0.0680-0.0259-0.0413***-0.0577-0.0248 (0.0103)(0.0084) Religious Schools-0.0312-0.06930.00690.0180-0.05740.0213 (0.0187)(0.0201) Matched Control Groups-0.0097-0.02260.0033-0.0069*-0.01480.0009 (0.0064)(0.0040) Convenience Control Groups-0.0073-0.02460.0101-0.0082-0.02430.0078 (0.0085)(0.0082) Pretest Included-0.0326***-0.0560-0.0092-0.0228*-0.04640.0008 (0.0115)(0.0120) Intercept (Baseline Effect Size)0.1097***0.05730.16200.0830*** 0.02430.1416 (0.0257)(0.0299) Adjusted R 2a Psuedo-R 2a I 2a Sample Size.44.68 25.01% 611 na 611

47. Obtaining Estimates for Meta-Regression Models OLS Estimator of Meta-Regression Model –E[b] = β = (X’X) -1 X’Θ[18] –var[b] = σ 2 (X’X) -1 [19] But OLS is incorrect for two reasons: 1.Θ i is heteroskedastic (recall assumptions) 2.OLS weights all observations equally (unequal variances means unequal certainty) 47

48. Generalized Least Squares in Meta-Regression In OLS, ee’ = σ 2 I In GLS, ee’ = σ 2 Ω –where Omega is a diagonal matrix with proportional error variances on main diagonal GLS estimates then are: –E[b gls ] = β gls = (X’Ω -1 X) -1 X’Ω -1 Θ [20] –var[b gls ] = σ 2 (X’Ω -1 X) -1 [21] 48

49. Weighted Least Squares in Meta-Regression Find a weight matrix W so that W’W = Ω -1 Then the WLS meta-regression estimates become: –E[b wls ] = β wls = (X’W’WX) -1 X’W’WΘ[22] –var[b wls ] = σ 2 (X’W’WX) -1 [23] 49

50. Weighted Least Squares in Meta-Regression (cont.) The good news is that we have a handy estimate of the diagonal elements of the weight matrix W: the effect size variances (or more properly, their square roots) The bad news is that fixed effects variance estimates are almost certainly wrong, so we need to use random effect variances. 50

51. RE Variance Component τ 2 Cannot include moderator variables for all independent variables or all design elements in original studies Random Effects variance component τ 2 can be conceived of as reflecting the impact of unobserved variables on Θ i 51

52. Weights in Random Effects Meta-Regression (v 1 + τ 2 )000 0(v 2 + τ 2 )00 ee’ remr = σ 2.... 000(v n + τ 2 ) With weights as the square roots of the diagonal elements of this matrix. 52

53. Estimating the Random Effects Meta-Regression Model in Practice Option 1: Use the WLS function in Stata, SAS, or some other statistical package. –Don’t do this. All WLS algorithms treat weights as proportional variance weights rather than as absolute variance weights. This means that meta-regression parameter standard errors will be incorrect. 53

54. Estimating the Random Effects Meta-Regression Model in Practice Option 2: Use the VWLS command in Stata. –This will work as long as you import the proper random effects variance weights (actually, their standard deviations). These must be calculated outside of the VWLS model. This works because VWLS treats the imported variances as known and absolute rather than as estimated and proportional. 54

55. Estimating the Random Effects Meta-Regression Model in Practice Option 3: Use the “Metareg” Command in Stata –While Stata does not include commands for meta-analysis, the Stata User Group (SUG) has produced a number of good meta- analysis ADO files, including a random effects meta-regression algorithm. These commands can be downloaded from the Stata website. 55

56. Estimating the Random Effects Meta-Regression Model in Practice Option 4: Use “Manually Weighted” Least Squares –Estimate the REMR model by first manually weighting each observation by the square root of the proper random effects variance, then using OLS on the transformed data. While this is more work than using the VWLS or Metareg commands, it is necessary when estimating more advanced meta-regression models. 56

57. Comparison of REMR Results in Environmental Justice (n=680) MetaregVWLSMWLS (MLE)(MOM)(MLE) Intercept.068.073.068 (.006)(.008)(.006) Pollution-.020-.020-.020 (.008)(.011)(.008) Census Tracts.011.009.011 (.006)(.008)(.006) Select on DV-.078-.093-.078 (.019)(.025)(.019) 57

58. Consequences of Inadequately Addressing Effect Size Heterogeneity Inadequately addressing heterogeneity in effect sizes means you have a misspecified REMR model. Consequences if r(M e,M k ) = 0? Consequences if r(M e,M k ) ≠ 0? Consequences of remaining effect size heterogeneity in REMR model are akin to specification error in OLS regression 13

59. Validity Check: Can Meta- Regression Really Recover the Quantity of Interest from Original Studies? 59

60. Simulation Strategy Two-level Monte-Carlo simulation –Assume Population Regression Model Y=β 0 + β 1 X 1 + β 2 X 2 + β 3 X 3 + ε. First level simulate 10,800 original studies by Generate X from distribution N(0,1,Cxx), generate Y using β 1 =.2,.7 (X 1 is focal predictor), generate ε ~ N (0, 30) Allow measurement of X and Y to vary across studies [X, X*1000, and ln(X)] Estimate sample regression models that employ combinations of X 1, X 2, and X 3 60

61. Simulation Strategy (cont.) Second level simulate 1000 meta-analyses of sample size 500 Estimate meta-regression model –Y=β 0 + β 1 M 1 + β 2 M 2 + ε –M represent moderator variables Key Question: Can REMR recover partial correlation parameter from population regression model in original studies? 61

62. Simulation Results 62

63. Section IV: Advanced Meta-Regression 63

64. “Meta-Analytic techniques developed to date have not addressed the problem of combining information on second-order effects, such as partial regression coefficients. Consequently, meta-analysis has not been very useful in the synthesis of evidence from causal/explanatory research." (Becker 1992: 342) 64

65. Remaining Problems in Meta-Regression 1.Non-Independence of Observations 2.Unequal Numbers of Effect Sizes per Original Study 65

66. Problem 1: Non-Independence of Observations Reasons for Correlated Observations 1.Common data sets across studies 2.Common research teams across studies 3.Single original study produces multiple effect sizes (generally incorporates both 1 and 2) Consequences of correlated observations for REMR: incorrect parameter st. errors 66

67. Addressing Problem 1: The Approach in Meta-Analysis Traditional Solutions in Meta-Analysis 1.Select one effect size per study to use in meta-analysis 2.Calculate average effect size per study and use this in meta-analysis 67

68. Addressing Problem 1: The Approach in Meta-Analysis Why the Traditional Solutions are Really Bad Ideas: 1.Assumes each additional effect size from a particular study contains no information 2.Cannot estimate effect of moderators within studies, only across studies 3.Dramatically reduces sample size and statistical power 68

69. Addressing Problem 1: The Approach in Econometrics The Traditional Solution in Econometrics: Cluster Robust Variance Estimation (CRVE) –i.e., Huber-White Standard Errors as adjusted for clusters by Liang and Zeger (1986) and Arellano (1987). –Effect sizes Θ ig clustered by original study G 69

70. Addressing Problem 1: The Approach in Econometrics In Matrix Form: V[b REMRg ] = (M’W’WM) -1 (Σ M g ’W’(e g e g ’)WM g ) (M’W’WM) -1 [24] 70

71. 71 (τ 2 +v 11 +σ 2 1 ) ρ1ρ1 0........0 ρ1ρ1 (τ 2 +v 21 +σ 2 1 ).. 0.σ 2 2 **ρ2ρ2 ρ2ρ2 ρ2ρ2.. ρ2ρ2 ρ2ρ2 ρ2ρ2.. ρ2ρ2 ρ2ρ2 ρ2ρ2.. ρ2ρ2 ρ2ρ2 ρ2ρ2.... ee’=..... 0. σ 2 g **ρgρg ρgρg. ρgρg ρgρg 0...... 0ρgρg ρgρg

72. Addressing Problem 1 Using Example REMR Model CRVEREMR Intercept.068.068 (.006)(.011) Pollution-.020-.020 (.008)(.015) Census Tracts.011.011 (.006)(.010) Select on DV-.078-.078 (.019)(.032) 72

73. Addressing Problem 1 Using CRVE CRVE REMR model should be standard practice in the Social Sciences -- BUT Only three examples of published CRVE CRVE models. Why? Researchers developing statistics of meta- analysis unfamiliar with developments in econometrics, and vice versa. 73

74. Extensions to the CRVE REMR Model 1.Small number of clusters? –Use the “Wild Cluster Bootstrap” –Cameron, Gelbach, and Miller 2008 2.Two-way clustering? E.g., effect sizes clustered by study and by data set? –Use “multi-level clustering” –Cameron, Gelbach, and Miller (2011) 74

75. Problem 2: Unequal Numbers of Effect Sizes Per Study Reasons for Unequal Numbers of Effect Sizes Per Study –Varying numbers of subgroup analyses –Varying sensitivity tests Consequences of Unequal Numbers of Effect Sizes Per Study –Few studies may dominate data set and REMR results, threatening external validity 75

76. Addressing Problem 2: The Approach in Meta-Analysis Traditional Solutions in Meta-Analysis 1.Select one effect size per study to use in meta-analysis 2.Calculate average effect size per study and use this in meta-analysis As we have seen, this is really bad advice 76

77. Addressing Problem 2: The Approach in Econometrics The Econometric approach to Problem 3 generally takes one of two forms. 1.Relative “importance weights” attached to observations (e.g., weight = 1/g) 2.Generalized Estimating Equations (GEE) 77

78. Addressing Problem 2 Using GEE A Simple Introduction to GEE –E[Θ i ] = h (Mb) [25] Where h is a link function (identity in regression) –V[Θ i |M i ] = σ 2 i = g([Mb i ] / φ[26] i.e., the variance is a function of the expectation φ (phi) is a scale parameter that we will ignore today 78

79. Addressing Problem 2 Using GEE A Simple Introduction to GEE (cont.) –Assume we have three effect sizes from two studies: [Θ 11, Θ 21, Θ 22 ] Θ = (1/3) Θ 11 + (1/3) Θ 21 + (1/3) Θ 22 [27] Θ = (1/2) Θ 11 + (1/2) Θ 21 [28] Θ = (1/2) Θ 11 + (1/4) Θ 21 + (1/4) Θ 22 [29] –All are unbiased estimates of population effect size Θ, but with different variances 79

80. Addressing Problem 2 Using GEE A Simple Introduction to GEE (cont.) –GEE estimates Θ using an optimal weighting scheme employing the correlation among observations from the same cluster (ρ). In this simple example: Θ = [((1+ρ) / (3+ρ)) Θ 11 ] + [(1 / (3+ρ)) Θ 21 ] + [(1 / (3+ρ)) Θ 22 ][30] 80

81. Addressing Problem 2 Using GEE A Simple Introduction to GEE (cont.) –We still need an estimate of the conditional variance of the sample effect size Θ i. GEE estimates this variance (σ 2 ) using: σ 2 = (A) 1/2 R (A) 1/2 [31] A has E[Θ i ] = h Mb on the diagonal R is the “working correlation matrix” For REMR, the exchangeable working correlation matrix is most appropriate 81

82. Addressing Problem 2 Using GEE Obtaining GEE Estimates Estimate a naïve model for Θ assuming independent observations to obtain starting values for b and e. Use e to estimate the elements of the working correlation matrix R. Re-estimate the structural model with weights derived from the estimated values of ρ in R. Iterate steps 2 and 3 to convergence. 82

83. Addressing Problem 2 Using GEE A Bonus from Using GEE to Estimate the REMR Model: If we use the “empirical” matrix ee’ rather than the “model” matrix ee’, GEE also controls for Problem 2 –GEE calculates parameter estimates controlling for clustering –Empirical GEE calculates parameter variance estimates controlling for clustering 83

84. Addressing Problem 4 Using Example REMR Model CRVEGEE REMRREMRREMR Intercept.068.068.067 (.006)(.011)(.011) Pollution-.020-.020-.015 (.008)(.015)(.014) Census Tracts.011.011.008 (.006)(.010)(.009) Select on DV-.078-.078-.072 (.019)(.032)(.021) 84

85. Addressing Problem 2 Using GEE GEE is an excellent choice for estimating the REMR model. GEE REMR is quite rare – fewer than half a dozen in the published literature Preferred over CRVE when number of effect sizes are highly unequal across studies. CRVE preferred in balanced samples. 85

86. Estimating Advanced REMR Models in Stata Stata cannot estimate directly either the CRVE REMR or the GEE REMR model. Neither VWLS nor Metareg can estimate these models. Models must be estimated using MWLS data in conjunction with “robust cluster” or “xtgee” commands in Stata. 86

87. The Take Aways 1.Meta-regression is a powerful and under- utilized tool for synthesizing research in the social sciences. 2.When estimating meta-regression models, social scientists should use the CRVE REMR or the GEE REMR model. 3.Social scientists should not use the standard REMR model or the HLM REMR model. 87

88. 88 “(Meta-analysis) is going to revolutionize how the sciences... handle data. And it is going to be the way many arguments will end.” T. Chalmers, quoted in Mann 1990: 480

89. Evan Ringquist eringqui@indiana.edu 89