Statistical Applications for Meta-Analysis Robert M. Bernard Centre for the Study of Learning and Performance and CanKnow Concordia University December 11, 2007 December 11, 2007 Module 2, Unit 13 of NCDDR’s course for NIDRR Grantees Developing Evidence-Based Products Using the Systematic Review Process

12/6/062 Two Main Purposes of a Meta-Analysis Estimate the population central tendency and variability of effect sizes between an intervention (treatment) condition and a control condition. Explore unexplained variability through the analysis of methodological and substantive coded study features.

12/6/063 Effect Size Extraction Effect sizes extraction involves locating and converting descriptive or other statistical information contained in studies into a standard metric (effect size) by which studies can be compared and/or combined.

12/6/064 What is an Effect size? A descriptive metric that characterizes the standardized difference (in SD units) between the mean of a treatment group (educational intervention) and the mean of a control group Can also be calculated from correlational data derived from pre-experimental designs or from repeated measures designs

12/6/065 Characteristics of Effect Sizes Can be positive or negative SD unitsInterpreted as a z-score, in SD units, although individual effect sizes are not part of a z-score distribution Can be aggregated with other effect sizes and subjected to statistical procedures such as ANOVA and multiple regression Magnitude interpretation: ≤ 0.20 is a small effect size, 0.50 is a moderate effect size and ≥ 0.80 is a large effect size (Cohen, 1992)

12/6/066 Zero Effect Size ES = 0.00 Control Condition Treatment Condition Overlapping Distributions

12/6/067 Moderate Effect Size Control Condition Treatment Condition ES = 0.40

12/6/068 Large Effect Size Control Condition Treatment Condition ES = 0.85

12/6/069 ES Calculation: Descriptive Statistics Note: this equation is the same as adding two SSs and dividing by df Total

12/6/0610 Adjustment for Small Samples: Hedges’g Cohen’s d is inaccurate for small samples (N < 20), so Hedges’ g was developed (Hedges & Olkin, 1985) g = Cohen’s d times a multiplier based on sample size

12/6/0611 Example of ES Extraction with Descriptive Statistics Study reports:Treatment mean = 42.8Control Mean = 32.5 Treatment SD = 8.6Control SD = 7.4 n = 26n = 31 Procedure: Calculate SD pooled Calculate d and g

12/6/0612 Alternative Methods of ES Extraction: Exact Statistics Study Reports: t (60) = 2.66, p <.05 Study Reports: F (1, 61) = 7.08, p <.05 Convert F to t and apply the above equation:

12/6/0613 Alternative Methods of ES Extraction: Exact p-value Study Reports: t (60) is sig. p = Look up t-value for p = t = 2.68

12/6/0614 Calculating Standard Error Standard Error: The standard error of g is an estimate of the “standard deviation” of the population, based on the sampling distribution of an infinite number of samples all with a given sample size. Smaller samples tend to have larger standard errors and larger samples have smaller standard errors.

12/6/0615 Test Statistic and Confidence Interval Z-test (Null test: g = 0): 95th Confidence Interval Conclusion: 2.62 > 1.96 (p 0 Conclusion: Confidence interval does not cross 0 (g falls within the 95th confidence interval).

12/6/0616 Other Important Statistics Variance: Inverse Variance (w): Weighted g (g*w): The variance is the standard error squared. The inverse variance (w) provides a weight that is proportional to the sample size. Larger samples are more heavily weighted than small samples. Weighted g is the weight (w) times the value of g. It can be + or –, depending on the sign of g.

12/6/0617 Average g (g+) is the sum of the weights divided by the sum of the weighted gs.

12/6/0618

ES Extraction Exercise Materials: EXCEL SE Calculator EXCEL SE Calculator 5 studies from which to extract effect sizes 5 studies from which to extract effect sizes

12/6/0620 Mean and Variability Variability ES+ Note: Results from Bernard, Abrami, Lou, et al. (2004) RER

12/6/0621 Mean Effect Size Conclusion: Mean g = 0.33 and it is significant. g+ Var SE z

12/6/0622 Variability (Q -Statistic) Question: How much variability surrounds g+ and is it significant? Are the effect sizes heterogeneous or homogeneous? Conclusion: Effect sizes are heterogeneous. Tested with the  2 distribution.

12/6/0623 Homogeneity vs. Heterogeneity of Effect Size If homogeneity of effect size is established, then the studies in the meta-analysis can be thought of as sharing the same effect size (i.e., the mean) If homogeneity of effect size is violated (heterogeneity of effect size), then no single effect size is representative of the collection of studies (i.e., the “true” mean effect size remains unknown)

12/6/0624 Statistics in Comprehensive Meta-Analysis™ Comprehensive Meta-Analysis is a trademark of BioStat® Interpretation: Moderate ES for all outcomes (g+ = 0.33) in favor of the intervention condition. Homogeneity of ES is violated. Q-value is significant (i.e., there is too much variability for g+ to represent a true average in the population).

12/6/0625 Back to ES Calculator 1. Interpretation of Mean Effect Size 2. Interpretation of Q-Statistic

12/6/0626 g+g+ Distribution1: Homogeneous Distribution 2: Heterogeneous Gray shaded area is variation left to be explained by moderators. No variation left to be explained by moderators. Homogeneity versus Heterogeneity of Effect Size

12/6/0627 Examining the Study Feature “Method of ES Extraction” g+ = Overall Effect Exact Descriptive Exact Statistics Estimated Statistics

12/6/0628 Tests of Levels of “Method of ES Extraction” Interpretation: Small to Moderate ESs for all categories in favor of the intervention condition. Homogeneity of ES is violated. Q-value is significant for all categories (i.e., “Method of ES Extraction” does not explain enough variability to reach homogeneity).

12/6/0629 Meta-Regression Seeks to determine if “Method of ES Extraction” predicts effect size. Conclusion: “Method of Extraction” design is a significant predictor of ES but ES is still heterogeneous.

12/6/0630 Sensitivity Analysis Tests the robustness of the findings Asks the question: Will these results stand up when potentially distorting or deceptive elements, such as outliers, are removed? Particularly important to examine the robustness of the effect sizes of study features, as these are usually based on smaller numbers of outcomes

12/6/0631 Sensitivity Analysis: Low Standard Error Samples

12/6/0632 One Study Removed Analysis

12/6/0633

12/6/0634 Steps in Controlling for Study Quality Step one: Are the effect sizes homogeneous? Step two: Does study quality explain the heterogeneity? Step three: Which qualities of studies matter? Step four: How do we deal with the differences?

12/6/0635 Controlling Study Quality Using Dummy Coding in Meta-Regression Categories of Study Quality Dummy 1Dummy 2Dummy 3Dummy

12/6/0636 Adjusting Effect Sizes

12/6/0637 Selected References Bernard, R. M., Abrami, P. C., Lou, Y. Borokhovski, E., Wade, A., Wozney, L., Wallet, P.A., Fiset, M., & Huang, B. (2004). How does distance education compare to classroom instruction? A meta-analysis of the empirical literature. Review of Educational Research, 74(3), Glass, G. V., McGaw, B., & Smith, M. L. (1981). Meta-analysis in social research. Beverly Hills, CA: Sage. Hedges, L. V., & Olkin, I. (1985). Statistical methods for meta- analysis. Orlando, FL: Academic Press. Hedges, L. V., Shymansky, J. A., & Woodworth, G. (1989). A practical guide to modern methods of meta-analysis. [ERIC Document Reproduction Service No. ED ].