1. Effect Size Calculation for Meta-Analysis Robert M. Bernard Centre for the Study of Learning and Performance Concordia University February 24, 2010 February 24, 2010

2. 10/30/20152 Main Purposes of a Meta-Analysis A meta-analysis attempts to …

3. 10/30/20153 What is an Effect size?

4. 10/30/20154 Types of Effect Sizes Most reviews use … d-family of effect sizes, including the standardized mean difference, or r-family of effect sizes, including the correlation coefficient, or the odds ratio (OR) family of effect sizes, including proportions and other measures for categorical data.

5. 10/30/20155 Effect Size Extraction Effect size (ES) extraction involves … Locating descriptive or other statistical information contained in studies. Converting statistical information into a standard metric (effect size) by which studies can be compared and/or combined.

6. 10/30/20156 Choice of an Effect Size When we have… continuous univariate data for two groups, we typically compute a raw mean difference or a standardized difference – an effect size from the d-family, continuous bivariate data, we typically compute a correlation (from the r-family), or binary data (the patient lived or died, the student passed or failed), we typically compute an odds ratio, a risk ratio, or a risk difference.

7. 10/30/20157 d- Family: Zero Effect Size ES = 0.00 Control Condition Treatment Condition Overlapping Distributions

8. 10/30/20158 d- Family: Moderate Effect Size Control Condition Treatment Condition ES = 0.40

9. 10/30/20159 d- Family: Large Effect Size Control Condition Treatment Condition ES = 0.85

10. Effect Size Interpretation 10/30/201510 Cohen’s (1988) Qualitative Descriptors

11. 10/30/201511 Research designs for d- Family Statistics Independent Groups (posttest-only)EXPY Post (Randomized or Non-randomized) CTY Post One-group (pretest-posttest)Y Pre EXPY Post Independent Groups (pre-post)Y Pre EXPY Post (Randomized or Non-randomized) Y Pre CTY Post EXP = Experimental ConditionCT = Control Condition

12. 10/30/201512 Statistics for d -Family Effect Size Extraction Effect sizes can be extracted using the following reported statistics: Descriptive statistics (means, SDs, sample sizes) Preferred (by far). Exact test statistics (t-values, F-values, etc.) Exact probability values (p =.013, etc.) Approximate comparisons of p to α (p <.05, etc.) By far, the least exact.

13. 10/30/201513 d - Family with Independent Groups (Basic Equation) Note: this equation is the same as adding two SSs and dividing by df Total

14. 10/30/2015 d Family Statistics: Means and Standard Deviations Procedure: 1) Calculate Pooled SD 2) Calculate d

15. 10/30/201515 Alternative Methods of ES Extraction: t-values and F-ratios Study Reports: t(60) = +2.66 Study Reports: F(1, 61) = 7.076 Important Note: Report must indicate direction of the effect (+/–)

16. 10/30/2015 Alternative Methods of ES Extraction: Exact p-value Study Reports: t(60) is sig. p =.01 Look up t-value for p =.01 (df = 60) t = 2.66 Important Note: Report must indicate direction of the effect (+/–)

17. Alternative Methods of ES Extraction: p < α Study Reports: p <.05, n T = 31, n C = 31 Important Note: Report must indicate direction of the effect (+/–) Estimate +t(60) = +2.00 10/30/2015 Compared with 0.676, this ES is only 75% accurate.

18. 10/30/201518 d Family: Adjustment for Small Samples Recommendation: If there are small samples and large samples, convert all d-family statistics to g. N = 60, g is 99% of dN = 40, g is 98% of d N = 20, g is 96% of dN = 10, g is 90% of d N = 60, g is 99% of dN = 40, g is 98% of d N = 20, g is 96% of dN = 10, g is 90% of d

19. 10/30/201519 d- Family Statistics with dependent Groups (pre-post)

20. Relationship Between Effect Size and Pre-Post Correlation 10/30/201520 Means and SDs: d = 0.21 SD Change: d = 0.21, using r = 0.80 Correlation

21. 10/30/201521 d- Family Statistics with Independent Groups (pre-post) Calculate the pooled SD.

22. 10/30/201522 Calculating Standard Error Standard Error: The standard error of g is an estimate of the “standard deviation” of the population, based on the sampling distribution of an infinite number of samples all with a given sample size. Smaller samples tend to have larger standard errors and larger samples have smaller standard errors.

23. 10/30/201523 95th% Confidence Interval 95th Confidence Interval Upper: Lower: Conclusion: Confidence interval does not cross 0 (g falls within the 95th confidence interval). Conclusion: Confidence interval does not cross 0 (g falls within the 95th confidence interval). The 95th Confidence Interval is the range within which it can be stated with reasonable confidence that the true population mean exists. As the standard error decreases (the sample size increases), the confidence interval decreases in width.

24. 10/30/201524 Forest Plot

25. 10/30/201525 Other Important Statistics Variance: Inverse Variance (w): Weighted g (g*w): The variance is the standard error squared. The inverse variance (w) provides a weight that is proportional to the sample size. Larger samples are more heavily weighted than small samples. Weighted g is the weight (w) times the value of g. It can be + or –, depending on the sign of g.

26. 10/30/201526 Average g (g+) is the sum of the weights divided by the sum of the weighted gs.

27. 10/30/201527 Selected References Borenstein, M. Hedges, L.V., Higgins, J.P..,& Rothstein, H.R. (2009). Introduction to meta- analysis. Chichester, UK: Wiley. Glass, G. V., McGaw, B., & Smith, M. L. (1981). Meta- analysis in social research. Beverly Hills, CA: Sage. Hedges, L. V., & Olkin, I. (1985). Statistical methods for meta-analysis. Orlando, FL: Academic Press. Hedges, L. V., Shymansky, J. A., & Woodworth, G. (1989). A practical guide to modern methods of meta-analysis. [ERIC Document Reproduction Service No. ED 309 952].