1. Effect Sizes

2. Coding Coding Effect Sizes Coding Studies
Usually the first thing I do because you can’t include a study in a meta-analysis if you can’t compute an effect size from it.
However, I will present it second here, after
Coding Studies
Separate coding of studies from coding of effect sizes to reduce biased coding.

3. Many Kinds of Effect Sizes
For Continuous Variables
d (the standardized mean difference statistic)
r (the correlation coefficient)
For Dichotomous Variables
o (the odds ratio, for dichotomous outcomes)
l (the log odds ratio)
The relative risk
The rate difference etc.
All have the effect of standardizing results into a common metric.

4. Effect Sizes for Dichotomous Data

5. The Odds Ratio
Most widely used effect size measure for dichotomous outcomes
Where A, B, C, and D are cell frequencies
OR = 1 is no effect.
Lower Bound 0 (Control better than Treatment)
upper bound infinity (Treatment < Control)

6. The Original Simpson & Pearson’s Hospital Staff Incidence Data

7. A Fourfold Table from Pearson’s Hospital Staff Incidence Data
Condition of Interest
Group Immune Diseased All
Inoculated A = B = M =297
Not Inoculated C = D = M =279
Total N = N = T = 576

8. OR from Proportions
The odds ratio can also be computed from proportions
And there are many variations on this formula depending on which proportions you do or do not know

9. Log Odds Ratio Easier to work with statistically
Makes interpretation more intuitive, similar in some respects to d.
0 = no effect
Range is +/-.
In example, LOR = ln(3.04) = 1.11

10. Converting LOR to d
May wish to do this if most of your effect sizes are in d and just a few in OR, and you want to pool them all:
Cox (1970):
Sanchez-Meca et al (Psych Methods) showed this approximation works well

11. The Relative Risk (Risk Ratio)
Also used for dichotomous outcomes
Where p1+ and p2+ are the marginal proportions of the first row and the second row, respectively.
Commonly converted to the Log Risk Ratio

12. Example Condition of Interest Group Immune Diseased All
Inoculated p11 = p12 = p1+ = .5156
Not Inoculated p21 = p22 = p2+ = .4843
Total p+1 = p+2 = .1858
The probability of being immune if inoculated is 1.22 times higher than the probability if not inoculated.

13. Converting RR to Odds Ratio
Which in our example is

14. Number Needed to Treat
How many units must be treated to produce a successful outcome
Where R1 is the success rate in the treatment group and R0 is the success rate in the control group
That is, treat 9 units with T to obtain one more success than would have occurred under C
A measure of cost-effectiveness of tmt

15. Difference Between Proportions (Risk Difference)
Intuitive:
For Pearson’s Data:
89.2% of those inoculated were immune
73.1% of those not inoculated were immute, so
The difference in immunity rates for those inoculated or not is about 16.1%

16. Analyzing Fourfold Tables by the Correlation
Tetrachoric, approximated by
Useful if binary measures created by dichotomizing continuous ones
Bad to use the phi coefficient (ordinary Pearson correlation applied to binary data)
Sensitive to marginal distributions

17. Preferences? Most tend to prefer OR or LOR
Likelihood ratio test of the null
But all measures for fourfold tables have potential interpretation problems
Reporting success rates, and NNT, advisable.
Bonnet (2007) American Psychologist, also Kraemer (2004) Statistics in Medicine are nice summaries of issues in summarizing fourfold tables

18. Effect Sizes for Continuous Variables
r (the Pearson correlation)
Not widely used or reported in treatment outcome studies
d (the standardized mean difference statistic)
By far the most widely used

19. d: The Standardized Mean Difference Statistic
Where s is an estimate of the standard deviation of the numerator, and standardizes the numerator.
It is often the pooled standard deviation (a weighted average of the standard deviation of each group)

20. Estimating d d itself Algebraic equivalents to d
Good approximations to d
Methods that require intraclass correlation
Methods that require ICC and change scores
Methods that underestimate effect
Note: Italicized methods will be covered.

21. Sample Data Set I: Two Independent Groups
Treatment Comparison
3 2
4 2
4 4
4 5
5 5
6 6
6 7
7 8
7 9
Mean
Standard Deviation
Sample Size 10 10
Correlation between treatment and outcome is r = -.055

22. Calculating d

23. Interpreting d Cohen suggested:
0 = no effect
.20 = small effect
.50 = medium effect
.80 = large effect
Lipsey and Wilson found a slightly narrower range empirically (.3, .5, .67)
However, remember that what counts as small, medium or large will vary from topic to topic (e.g.,SCDs)

24. More on Interpreting d Suppose d = .51 (Shadish et al 1993 MFT)
Convert to Cohen’s U3 index
From the Unit Normal Curve
Implies that a therapy client at the mean was better off than 69.5% of control clients; z
Below z
Above z
Between mean and z
Ordinate
0.50
0.6915
0.3085
0.1915
0.3521
0.51
0.6950
0.3050
0.1950
0.3503
0.52
0.6985
0.3015
0.1985
0.3485
0.53
0.7019
0.2981
0.2019
0.3467

25. More on Interpreting d
Use U3 to compute an improvement in percentile rank:
Improvement = U3 – 50%
= 69.5% - 50%
= 19.5%
The average treatment unit improves 19.5% compared to the average control unit

26. More on Interpreting d
Converts to a correlation coefficient of roughly .25 (Hedges & Olkin, 1985, p. 77)
So that treatment accounts for about r2 = = 6% of outcome variance

27. More on Interpreting d
Rosenthal and Rubin 1982: Translates into a treatment success rate of about 62% in marital and family therapies compared to 38% in control groups

28. Hedges’ Correction for Small Sample Bias
d overestimates effect size in small samples (< total)
Correction is
I always use this correction as it never harms estimation. In SPSS
COMPUTE D = ES*(1-(3/((4*(NT+NC))-9))).