1. H676 Week 7 – Effect sizes and other issues
Mayra
Issues related to effect size
Part 7 of the TEXT
Brian
Further methods
Part 8 of the TEXT
Moving from “what works” to “what happens”
Pettigrew, 2015

2. Mayra, you’re on TEXT Chapters 31-34 Effect sizes rather than p-values
Issues related to effect size
Effect sizes rather than p-values
Simpson’s Paradox
Generality of the basic inverse-variance method

3. Brian
TEXT Chapters 35-38
Further Methods
Meta-analysis methods based on direction and p-values
Further methods for dichotomous data
Psychometric meta-analysis

4. Meta-analysis methods based on direction and p-values
What are the different objectives of a meta-analysis of effect sizes vs a meta-analysis of p-values?

5. Different objectives MA of effect sizes MA of p-values
Addresses the magnitude of the effect
MA of p-values
Addresses the null-hypothesis
Tells us only that the effect is probably not zero

6. Other options
What are other options for MA when it is not possible to calculate effect-sizes?
What are the pros and cons of each?

7. Other options – Vote counting
Has no validity (recall chapter 28)
The absence of a sig effect is not evidence that the effect is absent. Why?
Gets even worse with more studies
Why? Small p could be due to small sample or lack of power

8. Other options – The sign test
Simply count N of studies with findings in each direction
This done without regard to statistical significance
Therefore has little value!
The statistical significance of the sign test can be tested using the sign test
In Excel, the =BINOMDIST(n,N,0.5,TRUE)
Where n=n of studies in smaller group, N=Total N of studies

9. Other options – Combining p-values
Makes use of exact p-values
Use only when sample sizes are not available
If sample size is available, can back calculate an ES
Convert 2-tailed tests to 1-tailed
1-tailed p-value includes information on direction as well as magnitude
Run MA of the 1-tailed p-values
Two possible statistical tests of the ultimate ES
They test the null hypothesis that the effect is zero in all of the studies

10. Tests of sig. of analysis of 1-tailed p-values
Fisher’s method:
X2 = minus 2 times the sum of the logged p-values
Distributed as X2 with degrees of freedom = 2*k
Stouffer’s method:
ZStouffer = sum of Zi divided by square root of k
Follows a normal distribution like any other Z

11. Further methods for dichotomous data
Two other methods for MA of odds ratios
The Mantel-Haenzel method
An alternative to the fixed-effect inverse-variance method
Uses weighted average of ORs rather than of log ORs
The one-step (Peto) method
Is an inverse-variance method
But uses an alternative approach to computing the odds ratio and variance for each study
Offers some advantages when some studies have empty cells
Both are available options in CMA

12. Psychometric meta-analysis
Also known as:
Validity generalization meta-analysis
Hunter-Schmidt meta-analysis
Two important issues:
Artifact correction (adjusting for methodological limitations) of interest to anyone
Psychometric MA – of interest when ESs are correlations

13. Psychometric meta-analysis
Effect sizes are corrected for attenuation due to:
Unreliability of measures
Restricted range in either dependent or independent variables for any of the study samples
Restricted variance when continuous variables are (artificially) dichotomized
All corrections lead to larger ES and smaller variance
In conventional MA, reliability would be examined in a meta-regression
Some issues are sources of disagreement among uses of psychometric meta-analysis

14. And now, back to Mayra Moving from “what works” to “what happens”
Pettigrew, 2015