1. I-squared
Conceptually, I-squared is the proportion of total variation due to ‘true’ differences between studies. Proportion of total variance due to random effects.

2. Comparison
Depnds on k
Depends on Scale Q X P
T-squared T
I-squared
I-squared does depend on the sample sizes (Ns) of the included studies. The random-effects variance has some size, which is indexed in units of the observed effect size (e.g., r). The larger the sample size, the smaller the sampling variance, and thus the larger I-squared. To me, the prediction interval (coming up) is the most interpretable.

3. Prediction or Credibility Intervals
Makes sense if random effects (REVC > 0). Otherwise, just CI for mean.
M is the random effects mean (summary effect).
The value of t is from the t table with your alpha and df equal to (k-2) where k is the number of independent effect sizes (studies). The variance VM is the squared standard error of the RE summary effect, and T2 is the REVC estimate. Conceptually, prediction interval is not an estimate of error of the mean. (See next slide.)

4. Confidence Intervals vs Prediction Intervals
The confidence interval is supposed to contain a parameter that has a single but unknown value.
95CI should contain the population mean 95 percent of the time it is computed
The prediction interval is supposed to contain a percentage of a normal distribution that has an unknown mean and variance.
95PI will contain 95 percent of the underlying ‘true’ values of the effect sizes that actually vary from situation to situation.
Difference between CI and PI widely misunderstood

5. CI vs PI CI contains the Mean.
From earlier slide.
CI contains the Mean.
PI contains the Distribution of random effects. PI tells you where to expect study outcomes; it is NOT an error variance.
Prediction Interval
Confidence Interval

6. CMA Exercise 3 Review Kvam results. Find and interpretQ
REVC (tau-squared)
I-squared
Download Excel PI calculation program
Find and interpret PI

7. Find and interpret Q REVC (tau-squared) I-squared Prediction Interval
The prediction interval is very wide. It indicates that the effect of exercise on depression is quite variable. The average effect size is only a crude approximation of what to expect, even without sampling error. It is reasonable to expect studies where exercise truly has no effect.

8. Analysis 3 – overall (summary) results
Number of studies, k = 23, total people, N = 977
Overall mean: g = -.68, CI = [-.92 to -.44]; moderate to large effect size
Heterogeneity: Q(22) = 68.74, p < I-squared = 67.99; moderate to large heterogeneity
Did not report REVC or prediction interval, but they should (tsk, tsk). The impact of heterogeneity is larger than what they seem to acknowledge. (Not sure if overall data includes drugs as control condiditions.)
Research question or Study aims
Search & eligibility
Coding, computation of effects, conversions
Analysis
Overall
Graphs
Moderators
Sensitivity
Discussion

9. Break (lunch) Coming up next -> Research question or Study aims
Search & eligibility
Coding, computation of effects, conversions
Analysis
Overall
Graphs
Moderators
Sensitivity
Discussion

10. Graphs Communicate results Reveal data problems
Suggest appropriate models

11. Graphs 1 Forest Plot Overall results Study information
Convention is to alphabetize, but many other ways
Estimate
Forest Plot
Overall results
Study information
Forest plot symbols
Overall mean
Conf Int
Signifcant vs.
Not

12. CMA graphics Exercise
Load the Kvam data

13. Get the forest plot and modify it
Run analyses -> select by -> Prepost -> uncheck 2 -> bottom left click Random
Right click Statistics for each study -> customize stats
uncheck Standard err, Variance, Zvalue
Effect measure -> Hedge’s g
Box with double down arrows to list sample size
Barchart box to get Weights
High resolution plot -> Edit -> Header -> type ‘Analysis 1’ ->Apply
Edit -> labels -> type ‘Exercise Control’ -> Apply
Edit -> footer -> delete ‘Meta Analysis’ -> Apply
Format -> set scale -> -4 to +4
Format -> study and summary symbols -> click the empty square

14. This is a close match to the article
This is a close match to the article. Not sure the problem with the relative weight. Can fix in PowerPoint.

15. Plot of follow-up results.
Note that the effect sizes are small, but we do not know what happened to the means. Need an extra graph or table.

16. Graphs 3
Some indication that exercise is about as effective as medication and that it may add to effects beyond medication. Too few studies to be conclusive.

17. Forest Plot
Sort by effect size, then plot. Steady progression? Missing middle? Heavy weight studies in the middle? Expect some curl at the ends for small samples.
Source:

18. Go to Data Entry
Right click the effect size (Hedges g).
Sort A to Z
Redo the graph
What is the problem with this one?

19. Forest Plot 4 - Cumulative
Pineles (2014). Miscarriage and exposure to tobacco smoke.
Shows that contrary to some authors, there has always been good evidence of risk.
Overall OR and CI plotted by adding each study over time.
Note on sociology of science; often largest ES first; decline over time.

20. Funnel Plots
Shows ES by precision. Expect a funnel shape if fixed-effects sampling distribution. Asymmetry suggests pub bias; excess variance suggests heterogeneity (moderators). This one shows pretty good symmetry but lots of variability (mortality rates by hospital).

21. Graphs 4
Funnel Plot
Trim-and-fill is one kind of sensitivity (what if?) analysis. This is opposite the usual pattern because of the negative mean.
Is there a correlation between sample size and effect size?

22. CMA Exercise 4 Kvam data Select posttest data (exclude follow-up)
Run a funnel plot and interpret
Run trim-and-fill and interpret

23. Funnel Plot -> Run analyses
-> Select data (prepost = 1 for these data)
-> Analyses
-> Publication bias (1)
-> Black and white (2) 2 1
You can customize for PowerPoint or for publication

24. Trim & fill -> Publication bias -> Next table (several)
Notice that it plots fixed
Effects and to the left by
Default. It says no
Imputed studies here.
Note mean = -.48

25. Trim & fill Random & right of mean Selected. Now it imputes
8 studies. Mean moves
From -.48 to -.30.
Then show the funnel plot.

26. Funnel with imputed studies
-> Plot observed and imputed studies