Understanding heterogeneity in systematic reviews and met-analysis meta-analysis generates a single best estimate of effectmeta-analysis generates a single best estimate of effect –what are the underlying assumptions? how to judge consistency of resultshow to judge consistency of results –4 strategies what to do if inconsistencywhat to do if inconsistency

The right question all cancer therapy for all cancersall cancer therapy for all cancers all antiplatelet agents for all atheroembolic events (heart, head, leg)all antiplatelet agents for all atheroembolic events (heart, head, leg) all aspirin doses for strokeall aspirin doses for stroke 30 to 300 mg. for stroke30 to 300 mg. for stroke what is guide about when right to pool?what is guide about when right to pool?

What were your criteria? what made you decide some were OK and some were not?what made you decide some were OK and some were not? across range ofacross range of –patients –interventions –comparators –outcomes effect more or less same effect more or less same if notif not –big effect in severe patients, no effect in mild –big effect in high dose, no effect in low –big effect in short term, none in long term

Are you happy pooling?

What criteria were you using? similarity of point estimatessimilarity of point estimates –less similar, less happy overlap of confidence intervalsoverlap of confidence intervals –less overlap, less happy

RRR (95% CI)

Homogenous test for heterogeneity what is the p-value? what is the null hypothesis for the test for heterogeneity? Ho: RR1 = RR2 = RR3 = RR4 p=0.99 for heterogeneity

Heterogeneous p-value for heterogeneity < test for heterogeneity what is the p-value?

Homogenous p=0.99 for heterogeneity I 2 =0% What is the I 2 ?

Heterogeneous p-value for heterogeneity < I 2 =89% What is the I 2 ?

I 2 Interpretation No worries 0% 25% Only a little concerned 50% Getting concerned 75% Very concerned 100% Why are we pooling?

Heterogeneous p-value for heterogeneity < I 2 =89% If this result, what next?

Does Vitamin D prevent non-vertebral fractures? systematic review and meta-analysis patients: over 60 intervention: Vitamin D (cholecalciferol or ergocalciferol) outcome: non-vertebral fractures – –follow-up at least a year methods: blinded RCTS

Relative Risk with 95% CI for Vitamin D Non-vertebral Fractures Chapuy et al, (2002) 0.85 (0.64, 1.13) Pooled Random Effect Model 0.82 (0.69 to 0.98) p= 0.05 for heterogeneity, I 2 =53% Chapuy et al, (1994) 0.79 (0.69, 0.92) Lips et al, (1996) 1.10 (0.87, 1.39) Dawson-Hughes et al, (1997) 0.46 (0.24, 0.88) Pfeifer et al, (2000) 0.48 (0.13, 1.78) Meyer et al, (2002) 0.92 (0.68, 1.24) Trivedi et al, (2003) 0.67 (0.46, 0.99) Favours Vitamin D Favours Control Relative Risk 95% CI

Relative Risk with 95% CI for Vitamin D (Non-Vertebral Fractures, Dose >400) Chapuy et al, (1994) 0.70 (0.69, 0.92) Dawson-Hughes et al, (1997) 0.46 (0.24, 0.88) Pfeifer et al, (2000).48 (0.13, 1.78) Chapuy et al, (2002) 0.85 (0.64, 1.13) Trivedi et al, (2003) 0.67 (0.46, 0.99) Pooled Random Effect Model 0.75 (0.63 to 0.89) p= 0.26 for heterogeneity, I 2 =24% Favours Vitamin D Favours Control Relative Risk 95% CI

Relative Risk with 95% CI for Vitamin D (Non-Vertebral Fractures, Dose = 400) Lips et al (1996) 1.10 (0.87, 1.39) Meyer et al (2002) 0.92 (0.68, 1.24) Pooled Random Effect Mode 1.03 (0.86 to 1.24) p = 0.35 heterogeneity, I 2 =0% Favours Vitamin D Favours Control Relative Risk 95% CI

Heterogeneity look for explanation patients interventions outcomes risk of bias No good explanation? What to do? decrease confidence in effect estimates

Beta-blockers in non-cardiac surgery: Stroke p=0.99 for heterogeneity I 2 = 0%

Vitamin D + Calcium vs Calcium

p= 0.32 for heterogeneity

Vitamin D + Calcium vs Calcium p= 0.32 for heterogeneity I 2 = 14%

Vitamin D + Calcium vs Placebo/Control

p= 0.06 for heterogeneity

Vitamin D + Calcium vs Placebo/Control p= 0.06 for heterogeneity I 2 = 50%

Summary starting assumption of pooled estimate – –across pts, intervention, outcome, similar effect broad criteria for meta-analysis desirable – –maximize precision – –maximize generalizability – –can check out assumption is there excessive heterogeneity? – –point estimates too variable – –confidence intervals non-overlapping – –low heterogeneity p-value – –high I 2 if so, look for explanation – –patients, intervention, outcome, methodology