1. META-ANALYSES: APPROPRIATE GROWTH OR MALIGNANT TUMOR? Ian Shrier MD, PhD, Dip Sport Med (FACSM) Associate Professor, Dept of Fam Med, McGill UniversityAssociate Professor, Dept of Fam Med, McGill University Centre for Clinical Epidemiology and Community Studies, SMBD-Jewish General Hospital and McGill UniversityCentre for Clinical Epidemiology and Community Studies, SMBD-Jewish General Hospital and McGill University Past-president, Canadian Academy of Sport MedicinePast-president, Canadian Academy of Sport Medicine Canadian Academy of Sport Medicine LAcadémie Canadienne de Médecine du Sport

2. GROWTH VS QUALITY? Cochrane Collaboration – 10 pointsCochrane Collaboration – 10 points collaboration collaboration building on the enthusiasm of individuals building on the enthusiasm of individuals avoiding duplication avoiding duplication minimizing bias minimizing bias keeping up to date keeping up to date striving for relevance striving for relevance promoting access promoting access ensuring quality ensuring quality continuity continuity enabling wide participation enabling wide participation Cochrane Collaboration – 10 pointsCochrane Collaboration – 10 points collaboration collaboration building on the enthusiasm of individuals building on the enthusiasm of individuals avoiding duplication avoiding duplication minimizing bias minimizing bias keeping up to date keeping up to date striving for relevance striving for relevance promoting access promoting access ensuring quality ensuring quality continuity continuity enabling wide participation enabling wide participation GrowthGrowth

3. GROWTH VS QUALITY? Cochrane Collaboration – 10 pointsCochrane Collaboration – 10 points collaboration collaboration building on the enthusiasm of individuals building on the enthusiasm of individuals avoiding duplication avoiding duplication minimizing bias minimizing bias keeping up to date keeping up to date striving for relevance striving for relevance promoting access promoting access ensuring quality ensuring quality continuity continuity enabling wide participation enabling wide participation Cochrane Collaboration – 10 pointsCochrane Collaboration – 10 points collaboration collaboration building on the enthusiasm of individuals building on the enthusiasm of individuals avoiding duplication avoiding duplication minimizing bias minimizing bias keeping up to date keeping up to date striving for relevance striving for relevance promoting access promoting access ensuring quality ensuring quality continuity continuity enabling wide participation enabling wide participation GrowthGrowth QualityQuality XX

4. DATA ENTRY

5. OBJECTIVE? A Meta-analysis is an Observational Study, Subject to Selection Bias, Confounding, and Measurement Error Like Any Other! Platt, personal communication Computer programs and standardized scoring make it very easy to make mistakes that are difficult for the reviewer to pick up.

6. OVERVIEW Inclusion Criteria / Data abstractionInclusion Criteria / Data abstraction AnalysesAnalyses Interpretation of EvidenceInterpretation of Evidence Inclusion Criteria / Data abstractionInclusion Criteria / Data abstraction AnalysesAnalyses Interpretation of EvidenceInterpretation of Evidence

7. INCLUSION CRITERIA - SEARCH Search StrategySearch Strategy Does Vitamin C prevent death due to cancer in the general population Does Vitamin C prevent death due to cancer in the general population Search StrategySearch Strategy Does Vitamin C prevent death due to cancer in the general population Does Vitamin C prevent death due to cancer in the general population33,300 Vitamin C #1 HitsTermsSearch 34,415 Ascorbic Acid #2 9,545Ascorbate#3 4,376 #3 NOT #2 #4 42,098 #1 OR #2 OR #3 #5

8. INCLUSION CRITERIA – RCTs?

9. Good (56%) Poor (44%) INCLUSION CRITERIA – RCTs? Furlan, 2006

10. DATA ABSTRACTION Peer-review for stretching article:Peer-review for stretching article: The RR 0.79 … was based on the article in Pope 2000. The RR 0.79 … was based on the article in Pope 2000. Peer-review for stretching article:Peer-review for stretching article: The RR 0.79 … was based on the article in Pope 2000. The RR 0.79 … was based on the article in Pope 2000. … multivariate analysis HR=1.04 (0.82,1.33) … multivariate analysis HR=1.04 (0.82,1.33) Actual articleActual article The X article … Cox regression … differing exposure times for different subjects. …univariate hazard ratio of 0.95 (CI: 0.77,1.18) The X article … Cox regression … differing exposure times for different subjects. …univariate hazard ratio of 0.95 (CI: 0.77,1.18) Actual articleActual article The X article … Cox regression … differing exposure times for different subjects. …univariate hazard ratio of 0.95 (CI: 0.77,1.18) The X article … Cox regression … differing exposure times for different subjects. …univariate hazard ratio of 0.95 (CI: 0.77,1.18)

11. OVERVIEW Inclusion Criteria / Data abstractionInclusion Criteria / Data abstraction AnalysesAnalyses Interpretation of EvidenceInterpretation of Evidence Inclusion Criteria / Data abstractionInclusion Criteria / Data abstraction AnalysesAnalyses Interpretation of EvidenceInterpretation of Evidence

12. ANALYSES Effect ModifiersEffect Modifiers Study-level confounders Study-level confounders Subject-level confounders Subject-level confounders Publication BiasPublication Bias Papers with significant results are more likely to be published (i.e. selected) Papers with significant results are more likely to be published (i.e. selected) Reporting BiasReporting Bias Outcomes that are statistically significant are more likely to be reported in publications (i.e. selected) Outcomes that are statistically significant are more likely to be reported in publications (i.e. selected) Effect ModifiersEffect Modifiers Study-level confounders Study-level confounders Subject-level confounders Subject-level confounders Publication BiasPublication Bias Papers with significant results are more likely to be published (i.e. selected) Papers with significant results are more likely to be published (i.e. selected) Reporting BiasReporting Bias Outcomes that are statistically significant are more likely to be reported in publications (i.e. selected) Outcomes that are statistically significant are more likely to be reported in publications (i.e. selected)

13. ANALYSES Fixed vs Random Effect Estimate RR vs OR Effect Estimate RR vs OR

14. ANALYSES Effect Estimate RR vs RD Effect Estimate RR vs RD

15. OVERVIEW Inclusion Criteria / Data abstractionInclusion Criteria / Data abstraction AnalysesAnalyses Interpretation of EvidenceInterpretation of Evidence Inclusion Criteria / Data abstractionInclusion Criteria / Data abstraction AnalysesAnalyses Interpretation of EvidenceInterpretation of Evidence

16. GRADE SCALE DECISIONS Atkins. BMC Health Services Research 2005 4 3 2 1 RecommendationBalance Benefit vs HarmRater 8 7 6 5 12 11 10 9 13 17 16 15 Toss up Do it Don't do it Probably do it Do it Probably do it Do it Toss up Don't do it Toss up Probably do it Net benefit Net benefit probably Uncertain net benefit Net benefit Trade offs Net benefit Not net benefit Uncertain net benefit Net benefit 14 Reasons: Lack of information in summaries Missing baseline risk What is sparse information? Reasons: Lack of information in summaries Missing baseline risk What is sparse information?

17. INTERPRETATIONS I believe magnesium has now been shown to be beneficial for patients during the post-MI period (SD-SA) Rev 8 Rev 7 Rev 6 Rev 5 Rev 4 Rev 3 Rev 2 Ag DA Ag DA SA Ag DA - Ag DA - SA Ag SD - Ag DA - Ag Rev 1 I2I2I2I2 Rand. OR Fixed OR Ag DA Ag SD Ag 59% 0.75 (0.61-0.92) 1.01 (0.96-1.07) 69,50559% 0.65 (0.48-0.87) 1.02 (0.96-1.08) 63,047 - DA Ag - SA Ag 21% 0.66 (0.53-0.81) 0.64 (0.52-0.79) 3,68556% 0.38 (0.21-0.66) 0.40 (0.28-0.61) 5970% 0.40 (0.18-0.86) 0.40 (0.19-0.83) 415 SD DA - - SD Ag SD n/a n/a n/a111N 1-231-201-101-51-31 # RCTs

18. CONCLUSION The explosion of meta-analyses has occurred because of apparently simple recipesThe explosion of meta-analyses has occurred because of apparently simple recipes However, a meta-analysis is an observational study with potential errors at many levelsHowever, a meta-analysis is an observational study with potential errors at many levels Authors of meta-analyses need to be appropriately trained Authors of meta-analyses need to be appropriately trained Readers need to be appropriately skeptical Readers need to be appropriately skeptical The benefits of meta-analyses may be lost if current growth remains unrestricted and becomes a malignancyThe benefits of meta-analyses may be lost if current growth remains unrestricted and becomes a malignancy The explosion of meta-analyses has occurred because of apparently simple recipesThe explosion of meta-analyses has occurred because of apparently simple recipes However, a meta-analysis is an observational study with potential errors at many levelsHowever, a meta-analysis is an observational study with potential errors at many levels Authors of meta-analyses need to be appropriately trained Authors of meta-analyses need to be appropriately trained Readers need to be appropriately skeptical Readers need to be appropriately skeptical The benefits of meta-analyses may be lost if current growth remains unrestricted and becomes a malignancyThe benefits of meta-analyses may be lost if current growth remains unrestricted and becomes a malignancy

19. Canadian Academy of Sport Medicine LAcadémie Canadienne de Médecine du Sport