1. How to Critically Appraise Literature
James M. Bolton MD
Associate Professor, University of Manitoba
Departments of Psychiatry, Psychology, Community Health Sciences
Adjunct Scientist, Manitoba Centre for Health Policy
Medical Director, Crisis Response Centre, WRHA
Summer Rounds, University of Manitoba
June 28, 2016

2. Disclosures Relationships with commercial interests:
Grants/Research Support:
Canadian Institutes of Health Research
New Investigator Award (113589)
Brain and Behavior Research Foundation (NARSAD)
Young Investigator Award
Speakers Bureau/Honoraria: Nil
Consulting Fees: Nil
Other: Nil

3. Objectives
Learn key questions to ask when reviewing a scientific article
Understand the importance of bias

4. Ten Key Questions in Critical Appraisal
Is the study relevant?
Does the study add anything new?
What type of research question is being asked?
Is the study design appropriate for the research question?
Did the study methods address sources of bias?
Was the study performed according to original protocol?
Did it test a stated hypothesis?
Were statistical analyses performed correctly?
Do the data justify the conclusions?
Are there any conflicts of interest?
Young JM & Solomon MJ. Nat Clin Pract Gastroenterol Hepatol. 2009

5. Question #1 So what? Is the study important? What to look for
The best done study is irrelevant if it does not address an important question
What to look for
The introduction
Authors need to convince a reviewer that the topic/area is important to health, the public, etc.

6. Question #2 Does it add anything new?
What to look for
The introduction
Literature review – how thorough is the review, how well is it summarized?
The reviewer should have a clear idea that this is not only something new but also not simply an incremental addition to previous knowledge
You may need to do a quick literature search
Different journals have different thresholds

7. Is the Study Design Appropriate?
Relates to the research question
Efficacy of an intervention: RCT
Frequency of events/risk factors: observational studies

8. Hierarchy of Evidence Systematic reviews and meta-analyses of RCTs
Cohort studies
Case-control studies
Cross sectional surveys
Case reports

9. Do the Methods Address Bias?
When the results of a study deviate from the truth
When bias is due to the methodology of a study (vs. random chance) – this is called systematic bias

10. Pannucci CJ & Wilkins EG. Plast Reconstuct Surg. 2010

11. Addressing Bias: Example 1
Study: Examining rates of mental disorders in parents bereaved by MVC death of their child
Method: Parents surveyed in a bereavement support group

12. Addressing Bias: Example 1
Examining rates of mental disorders in parents bereaved by MVC death of their child
Method: Parents surveyed in a bereavement support group
Potential Bias: Selection Bias
Attendees in bereavement groups have been shown to be healthier than non-attendees
Solution: conduct a population-based study of bereaved parents using administrative data

13. Addressing Bias: Example 2
Study: Is cancer associated with suicide?
Design: Using CancerCare registries, examining all people in Manitoba who were newly diagnosed with cancer, and examining suicide rates within 1 year
Statistical analysis: unadjusted logistic regression models that examined the association with cancer diagnosis and suicide

14. Addressing Bias: Example 2
Potential Bias: Confounding
Mental disorder are associated with both the predictor variable (cancer) and outcome variable (suicide)
Solution: multivariate regression that adjusts for mental disorders

15. Do the Data Justify the Conclusions?
Meta-analysis of antidepressant trials based on clinical study reports
Main outcomes: mortality and suicidality
Results:
Mortality: Odds ratio 1.28; 95% CI
Suicidality: Odds ratio 1.21; 95% CI

16. Do the Data Justify the Conclusions?
Meta-analysis of antidepressant trials based on clinical study reports
Main outcomes: mortality and suicidality
Results:
Mortality: Odds ratio 1.28; 95% CI
Suicidality: Odds ratio 1.21; 95% CI
Conclusions:
“We suggest minimal use of antidepressants in children, adolescents, and young adults, as the serious harms seem to be greater.”
Sharma T et al. BMJ. 2016

17. CASP Critical Appraisal Skills Program
casp-uk.net
Has downloadable checklists that you can use for any type of study

18. Ten Key Questions in Critical Appraisal
Is the study relevant?
Does the study add anything new?
What type of research question is being asked?
Is the study design appropriate for the research question?
Did the study methods address sources of bias?
Was the study performed according to original protocol?
Did it test a stated hypothesis?
Were statistical analyses performed correctly?
Do the data justify the conclusions?
Are there any conflicts of interest?
Young JM & Solomon MJ. Nat Clin Pract Gastroenterol Hepatol. 2009

19. Suggestions Go toward the fear – don’t avoid!
Participate in journal clubs
Take opportunities to review for journals
Learn what top researchers are working on
See comments by other reviewers
If an option, participate in grant review committees
Remember: You don’t need to be a stats wizard!
The core of critical appraisal is based on the importance of the study and the approach to the question
Clinical knowledge goes a long way