1. Nut and Bolts of Critical Appraisal of Medical Literature
Focus on Diagnosis
Nick Myles, MD
Pathologist
St.Paul’s Hospital

3. Objectives
Learn why research evidence on diagnosis should be critically appraised
Learn Evidence-Based Medicine tools for critical appraisal of medical literature on diagnosis

4. Objectives
Learn why research evidence on diagnosis should be critically appraised
Learn Evidence-Based Medicine tools for critical appraisal of medical literature on diagnosis

5. Why Critical Appraisal and not just use any published book/paper in practice?
Your thoughts?

7. Examples from Clinical Medicine
Dr Spock: recommends Prone against Supine Position
Crib deaths best evidence: risk of crib death is much higher in Prone position

8. Examples from Clinical Medicine
Some anti- arrhythmia drugs is MI:
Pathophysiologic mechamism: prevent arrhythmia to prevent death
RCT evidence: some anti-arrhythmics increase mortality

9. Examples from Clinical Medicine
Pathophysiology: Vitamins A and E are beneficient antooxidants, so use widely
RCT: use of vitamins A and E increase mortality

10. Example from Lab Medicine
Some pathology experts claim 0% error and 0% tolerance to error
EMB claims there are no tests which are 100% accurate

11. Examples from Lab Medicine
Some experts claim 0% error tolerance
EMB claims there are no tests which are 100% accurate

14. What to expect from the test?
If I test breast cancer for hormone receptors and get negative result, how can I be certain that it’s indeed negative?
What do you need to know?
Your thoughts

15. What to expect from the test?
If my wife gets her mammogram and it is negative, how can I be certain that it’s indeed negative?
What do you need to know?
Your thoughts

16. Formulate answerable question:
PICO PIRTO
Population Population
Intervention Index test
Comparator Reference test
Outcome Time
Outcome
for Interventions for Diagnosis

19. Grades of Diagnostic Evidence
Grade 1: Systematic Reviews of good cohorts with reference; absolute Sp or Sn
Grade 2: SRs of cohorts with split sample validation; with reference test
Grade 3: SRs of non-consequitive cohorts with inconsistent reference tests
Grade 4: case controls, poor, non-independent inconsistent reference
Grade 5: EXPERT OPINION

20. consistent level 1 studies B
Grades of Recommendation A
consistent level 1 studies B
consistent level 2 or 3 studies or extrapolations from level 1 studies C
level 4 studies or extrapolations from level 2 or 3 studies D
level 5 evidence or troublingly inconsistent or inconclusive studies of any level

21. Critical Appraisal Steps
Formulate answerable question:
PICO or PIRTO
Search for the best already APPRAISED evidence (not only PubMed!)
Check:
NICE guidance (National Institute for Clinical Excellence)
Cochrane library of systematic reviews
If still no answer, then consider doing the following:

22. Critical Appraisal Steps
Formulate answerable question:
PICO or PIRTO
Search for the best evidence (not only PubMed!)
Extract and Appraise the best evidence
Apply

23. Evidence Search Pubmed vs others or several engines WARNING
If single engine used, 30-40% studies will not show up with the same key words

24. Select the Best Evidence
Hierarchy of Evidence: WHY ?
Your thoughts?

25. When finding evidence one has to go from top
And only then down
Why? Your thoughts?

27. Objectives
Learn why research evidence on diagnosis should be critically appraised
Learn Evidence-Based Medicine tools for critical appraisal of medical literature on diagnosis

28. Objectives
Learn why research evidence on diagnosis should be critically appraised
Learn Evidence-Based Medicine tools for critical appraisal of medical literature on diagnosis

29. EBM Validated Tools
Easy and proven, yet comprehensive:
YOU
Can Do It

30. EBM Tools Checklists: STARD and QUADRAS
Online Calculators: CEBM and U of T, +others
Software: CAT Maker
All open access, free

31. EBM Tools Checklists: STARD (25 items)
Standards of Reporting Diagnostic Accuracy
Adapted by 200+ biomedical journals (as per 2009), with only one pathology journal listed
to allow readers to assess the potential for bias in the study (internal validity) and to evaluate its generalisability (external validity).

32. EBM Tools Checklists: QUADRAS
a tool for the quality assessment of diagnostic accuracy studies
risk of bias assessment when extracting data from multiple studies for systematic review

34. EBM Tools Online Calculators: CEBM www.cebm.net
Plenty of support, free access
Critical appraisal checklists for all types of clinical medical literature
Example: Diagnosis (next slide)

40. EBM Tools Online Calculators: U of Toronto
Easy, excellent graphic output, concise
Examples: diagnosis, RCTs, Cohorts, NNT etc
Perfect for data simulation
Example: diagnosis next page

43. EBM Tools Software: CAT Maker All-in-one solution
Allows you to generate one page
Critical Appraisal Topic Summary with all numeric output
May crush..

50. ER-positive test
ER negative test
Prev
50%

51. Prev
80%

52. Simulation (if we are online) Task 1a
What’s the diagnostic accuracy of mammography test (90% sensitive and 95% specific) in 40 y.o. women?
So, how many errors (false +, false -) do you expect after 1000 mammograms? Overall, about 1% of women in this age group have breast cancer
Need one minute?

53. Simulation (if we are online) Task 1b
What’s the diagnostic accuracy of mammography test in 70 y.o. women?
(assuming 90% sensitive and 95% specific)
How many errors (false +, false -) do you expect after 1000 mammograms? Overall, 15% of women in this age group have breast cancer.
Need one minute?
Your answer: FP ? FN?

54. Simulation (if we are online) Task 1a
prevalence: 1% (40 y.o)
Predicted errors:
False positive 50/59 total positive
False negative 1/940

55. Task 1a

56. Simulation (if we are online) Task 1b
prevalence: 1% (40 y.o.)
Predicted errors:
False positive 43/183 total test positive
False negative 15/823 total test neg

58. NL inquiry into Estrogene Receptor Lab Accuracy Done in 1998-2006

60. Task 2
Adequate (ASCO2010) Breast cancer Estrogen test is at least 90% sensitive and 90% specific.
NL re-tested cases in Ontario and found 400+ false negative results.
The case is settled for 17M in 2009.
What is the expected False Neg error rate of adequate ER test?

61. Task 2 answer
Conclusion: under condition of 80% prevalence of ER positive breast cancers,
expected NPV of the adequate test is 64-69%

62. Objectives
Learn why research evidence on diagnosis should be critically appraised
Learn Evidence-Based Medicine tools for critical appraisal of medical literature on diagnosis

63. Summary
Critical Appraisal is necessary to assess validity of the studies before they are applied to clinical practice
EBM tools are easy to use and allow for an independent analysis of the published data and claimed results.

64. Thank you