2. EBM Workshop: Treatment
Abdolmehdi Baghaei MD.
Resident of Internal Medicine
Oct

3. OPTIONAL COMPONENTS TO BE ADDED BY THE PHYSICIAN
PATIENT
Values, Preferences
Concerns, Expectations
Life predicament
HUMILITY
Non-authoritarian
practice
CHARITY
EBM is not a required practice (yet)
EBM
Charity – EBM not mandatory (not yet)
Humility – EBM is not for those who prefer an authoritarian practice of medicine
Enthusiasm – have fun with the challenge, variety and change!!!
PHYSICIAN
Training
Expertise
Continued Learning
Demand for proof
INFORMATION
Clinically relevant
Proven by research
Current, up to date
ENTHUSIASM
Challenge, Variety,
Change

4. THE FIVE BASIC STEPS OF EBM
Clinical Question
Patient-focused, problem-oriented
2. Find Best Evidence
Literary Search
3. Critical Appraisal
Evaluate evidence for quality and usefulness
4. Apply the Evidence
Implement useful findings in clinical practice
5. Evaluate
The information, intervention, and EBM process

5. Objectives:
To define characteristics of a suitable report of clinical trials
To define the items of paper evaluation in clinical practice
To demonstrate peer review check lists for papers that report clinical trials

6. EBM QUESTION: Should include multiple factors
(Examples)
P PATIENT type of patient or population
Ex: 47 yr male w/DM2 and cellulitis toe, 25 yr female w/DVT and chest pain
E EXPOSURE environmental, personal, biological
Ex: TB, tobacco, drug, diet, pregnancy or menopause, MRSA, allergy
I INTERVENTION clinical intervention
Ex: medication, procedure, test, surgery, radiation, drug, vaccine
C COMPARISON compare alternative treatment
Ex: other prior, new or existing therapy
O OUTCOME clinical outcome of interest
Ex: Reduced death rate in 5 yrs, decreased infections, fewer hospitalizations

7. FRAMING THE QUESTION (Example: PICO) ELEMENT PROMPTS THE QUESTION:
Patient How would I describe a group of patients similar to mine?
Intervention What main action am I considering?
Comparison What is/are the other options?
Outcome What do I (or the patient) want to happen (or not happen)?
Example:
P: In kids under age 12 with poorly controlled asthma on metered dose inhaled steroids…
I: would the addition of salmetrol to the current therapy
C: compared to increasing the dose of current steroid
O: lead to better control of symptoms without increasing side effects?

8. CATEGORY OF QUESTION MAJOR CATEGORIES Diagnosis Prognosis
Therapy/ Treatment PICO
Harm (iatrogenic, other) PEO
MISCELLANEOUS
Quality of care
Health economics
Office Management
Etc.

9. CRITICAL APPRAISAL
Interpreting the evidence
How to read a paper
How to do the math

10. CRITICAL APPRAISAL
IMPORTANT!
You do NOT have to become a researcher, epidemiologist, or statistician to practice EBM.
Focus on how to USE research reports – not on how to generate them!

11. Four levels of evidence:
To help clinicians rank quality between evidence sources David Sackett, MD, popularized the evidence-based medicine pyramid.
Four levels of evidence:
Most desirable at top

18. Internal validity Components of internal and external validity
of controlled clinical trials
Internal validity
Extent to which systematic error (bias) is minimized in clinical trials
How to Evaluate
Assessment Index
Biased allocation to comparison groups
Selection bias
Unequal provision of care apart from treatment under evaluation
Performance bias
Biased assessment of outcome
Detection bias
Biased occurrence and handling of deviations from protocol and loss to follow up
Attrition bias

19. External validity Components of internal and external validity
of controlled clinical trials
External validity
Extent to which results of trials provide a correct basis for generalization to other circumstances.
How to Evaluate
Assessment Index
Age, sex, severity of disease and risk factors, comorbidity
Patients
Dosage and route of administration, type of treatment within a class of treatments, concomitant treatments
Treatment Regimen
Level of care (primary to tertiary) and experience and specialization of care provider
Setting
Type or definition of outcomes and duration of follow up be assessed
Modalities of outcomes

23. Quantifying Treatment Effects

24. Rationale Any treatment involves tradeoffs
Weigh benefits against risks/costs
Benefit $$
Harm

25. Rationale
Sometimes the decision is difficult! $$
Harm
Benefit

26. Rationale
And this one?
How big is this box? $$
Harm
Benefit

27. Rationale
Tests can help us understand who is most likely to benefit from a treatment
And this one?
How big is this box? $$
Harm
Benefit

28. Rationale
Tests can help us understand who is most likely to benefit from a treatment
Rapid strep to decide who will benefit from penicillin
BNP to decide who will benefit from furosemide
CRP to decide who will benefit from statins

29. Rationale The utility of a test depends on:
How beneficial the treatment is
How harmful the treatment is
How much the test tells us about these benefits and harms in a given individual
Risk of harm from the test itself

30. Rationale The topic for this lecture The utility of a test depends on:
How beneficial the treatment is
How harmful the treatment is
How much the test tells us about these benefits and harms in a given individual
Risk of harm from the test itself
The topic for this lecture

31. Is the intervention beneficial?
Randomized trials compare an outcome in treated to untreated persons
MI in 10% vs. 15%
Duration of flu symptoms 3 vs. 5 days

32. Is the intervention beneficial?
Randomized trials compare an outcome in treated to untreated persons
MI in 10% vs. 15%
Duration of flu symptoms 3 vs. 5 days
*Statistics* are used to decide if should reject the “null hypothesis” and accept that the intervention is beneficial

33. Is the intervention beneficial?
But statistics cannot help us interpret effect size

34. Quantifying the Benefit
Effect size
How do we summarize and communicate this?
What is really important for clinicians and policymakers?
Clinical Significance Vs
Statistical Significance

35. Quantifying the Benefit
Effect size
How do we summarize and communicate this?
What is really important for clinicians and policymakers?
Example: MI in 10% vs. 15%
Q: What do we do with these two numbers?

36. Quantifying the Benefit
Two simple possibilities:
10% / 15% = 0.66
15% - 10% = 5%

37. Quantifying the Benefit
Two simple possibilities:
10% / 15% = 0.66
15% - 10% = 5%
Relative Risk (RR)
Absolute Risk Reduction (ARR)

38. Quantifying the Benefit
Relative risk as a measure of effect size
RR = 0.66 – is this big or small?

39. Quantifying the Benefit
Relative risk as a measure of effect size
RR = 0.66 – is this big or small?
MI: 10% vs. 15% in 10 years
Death: 50% vs. 75% in 3 years
Basal Cell CA: 2% vs. 3% in lifetime

40. Quantifying the Benefit
Relative risk as a measure of effect size
RR = 0.66 – is this big or small?
MI: 10% vs. 15% in 10 years
Death: 50% vs. 75% in 3 years
Basal Cell CA: 2% vs. 3% in lifetime
Medium
Big
Small

41. Quantifying the Benefit
Relative risk as a measure of effect size
RR = 0.66 – is this big or small?
MI: 10% vs. 15% in 10 years
Death: 50% vs. 75% in 3 years
Basal Cell CA: 2% vs. 3% in lifetime
RR is NOT the best measure of effect size

42. Quantifying the Benefit
Absolute risk reduction (ARR) is better
ARR = Risk difference = Risk2 – Risk1

43. Quantifying the Benefit
Absolute risk reduction (ARR) is better
RR ARR
MI: 10% vs. 15% in 10 years %
Death: 50% vs. 75% in 3 years %
Basal Cell CA: 2% vs. 3% in lifetime %

44. Q: What does the 34% reduction mean?

45. Nimotop® Ad Graph 33% 22% Risk1 = 61/278 = 21.8% Risk2 = 92/276 = 33%
Nimotop bar is ½ the height of the control group bar. The control group bar looks like it’s 68% of the way to the top, so the implication is that the RRR = 50% and the ARR = 0.5 x 68% = 34%, but of course, it’s the RRR that’s 34%, and the ARR is 0.11
33%
22%
Risk1 = 61/278 = 21.8%
Risk2 = 92/276 = 33%
RR = 22%/33% = .66
ARR = 33% - 22% = 11%

46. Nimotop® Ad Graph What is 34%? 33% 22% Risk1 = 61/278 = 21.8%
Nimotop bar is ½ the height of the control group bar. The control group bar looks like it’s 68% of the way to the top, so the implication is that the RRR = 50% and the ARR = 0.5 x 68% = 34%, but of course, it’s the RRR that’s 34%, and the ARR is 0.11
33%
22%
Risk1 = 61/278 = 21.8%
Risk2 = 92/276 = 33%
RR = 22%/33% = .66
ARR = 33% - 22% = 11%
What is 34%?

47. Nimotop® Ad Graph 33% 22% Risk1 = 61/278 = 21.8% Risk2 = 92/276 = 33%
Nimotop bar is ½ the height of the control group bar. The control group bar looks like it’s 68% of the way to the top, so the implication is that the RRR = 50% and the ARR = 0.5 x 68% = 34%, but of course, it’s the RRR that’s 34%, and the ARR is 0.11
33%
22%
Risk1 = 61/278 = 21.8%
Risk2 = 92/276 = 33%
RR = 22%/33% = .66
ARR = 33% - 22% = 11%
Relative risk reduction (RRR) =
1 – RR = = .34 or 34%

48. Quantifying the Benefit
RRR is no better than RR
RR RRR
MI: 10% vs. 15% in 10 years %
Death: 50% vs. 75% in 3 years %
Basal Cell CA: 2% vs. 3% in lifetime %

49. Quantifying the Benefit
RRR is ALWAYS bigger than ARR
(unless untreated risk is 100%)

50. Quantifying the Benefit
BEWARE of risk reduction language!!
ARR or RRR?
“We reduced risk by 34%”
“Risk was 34% lower”

51. Quantifying the Benefit
BEWARE of risk reduction language!!
ARR or RRR?
“We reduced risk by 34%” can’t tell
“Risk was 34% lower” can’t tell
Very hard to be unambiguous!

52. Quantifying the Benefit
Another reason that ARR is better:
Translate it into “Number Needed to Treat”
NNT = 1/ARR

53. Why is NNT = 1/ARR? 100 SAH patients treated 67 no stroke anyway
33 strokes with no treatment R2
11 strokes prevented R1
22 strokes with treatment
22 strokes with Nimotop®

54. Why is NNT 1/ARR? Treat 100 SAH patients  prevent 11 strokes
Ratio manipulation:
100 treated 1 treated 9.1 treated
11 prevented .11 prevented 1 prevented = =

55. Why is NNT 1/ARR? Treat 100 SAH patients  prevent 11 strokes
Ratio manipulation:
100 treated 1 treated 9.1 treated
11 prevented .11 prevented 1 prevented = =
1/ARR = NNT

56. Why is NNT 1/ARR? NNT best expressed in a sentence:
“Need to treat 9.1 persons with SAH using nimodipine to prevent 1 cerebral infarction”

57. Quantifying the Benefit
NNT calculation practice
RR ARR NNT?
MI: 10% vs. 15% in 10 years %
Death: 50% vs. 75% in 3 years %
Basal Cell CA: 2% vs. 3% in lifetime %

58. Quantifying the Benefit
NNT calculation practice
RR ARR NNT?
MI: 10% vs. 15% in 10 years % 20
Death: 50% vs. 75% in 3 years %
Basal Cell CA: 2% vs. 3% in lifetime %

59. Quantifying the Benefit
NNT calculation practice
RR ARR NNT?
MI: 10% vs. 15% in 10 years % 20
Death: 50% vs. 75% in 3 years % 4
Basal Cell CA: 2% vs. 3% in lifetime %

60. Quantifying the Benefit
NNT calculation practice
RR ARR NNT?
MI: 10% vs. 15% in 10 years % 20
Death: 50% vs. 75% in 3 years % 4
Basal Cell CA: 2% vs. 3% in lifetime %

61. Quantifying the Benefit
NNT expression practice
RR ARR NNT?
MI: 10% vs. 15% in 10 years % 20
Death: 50% vs. 75% in 3 years % 4
Basal Cell CA: 2% vs. 3% in lifetime %
Statins
Chemo
Sunscreen every day

62. Quantifying the Benefit
NNT expression practice
“Need to treat 20 patients with statins for 10 years to prevent 1 MI”
“Need to treat 4 patients with chemo for 3 years to prevent 1 death”
“Need to treat 100 patients with sunscreen every day for their whole life to prevent 1 basal cell”

63. Population: hypertensive 60-year-olds Outcome: stroke over 5 years Depiction of Results in Control Group
Ref:
Ref:

64. Population: hypertensive 60-year-olds Outcome: stroke over 5 years Depiction of Results in Treatment Group
Ref:

65. Example 1
Randomized controlled trial of the effects of hip replacement vs. screws on re-operation in elderly patients with displaced hip fractures.
Parker MH et al. Bone Joint Surg Br. 84(8):

66. Example 1 Re-operation No Re-operation Hip Replacement 12 217 229
Internal Fixation with Screws 90
136
226
Parker MH et al. Bone Joint Surg Br. 84(8):

67. Example 1 Re-operation No Re-operation Risk Hip Replacement 12 217 229
12/229 =
5.2%
Internal Fixation with Screws 90
136
226
90/226 =
39.8%

68. Example 1 Re-operation No Re-operation Risk Hip Replacement 12 217 229
12/229 =
5.2%
Internal Fixation with Screws 90
136
226
90/226 =
39.8%
RR = R1/R2 = 5.2% / 39.8% = .13
RRR = 1-RR = = 87%
ARR = R2 – R1 = 39.8% - 5.2% = 34.6%
NNT = 1/ARR = 1/ = 3
“Need to treat 3 patients with hip replacement instead of screws to prevent 1 from needing a re-do operation”

69. Example 2
JUPITER: Randomized controlled trial of high dose rosuvastatin in patients with LDL<130 and CRP>2.0
Ridker et al. NEJM 2008; 359:

70. Example 2
Ridker et al. NEJM 2008; 359:

71. Example 2
Ridker et al. NEJM 2008; 359:

72. Example 2 HR = (R1/R2) (from regression) = .56
RRR = 1-HR = = 44%
ARR = R2 – R1 = = .59 / 100py*
= / py
NNT = 1/ARR = 1/.0059 = 100/.59 = 169 pys
“Need to treat 169 patients for a year to prevent 1 CVD event”
Or better:
“Need to treat 85 patients for 2 years to prevent 1 CVD event”
(average treatment duration in trial was 1.9 years)
* py = person-years

73. Example 4 Warfarin vs. placebo for atrial fibrillation
Warfarin Placebo
Risk of major bleed (/yr) 1.2% 0.7%
Ann Intern Med 1999; 131:

74. Example 4 Warfarin vs. placebo for atrial fibrillation
RR = R1/R2 = 1.2% / .7% = 1.7
RR (flipped) = R2/R1 = .7% / 1.2% = .59
RRR (flipped) = 1-RR = = 41%
ARR = R2 – R1 = .7% - 1.2% = -.5%
“ARI” – Absolute risk increase = 0.5%
NNT = 1/ARR = 1/-.5% = -200
“NNH” – Number needed to harm = -NNT = 1/ARI = 200
“If you treat 200 Afib patients with warfarin, you will cause 1 major bleed”

75. Circling back to test utility…
Tests help determine:
If the RR applies
Treatment for a disease doesn’t help if you don’t have the disease!
Interactions (RR is higher or lower than average)
Statins more effective if CRP is high?
Patients with gene XYZ more likely to have a side effect
Baseline risk
The higher the risk, the larger the ARR, the smaller the NNT

76. Key Concepts Test utility depends on how good the treatment is
RR and p-values good for hypothesis testing/statistics
ARR and NNT (and NNH) better for interpreting clinical importance
ARR = risk difference
NNT = 1/NNT
Beware RRR and ambiguous language

77. Where To Start

78. Where To Start
Trials are usually published in a set format: abstract; introduction; methods; results and discussion plus a conclusion.
Use a check list when you read a RCT (e.g. CONSORT, CASP). But always start by asking two basic question:
Where is the trial published?
Is the trial sponsored?

79. Patient Intervention Comparator Outcome.
Critical appraisal starts with a well-formulated question. This typically has four parts and the mnemonic is PICO:
Patient
Intervention
Comparator
Outcome.

80. METHODS Is the precise aim of the study described?
Dose the section contains of all steps taken to avoid bias?
Randomization
Blindness
Baseline data
etc.
What were the inclusion and exclusion criteria of the trial?

81. METHODS Was the group size and duration of the study sensible?
Is the new drug being compared with the gold standard?
Are realistic comparative doses being used?
Is the variable selected for measurement really related to the answer being sought?

82. RESULT
Statistical tests and analysis (beta error, alpha error, p- value, etc.)
Confidence intervals
How are the results expressed? (relative or absolute measures; NNT)
Are all patients accounted for? (intention to treat versus pre-protocol analysis)
Surrogate markers versus hard clinical end point.
Statistical significance versus clinical significance

83. DICUSSION Can the results be applied to the local patient population?
What is the authors’ conclusion?
What are the main defects of the study?

84. FINAL CONSIDERATION: “STEP acronym”
SAFETY
TOLERABILITY
EFFICACY
PRICE

85. Judging the quality of research
JAMA articles
Checklists e.g. CASP
NHS Critical Appraisal Skills Programme
A framework for thinking about the quality of research

86. Intent to Treat Analysis
People in a trial still sometimes do what they want instead of what you told them to do
Some people assigned to placebo will go to their private doctor and be treated with the experimental drug
Some people assigned to the active drug will not take their medicine
How do you analyze these people?

87. Intent to Treat Analysis
People should be analyzed in the groups that they were assigned to
If assigned to placebo, analyze as placebo
If assigned to active drug, analyze as active drug, even if you have evidence that they did not take the drug

88. Intent to Treat Analysis: Example
Coronary Drug Project
Men given clofibrate vs. placebo
Outcome was 5-year mortality

89. Intent to Treat Analysis: Example
Intent-to-treat analysis of 5 year mortality
20.0% in clofibrate group
20.9% in placebo group
On-treatment analysis of the men who took more than 80% of the clofibrate over 5 years
15.0% 5-year mortality in clofibrate compliant

90. Intent to Treat Analysis
An individual assigned to a particular intervention group is included in that group’s outcome statistics even if he/she never receives the intervention
Preserves the full value of randomization

91. Example: Experimental Designs
Coronary Primary Prevention trial, 1985
Treatment groups: men with high serum cholesterol randomized to receive either cholestyramine or an equally foul-tasting placebo
Outcome was cardiac death after 7-10 years

92. Grades of Recommendations for a Specified Level of Baseline Risk
RCTs, no heterogeneity, CIs all on one side of threshold NNT A1
RCTs, no heterogeneity, CIs overlap threshold NNT A2
RCTs, heterogeneity, CIs all on one side of threshold NNT B1
RCTs, heterogeneity, CIs overalp threshold NNT B2
Observational studies, CIs all on one side of threshold NNT C1
Observational studies, CIs overlap threshold NNT C2