1. CRITICAL APPRAISAL Dr. Cristina Ana Stoian Resident Journal Club
June 14, 2012

2. Objectives
Recognize strengths and limitations of various study designs
Calculate study parameters
Sensitivity, specificity, PPV, NPV
RR, ARR, RRR, NNT
Define some basic terms
P-value, type I error, type II error
Discuss general approach to critical appraisal

3. Study Designs Design ‘Starting point’ What is assessed Strength
Limitations
RCT
Exposure status
Outcome
Controls bias
Feasibility
Cohort
Ability to determine incidence; temporality
Time and cost; rare outcome
Case control
Less time and cost
Bias (++)

5. The famous 2x2 table Outcome Yes No Exposure Yes (treatment) a b
(control) c d

6. Calculations Control event rate (CER) = c/c+d
Experimental event rate (EER) = a/a+b
Relative Risk (RR) = EER/CER=(a/a+b)/(c/c+d)
Relative Risk Reduction (RRR) = CER-EER/CER (commonest reported measure of dichotomous treatment effect)
Absolute Risk Reduction (ARR) = CER-EER
Number Needed to Treat (NNT) = 1/ARR
A certain risk reduction may appear impressive but how many patients would you have to treat before seeing a benefit? This concept is called ‘number needed to treat’ and is one of the most intuitive statistics for clinical practice.

7. The famous 2x2 table
Assume a study which recruited 200 patients and randomized 100 patients in each of treatment and control groups.
There were 15 deaths in the treatment group and 20 deaths in the control group
Outcome
Yes No
Exposure
Treatment 15 85
Control 20 80

8. Calculations
Assume a study which recruited 200 patients and randomized 100 patients in each of treatment and control groups.
There were 15 deaths in the treatment group and 20 deaths in the control group
EER: 15/100 (0.15 or 15%)
CER: 20/100 (0.20 or 20%)
RR: 0.15/0.20 = 0.75
ARR: 0.20 – 0.15 = 0.05
RRR: ( )/0.20 = 0.25 (25%)
NNT: 1/0.05 = 20

9. The famous 2x2 table Disease Positive Negative Test a TP b FP
PPV=TP/TP+FP c FN d TN
NPV=TN/TN+FN
Sn=TP/TP+FN
Sp=TN/TN+FP
PPV is directly proportional to disease prevalence. Therefore, only use NPV
or PPV if the ratio of disease/healthy patients is same as pop’l prevalence.
Otherwise, use Likelihood Ratios.

10. Sensitivity Proportion of persons with condition who test positive
Sensitivity = true positive test result/ all patients with disease
Sensitivity = a/a + c (TP/TP+FP)
SnNOut = a highly sensitive test that gives a negative result rules out the Dx
Desired for screening tests
POWER = sensitivity = 1 – beta error

11. Specificity Proportion of persons without condition who test negative
Specificity =true negative test results/all pts without disease
Specificity = d/ d + b (TN/TN+FN)
SpPIn = if using a highly specific test, a positive test result rules in the Dx

12. Example
Some researchers have conducted a QI project to assess the diagnostic accuracy of the urine dipstick leukocyte esterase test to predict a bacterial UTI. As part of their study, they assessed 100 patient urine samples which underwent both a dipstick and a urine culture. A dipstick was defined as positive if it showed ‘trace’ leukocytes or greater. A urine culture was defined as positive if it grew a single organism at > 1 x 107
75 patients had positive urine cultures. Of these, 60 had a positive dipstick. Of the 25 patients with negative cultures, 5 had a positive dipstick
Please calculate the Sn, Sp, PPV and NPV of the dipstick test

13. The famous 2x2 table UTI (Urine culture) Positive Negative
Dipstick test 60 5 65
PPV=60/65=92% 15 20 35
NPV=20/35=57% 75
Sn=60/75=80% 25
Sp=20/25=80%
100
PPV is directly proportional to disease prevalence. Therefore, only use NPV
or PPV if the ratio of disease/healthy patients is same as pop’l prevalence.
Otherwise, use Likelihood Ratios.

14. Another example
Test for strep throat. You test 100 swabs from patients complaining of sore throat. Test was positive 40 times and in 20 cases the culture came back positive for strep. Test was negative 60 times and only in 5 of these cases the cultures were positive.
Which of the following is true about the new test?
Positive predictive value = 50%
Sensitivity = 80%
Negative predictive value >90%
All of the above

15. The famous 2x2 table Strep culture Positive Negative Swab test 20 40
PPV=20/40=50% 5 55 60
NPV=55/60=92% 25
Sn=20/25=80% 75
Sp=55/75=73%
Total=100

16. Effectiveness vs Efficacy
Effectiveness = ability of an intervention to achieve the desired results under USUAL conditions, ie day-to-day use. Effectiveness describes how well drug works, its side effects and ease of use
Efficacy = ability of an intervention to achieve the desired results under IDEAL conditions, eg clinical trial. Does not describe how well tolerated/ease of use, only how well it gives the desired result.

17. Definitions
Null hypothesis – there is no difference between the groups being compared
P-value – the probability that the difference observed in the study is simply due to chance (also referred as type I error)
Type II error – based on the estimate of the ‘true difference’ between groups, what is the chance that the difference in the study sample will not be statistically significant (incorrectly accepting the null hypothesis when there is a real effect)
Power = 1 – type II error

18. Measuring Outcomes You can use the 2 x 2 table to look at error
Null Hypothesis
Outcome
Chance
Real effect
Accept
Correct
Incorrect
Type II error
Reject
Type I error

19. General approach to critical appraisal
What is the primary research question?
Methods
Study design
Sample, exposure, outcome
Measurement
Analysis
Results
Limitations
Conclusions

20. Key points in critical appraisal
Internal validity
The degree to which the study truly answers the question it poses
External validity
Can the study results be generalized to other populations?
(can the results help me in caring for my patients)

21. Internal Validity Were study participants randomized?
Were all participants who entered the study properly accounted for (follow-up)?
Was the study ‘blinded’?
Were the groups similar at the start of the trial?
Aside from study intervention, were the groups treated equally?

22. External Validity Can the results be applied to my patient care?
Consider inclusion/exclusion criteria
Study population
Were all clinically important outcomes considered?
Are the likely treatment benefits worth the potential harm and costs?

23. THANK YOU !