Appraising A Diagnostic Test Clinical Epidemiology and Evidence-based Medicine Unit FKUI-RSCM

What is diagnosis ? Increase certainty about presence/absence of disease Disease severity Monitor clinical course Assess prognosis – risk/stage within diagnosis Plan treatment e.g., location Stalling for time! Knottnerus, BMJ 2002

Key Concept Pre-test Probability The probability of the target condition being present before the results of a diagnostic test are available. Post-test Probability The probability of the target condition being present after the results of a diagnostic test are available.

Key Concept Pre-test Probability Post-test Probability The probability of the target condition being present before the results of a diagnostic test are available. Post-test Probability The probability of the target condition being present after the results of a diagnostic test are available. 4

Basic Principles (1) Ideal diagnostic tests – right answers: (+) results in everyone with the disease and ( - ) results in everyone else Usual clinical practice: The test be studied in the same way it would be used in the clinical setting Observational study, and consists of: Predictor variable (test result) Outcome variable (presence / absence of the disease)

Basic Principles (2) Sensitivity, specificity Prevalence, prior probability, predictive values Likelihood ratios Dichotomous scale, cutoff points (continuous scale) Positive (true and false), negative (true & false) ROC (receiver operator characteristic) curve

General structure : 2 X 2 table Target disorder Positive (disease) Negative (normal) Predictor Test positive True positive TP a False positive FP b negative False negative FN c True negative TN d

Disease (+) (-) Total Test (+) True pos a False pos b a+b Test (-) False neg c True neg d c+d a+c b+d a+b+c+d a+c a+b+c+d Prevalence Pretest probability

Sensitivity The proportion of people who truly have a designated disorder who are so identified by the test. Sensitive tests have few false negatives. When a test with a high Sensitivity is Negative, it effectively rules out the diagnosis of disease. SnNout

Specificity The proportion of people who are truly free of a designated disorder who are so identified by the test. Specific tests have few false positives When a test is highly specific, a positive result can rule in the diagnosis. SpPin

Disease (+) (-) Totals a b a+b c d c+d a+c b+d +c+d Test (+) a b a+b Test (-) c d c+d a+c b+d +c+d SpPIn SnNOut d/b+d a/a+c Sensitivity Specificity Probability of negative test result in patients without the disease Probability of positive test result in patients with the disease

SnNOut The sensitivity of dyspnea on exertion for the diagnosis of CHF is 100% (41/(41+0)), and the specificity 17% (35/(183+35)). If DOE, it is very unlikely that they have CHF (0 out of 41 patients with CHF did not have this symptom). "SnNOut", which is taken from the phrase: "Sensitive test when Negative rules Out disease".

SpPin Conversely, a very specific test, when positive, rules in disease. "SpPIn"!  The sensitivity of gallop for CHF is only 24% (10/41), but the specificity is 99% (215/218).  Thus, if a patient has a gallop murmur, they probably have CHF (10 out of 13).

Iron deficiency anemia result (Serum ferritin) Sensitivity=a/a+c=90% Specificity =d/b+d=85% LR + = sn/(1-sp)=90/15=6 Pos predictive value=a/a+b=73% Neg predictive value=d/c+d=95% Prevalence= (a+c)/(a+b+c+d)= 32% Iron deficiency anemia Totals Present Absent Diag nostic test result (Serum ferritin) (+) <65 mmol/L 731 a 270 b 1001 a+b (-) >65 mmol/L 78 c 1500 d 1578 c+d 809 a+c 1770 b+d 2579 a+b+c+d Outcome Predictor Posttest odd = Pretest odd x Likelihood Ratio

Odds = ratio of two probabilities Odds = p/1-p Probability = odds/1+odds Likelihood ratio (+): Prob (+) result in people with the disease Prob (+) result in people w/out the disease Pretest Odds X LR = Posttest Odds

probability of a test result in pts with disease Key Concept Likelihood Ratio Relative likelihood that a given test would be expected in a patient with (as opposed to one without) a disorder of interest. probability of a test result in pts with disease LR= probability of the test result in pts without disease

Likelihood ratios (LR) General Rules of Thumb LR > 10 or < 0.1 produce large changes in pre-test probability LR of 5 to 10 or 0.1 to 0.2 produce moderate changes LR of 1 to 2 or 0.5 to 1 produce small changes in pre-test probability

Likelihood ratio Test Test A B C (1-Sn)/Sp= - + = Sn/(1-Sp) do not test get on with treatment do not test treat Test Test A B C 0 .10 .20 .30 .40 .50 .60 .70 .80 .90 1 pretest probability posttest probability PreTest odds x LR pretest probability

The usefulness of five levels of a diagnostic test result Serum ferritin (mmol/L) Iron def positive Iron def negative Likelihood ratio Diagnostic impact No % Very positive <15 474 59 (474/809) 20 1.1 (20/1770) 52 Rule in SpPin Moderately positive 15-34 175 22 (175/809) 79 4.5 (79/1770) 4.8 Intermed High Neutral 35-64 82 10 (82/809) 171 (171/1770) 1 Indeter mine Moderately negative 65-94 30 3.7 (30/809) 168 9.5 (168/1770) 0.39 Intermed low Extremely negative >95 48 5.9 (48/809) 1332 75 (1332/1770) 0.08 Rule out SnNout 809 100 (809/809) 1770 (1770/1770)

Pretest probability Likelihood ratio Posttest probability

T4 level in suspected hypo- thyroidism in children T4 value Hypo thyroid Eu 5 or less 18 1 5.1 – 7.0 7 17 7.1 – 9.0 4 36 9 or more 3 39 Totals 32 93 T4 value Hypo thyroid Eu ≤ 5 18 1 > 5 14 92 Totals 32 93 T4 value Hypo thyroid Eu ≤ 7 25 18 > 7 7 75 Totals 32 93 T4 level in suspected hypo- thyroidism in children Cutoff point Sens Spec 5 0.56 0.99 7 0.78 0.81 9 0.91 0.42 T4 value Hypo thyroid Eu ≤ 9 29 54 > 9 3 39 Totals 32 93 For tests / predictors with continuous values result , cutoff points should be determine to choose the best value to use in distinguishing those with and without the target disorder

Cutoff point Sens Spec Cutoff point Sens TP 1-Spec FP 5 0.56 0.99 7 0.78 0.81 9 0.91 0.42 Cutoff point Sens TP 1-Spec FP 5 0.56 0.01 7 0.78 0.19 9 0.91 0.58

Accuracy of the test The accuracy of the test depends on how well the test separates the group being tested into those with and without the disease in question Accuracy is measured by the area under the ROC curve. An area of 1 represents a perfect test; an area of 0.5 represents a worthless test (AUC) 0.90-1.00 = excellent (A) 0.80-0.90 = good (B) 0.70-0.80 = fair (C) 0.60-0.70 = poor (D) 0.50-0.60 = fail (F)

An ROC curve demonstrates several things: It shows the tradeoff between sensitivity and specificity any increase in sensitivity will be accompanied by a decrease in specificity The closer the curve follows the left-hand border and then the top border of the ROC space, the more accurate the test. The closer the curve comes to the 45-degree diagonal of the ROC space, the less accurate the test. The slope of the tangent line at a cutoff point gives the likelihood ratio (LR) for that value of the test.

Appraising DxTest Is the evidence valid? (V) Was there an independent, blinded comparison with a gold standard? Was the test evaluated in an appropriate spectrum of patients? Was the reference standard applied regardless of the test result? Was the test validated in a second, independent group of patients?

Can I trust the accuracy data? RAMMbo Recruitment: Was an appropriate spectrum of patients included? (Spectrum Bias) Maintainence: All patients subjected to a Gold Standard? (Verification Bias) Measurements: Was there an independent, blind or objective comparison with a Gold standard? (Observer Bias; Differential Reference Bias) Guyatt. JAMA, 1993

Critical Appraisal Is this valid test important? (I) Distinguish between patients with and those without the disease Two by two tables Sensitivity and Specificity SnNOut SpPIn ROC curves Likelihood Ratios

Critical Appraisal Can I apply this test to my patient (A) Similarity to our patient Is it available Is it affordable Is it accurate Is it precise

Critical Appraisal Can I apply this test to my patient? Can I generate a sensible pre-test probability Personal experience Practice database Assume prevalence in the study

Critical Appraisal Diagnosis Can I apply this test to a specific patient Will the post-test probability affect management Movement above treatment threshold Patient willing to undergo testing

Thank You