Timothy Wiemken, PhD MPH Assistant Professor Division of Infectious Diseases Diagnostic Tests

Overview 1. Understand the calculation of measures of diagnostic accuracy 2. Understand the clinical utility of measures of diagnostic accuracy 3. Review concepts related to receiver operating characteristic curves

Question If you ask, “how many patients do I need for my study?” and your grant is due tomorrow, what am I going to tell you? a.3 b.Leave me alone c.Make stuff up d.%^$%# e.B and D

Overview 1. Understand the calculation of measures of diagnostic accuracy 2. Understand the clinical utility of measures of diagnostic accuracy 3. Review concepts related to receiver operating characteristic curves

Calculations Example Researchers develop a new test to detect HIV-1 from the blood based on nanotechnology.

Calculations Example How good is this new test? Researchers develop a new test to detect HIV-1 from the blood based on nanotechnology.

Calculations Any new test is compared to a gold standard test to calculate diagnostic accuracy statistics

Calculations Any new test is compared to a gold standard test to calculate diagnostic accuracy statistics 1.Run both tests on a set number of specimens (e.g. gold standard and new test)

Calculations Example What is the gold standard for HIV-1 detection?

Calculations Example What is the gold standard for HIV-1 detection? PCR HIV RNA detection

Calculations Any new test is compared to a gold standard test to calculate diagnostic accuracy statistics 1.Run both tests on a set number of specimens (e.g. gold standard and new test) 2. Compile results into a contingency table

Calculations Any new test is compared to a gold standard test to calculate diagnostic accuracy statistics 1.Run both tests on a set number of specimens (e.g. gold standard and new test) 2. Compile results into a contingency table What the heck is a contingency table?

Calculations Test +Test - Test + Test - 2x2 contingency tables are the backbone of statistics

Calculations Gold Standard Test Result Test +Test - New Test Result Test + Test - This is the layout for calculating diagnostic accuracy statistics

Calculations Gold Standard Test Result Test +Test - New Test Result Test + Test - Each cell represents a specific result.

Calculations Gold Standard Test Result Test +Test - New Test Result Test +TP Test - Each cell represents a specific result. TP: True Positive Both tests agree that the result is positive

Calculations Gold Standard Test Result Test +Test - New Test Result Test +TP Test -TN Each cell represents a specific result. TN: True Negative Both tests agree that the result is negative

Calculations Gold Standard Test Result Test +Test - New Test Result Test +TP Test -FNTN Each cell represents a specific result. FN: False Negative Gold standard says positive, new test says negative

Calculations Gold Standard Test Result Test +Test - New Test Result Test +TPFP Test -FNTN Each cell represents a specific result. FP: False Positive Gold standard says negative, new test says positive

Calculations Gold Standard Test Result Test +Test - New Test Result Test +TPFP Test -FNTN Each cell represents a specific result. Always believe the gold standard. However, it may be wrong. This is a limitation of all diagnostic accuracy statistics.

Calculations Gold Standard Test Result Test +Test - New Test Result Test +TPFP Test -FNTN Each cell represents a specific result. Obviously you always want a lot of true results and very few false results (if any).

Calculations Gold Standard Test Result Test +Test - New Test Result Test +TPFP Test -FNTN Each cell represents a specific result. If this is the case, the new test works just as well as the gold standard.

Calculations Gold Standard Test Result Test +Test - New Test Result Test Test -525 Say these are the data we get after running the two HIV-1 detection tests

Calculations Gold Standard Test Result Test +Test - New Test Result Test Test : Sensitivity Sensitivity is the proportion of people with the disease (as measured by the gold standard) that test positive on the new test.

Calculations Gold Standard Test Result Test +Test - New Test Result Test +25 (TP)45 (FP) Test -5 (FN)25 (TN) 1: Sensitivity Formula is (TP / TP+FN) x 100

Calculations Gold Standard Test Result Test +Test - New Test Result Test +25 (TP)45 (FP) Test -5 (FN)25 (TN) 1: Sensitivity Sensitivity = 25 / (5+25) x 100 = 83%

Calculations 1: Sensitivity Sensitivity = 25 / (5+25) x 100 = 83% What does this mean?

Calculations 1: Sensitivity Sensitivity = 25 / (5+25) x 100 = 83% Of all patients with HIV-1, 83% of them will test positive with the new test. 17% will be false negative (100% - 83%)

Calculations Gold Standard Test Result Test +Test - New Test Result Test Test : Specificity Specificity is the proportion of people without the disease (as measured by the gold standard) that test negative on the new test.

Calculations Gold Standard Test Result Test +Test - New Test Result Test +25 (TP)45 (FP) Test -5 (FN)25 (TN) 2: Specificity Formula is (TN / TN+FP) x 100

Calculations Gold Standard Test Result Test +Test - New Test Result Test +25 (TP)45 (FP) Test -5 (FN)25 (TN) 2: Specificity Specificity = 25 / (25+45) x 100 = 36%

Calculations 2: Specificity What does this mean? Specificity = 25 / (25+45) x 100 = 36%

Calculations 2: Specificity Of all patients without HIV-1, 36% will test negative with the new test. Specificity = 25 / (25+45) x 100 = 36% 64% will be False Positive (100% - 36%)

Calculations Gold Standard Test Result Test +Test - New Test Result Test +25 (TP)45 (FP) Test -5 (FN)25 (TN) 3: Positive Predictive Value (PPV) PPV is the proportion of people that test positive with the new test of everyone that tests positive.

Calculations Gold Standard Test Result Test +Test - New Test Result Test +25 (TP)45 (FP) Test -5 (FN)25 (TN) 3: Positive Predictive Value (PPV) Formula is (TP / TP+FP) x 100

Calculations Gold Standard Test Result Test +Test - New Test Result Test +25 (TP)45 (FP) Test -5 (FN)25 (TN) 3: Positive Predictive Value (PPV) PPV = 25 / (25+45) x 100 = 36%

Calculations 3: PPV Of everyone who tests positive with the new test, 36% will actually have HIV-1. PPV = 25 / (25+45) x 100 = 36% 64% (100%-36%) Will be False Positives (test positive but do not have the disease)

Calculations Gold Standard Test Result Test +Test - New Test Result Test +25 (TP)45 (FP) Test -5 (FN)25 (TN) 4: Negative Predictive Value (NPV) NPV is the proportion of people that test negative with the new test of everyone that tests negative.

Calculations Gold Standard Test Result Test +Test - New Test Result Test +25 (TP)45 (FP) Test -5 (FN)25 (TN) 4: Negative Predictive Value (NPV) Formula is (TN / TN+FN) x 100

Calculations Gold Standard Test Result Test +Test - New Test Result Test +25 (TP)45 (FP) Test -5 (FN)25 (TN) 4: Negative Predictive Value (NPV) PPV = 25 / (25+5) x 100 = 83%

Calculations 4: NPV Of everyone who tests negative with the new test, 83% will not have HIV-1. NPV = 25 / (25+5) x 100 = 83% 13% (100%-83%) Will be False Negative (test negative but have the disease)

Overview 1. Understand the calculation of measures of diagnostic accuracy 2. Understand the clinical utility of measures of diagnostic accuracy 3. Review concepts related to receiver operating characteristic curves

False Positives and Negatives FP and FN are critically important No test is perfect. Must understand the ramifications of each

False Positives and Negatives FP and FN are critically important Unnecessary treatment (costs, side effects) Anxiety Social stigma Outbreak investigations Unnecessary interventions False Positives

False Positives and Negatives FP and FN are critically important Disease transmission (between and within) Delayed treatment initiation Poor outcomes False Negatives

False Positives and Negatives FP and FN are critically important Which one is more important depends on the situation and disease

Using Statistics in Real Life Sn, Sp, PPV, NPV Remember Sn, Sp tell you the proportion of people who test positive with the new test out of everyone with the disease.

Using Statistics in Real Life Sn, Sp, PPV, NPV Remember Sn, Sp tell you the proportion of people who test positive with the new test out of everyone with the disease. When do you know everyone who has the disease in your practice?

Using Statistics in Real Life SnOut and SpIn Sensitivity and Specificity can still be clinically useful. vs

Using Statistics in Real Life It all comes back to false positives and false negatives Gold Standard Test Result Test +Test - New Test Result Test +25 (TP)45 (FP) Test -5 (FN)25 (TN) Sensitivity (Sn) is concerned with FN TP / (TP + FN) Specificity (Sp) is concerned with FP TN / (TN + FP) SnOut and SpIn

Using Statistics in Real Life SnOut (Sn=sensitivity) How does one get a HIGH SENSITIVITY? Gold Standard Test Result Test +Test - New Test Result Test +25 (TP)45 (FP) Test -5 (FN)25 (TN) Must have VERY FEW False Negatives Few false negatives provides a number closer to 1 (100%) TP / (TP + FN)

Using Statistics in Real Life SnOut A test with a high sensitivity (sn) must have very few false negatives.

Using Statistics in Real Life SnOut A test with a high sensitivity (sn) must have very few false negatives. Therefore, if a highly sensitive test gives you a negative result, it must be a TRUE NEGATIVE (there are few false negatives).

Using Statistics in Real Life SnOut A negative result for a highly sensitive test helps you rule out the disease (SnOut) Oink!

Using Statistics in Real Life SpIn How does one get a HIGH SPECIFICITY? Gold Standard Test Result Test +Test - New Test Result Test +25 (TP)45 (FP) Test -5 (FN)25 (TN) Must have VERY FEW False Positives Few false positives provides a number closer to 1 (100%) TN / (TN + FP)

Using Statistics in Real Life SpIn A test with a high sensitivity (sp) must have very few false positives. Therefore, if a highly specific test gives you a positive result, it must be a TRUE POSITIVE (there are few false positives).

Using Statistics in Real Life SpIn A positive result for a highly specific test helps you rule in the disease (SpIn) I have nothing clever to say!

Using Statistics in Real Life Issues with PPV, NPV These statistics vary with the prevalence of the disease in your population

Using Statistics in Real Life Issues with PPV, NPV For example, if you look at PPV and NPV of a tuberculosis test, it will vary significantly if you are testing people in a homeless shelter versus employees at the University of Louisville.

Using Statistics in Real Life Issues with PPV, NPV Sensitivity and Specificity do not suffer from this fault – these values are “inherent to the test” and do not change.

Using Statistics in Real Life HIV-1 Rapid Test Gold Standard Test Result Test +Test - New Test Result Test +20 (TP)33 (FP) Test -10 (FN)37 (TN) Bathhouse: 30% Prevalence Gold Standard Test Result Test +Test - New Test Result Test +10 (TP)40 (FP) Test -5 (FN)45 (TN) CTRSC Course: 15% Prevalence Calculate Sensitivity and Specificity

Using Statistics in Real Life HIV-1 Rapid Test Gold Standard Test Result Test +Test - New Test Result Test +20 (TP)33 (FP) Test -10 (FN)37 (TN) Bathhouse: 30% Prevalence Gold Standard Test Result Test +Test - New Test Result Test +10 (TP)40 (FP) Test -5 (FN)45 (TN) CTRSC Course: 15% Prevalence Sensitivity = 67%, Specificity = 53%

Using Statistics in Real Life HIV-1 Rapid Test Gold Standard Test Result Test +Test - New Test Result Test +20 (TP)33 (FP) Test -10 (FN)37 (TN) Bathhouse: 30% Prevalence Gold Standard Test Result Test +Test - New Test Result Test +10 (TP)40 (FP) Test -5 (FN)45 (TN) CTRSC Course: 15% Prevalence Calculate PPV and NPV

Using Statistics in Real Life HIV-1 Rapid Test Gold Standard Test Result Test +Test - New Test Result Test +20 (TP)33 (FP) Test -10 (FN)37 (TN) Bathhouse: 30% Prevalence Gold Standard Test Result Test +Test - New Test Result Test +10 (TP)40 (FP) Test -5 (FN)45 (TN) CTRSC Course: 15% Prevalence PPV= 38%, NPV= 79%PPV= 20%, NPV= 90%

Using Statistics in Real Life Relationship between NPV, PPV, and Prevalence

Overview 1. Understand the calculation of measures of diagnostic accuracy 2. Understand the clinical utility of measures of diagnostic accuracy 3. Review concepts related to receiver operating characteristic curves

Receiver-Operating Characteristic Some tests have continuous data instead of yes/no Examples Pneumonia Severity Index (PSI) APACHE Scores Glasgow Coma Scale Sequential Organ Failure Assessment Score (SOFA) Simplified Acute Physiology Score (SAPS) Clinical Pulmonary Infection Score (CPIS)

Receiver-Operating Characteristic Some tests have continuous data instead of yes/no How do we identify cutoffs of values that give us the best sensitivity and specificity? E.g. what level of the PSI is best to use as a cutoff to predict mortality in a pneumonia patient?

Receiver-Operating Characteristic ROC Curves

Receiver-Operating Characteristic ROC Curves: Evaluation At a Sensitivity= 90% Expect 55% false positives e.g. Specificity = 45% (False Positive Rate)

Receiver-Operating Characteristic ROC Curves Area Under the Curve (AUC) = how good the test is More area = High Sensitivity and Specificity

Receiver-Operating Characteristic ROC Curves 45 degree angle = 50/50 guess

Receiver-Operating Characteristic Comparing Tests via ROC Curves

Receiver-Operating Characteristic Comparing Tests via ROC Curves

Receiver-Operating Characteristic Comparing Tests via ROC Curves

Overview 1. Understand the calculation of measures of diagnostic accuracy 2. Understand the clinical utility of measures of diagnostic accuracy 3. Review concepts related to receiver operating characteristic curves