Presentation on theme: "Performance of a diagnostic test"— Presentation transcript:
1 Performance of a diagnostic test Dagmar RimekEPIET-EUPHEM Introductory Course 2012Lazareto, Menorca, SpainBased on the Lecture of by Steen Ethelberg
2 Outline Performance characteristics of a test SensitivitySpecificityChoice of a thresholdPerformance of a test in a populationPositive predictive value of a test (PPV)Negative predictive value of a test (NPV)Impact of disease prevalence, sensitivity and specificity on predictive values
3 Performance characteristics of a test in a laboratory setting
4 Population with affected and non-affected individuals
5 A perfect diagnostic test identifies the affected individuals only Non-affected
6 In reality, tests are not perfect AffectedNon-affected
7 Proportion of persons testing positive among affected individuals Sensitivity of a testThe sensitivity of a test is the ability of the test to identify correctly the affected individualsProportion of persons testing positive among affected individualsAffected personsTest result+-True positive (TP)False negative (FN)Sensitivity (Se) = TP / (TP + FN)7
8 Estimating the sensitivity of a test Identify affected individuals with a gold standardObtain a wide panel of samples that are representative of the population of affected individualsRecent and old casesSevere and mild casesVarious ages and sexesTest the affected individualsEstimate the proportion of affected individuals that are positive with the test
9 Example: Estimating the sensitivity of a new ELISA IgM test for acute Q-fever Identify persons with acute Q-fever with a gold standard (IgM Immunofluorescence Assay)Obtain a wide panel of samples that are representative of the population of individuals with acute Q-feverRecent and old casesSevere and asymptomatic casesVarious ages and sexesTest the persons with acute Q-feverEstimate the proportion of persons with acute Q-fever that are positive with the ELISA IgM test
10 Example: Sensitivity a new ELISA IgM test for acute Q-fever Patients with acute Q-feverELISA IgM test result+True positive (TP)148-False negative (FN)2150Sensitivity =TP / (TP + FN) 148 / 150 = 98.7%10
11 What factors influence the sensitivity of a test? Characteristics of the affected persons?YES: Antigenic characteristics of the pathogen in the area (e.g., if the test was not prepared with antigens reflecting the population of pathogens in the area, it will not pick up infected persons in the area)Characteristics of the non-affected persons?NO: The sensitivity is estimated on a population of affected personsPrevalence of the disease?Sensitivity is an INTRINSIC characteristic of the test
12 Proportion of persons testing negative among non-affected individuals Specificity of a testThe specificity of a test is the ability of the test to identify correctly non-affected individualsProportion of persons testing negative among non-affected individualsNon-affected personsTest result+-False positive (FP)True negative (TN)Specificity (Sp) = TN / (TN + FP)12
13 Estimating the specificity of a test Identify non-affected individualsNegative with a gold standardUnlikely to be infectedObtain a wide panel of samples that are representative of the population of non-affected individualsTest the non-affected individualsEstimate the proportion of non-affected individuals that are negative with the test
14 Example: Estimating the specificity of a new ELISA IgM test for acute Q-fever Identify persons without Q-feverPersons without sign and symptoms of the infectionPersons at low risk of infection, negative with gold standard (IgM Immunofluorescence Assay)Obtain a wide panel of samples that are representative of the population of individuals without Q-feverTest the persons without Q-feverEstimate the proportion of persons without Q-fever that are negative with the new ELISA IgM test
15 Specificity of a new ELISA IgM test for acute Q-fever Persons without acute Q-feverELISA IgM test result+False positive (FP)10-True negative (TN)190200Specificity =TN / (TN + FP) 190 / 200 = 95%15
16 What factors influence the specificity of a test? Characteristics of the affected persons?NO: The specificity is estimated on a population of non- affected personsCharacteristics of the non-affected persons?YES: The diversity of antibodies to various other antigens in the population may affect cross reactivity or polyclonal hypergammaglobulinemia may increase the proportion of false positivesPrevalence of the disease?Specificity is an INTRINSIC characteristic of the test
17 + - Performance of a test Disease Test TP FN Yes FP TN No TP Se = Sp =TN + FP
18 To whom sensitivity and specificity matters most? INTRINSIC characteristics of the test► To laboratory specialists!
19 Distribution of quantitative test results among affected and non-affected people Ideal situationNon-affected:Threshold forpositive resultAffected:Number of people testedTNTPQuantitative result of the test
20 Distribution of quantitative results among affected and non-affected people Realistic situationNon-affected:Threshold forpositive resultAffected:TNTPNumber of people testedFNFPQuantitative result of the test
21 Effect of Decreasing the Threshold Non-affected:Threshold forpositive resultAffected:FPNumber of people testedTPTNFNQuantitative result of the test
22 Effect of Decreasing the Threshold DiseaseTestTPFNYes+-FPTNNoTPSe =TP + FNTNSp =TN + FP
23 Effect of Increasing the Threshold Non-affected:Threshold forpositive resultAffected:Number of people testedTNTPFNFPQuantitative result of the test
24 Effect of Increasing the Threshold DiseaseTestTPFNYes+-FPTNNoTPSe =TP + FNTNSp =TN + FP
25 Performance of a test and threshold Sensitivity and specificity vary in opposite directions when changing the threshold (e.g. the cut-off in an ELISA)The choice of a threshold is a compromise to best reach the objectives of the testconsequences of having false negatives?consequences of having false positives?
26 Using several testsOne way out of the dilemma is to use several tests that complement each otherFirst use test with a high sensitivity (e.g. screening for HIV by ELISA, or for syphilis by TPHA)Second use test with a high specificity (e.g. confirmation of HIV or syphilis by western blot)
27 ROC curves Receiver Operating Characteristics curve Representation of relationship between sensitivity and specificity for a testSimple tool to:Help define best cut-off value of a testCompare performance of two tests
28 Prevention of blood transfusion malaria: Choice of an indirect IFA threshold Sensitivity (%)1001/201/101/40801/801/16060IFA Dilutions1/320401/6402020406080100100 - Specificity (%): Proportion of false positives
29 Comparison of performance of IFA and ELISA IgM tests for detection of acute Q-fever Sensitivity (%)10080IFAELISA6040Area under the ROC curve (AUC)20255075100100 - Specificity (%)
31 How well does the test perform in a real population? The test is now used in a real populationThis population is made ofAffected individualsNon-affected individualsThe proportion of affected individuals is the prevalenceStatus of personsAffectedNon-affectedTestPositiveTrue +False +A+BNegativeFalse -True -C+DA+CB+DA+C+B+D
32 Predictive value of a positive test The predictive value of a positive test is the probability that an individual testing positive is truly affectedProportion of affected persons among those testing positive
33 Positive predictive value (PPV) of a test Status of personsAffectedNon-affectedTestPositiveABA+BNegativeCDC+DA + CB+DA+C+B+DPPV = A / (A+B)This is only valid for the sample of specimens tested33
34 What factors influence the positive predictive value of a test? Status of personsAffectedNon-affectedTestPositiveABA+BNegativeCDC+DA + CB+DA+C+B+DSensitivity?YES: To some extend.Specificity?YES: The more the test is specific, the more it will be negative for non-affected persons (less false-positive results).Prevalence of the disease?YES: Low prevalence: Low pre-test probability for positives. The test will pick up more false positives.YES: High prevalence: High pre-test probability for positives. The test will pick up more true positives.
35 Positive predictive value of a test according to prevalence and specificity PPV (%)
36 Predictive value of a negative test The predictive value of a negative test is the probability that an individual testing negative is truly non-affectedProportion of non-affected persons among those testing negative
37 Negative predictive value (NPV) of a test Status of personsAffectedNon-affectedTestPositiveABA+BNegativeCDC+DA+CB+DA+C+B+DNPV = D / (C+D)This is only valid for the sample of specimens tested37
38 What factors influence the negative predictive value of a test? Status of personsAffectedNon-affectedTestPositiveABA+BNegativeCDC+DA+CB+DA+C+B+DSensitivity?YES: The more the test is sensitive, the more it captures affected persons (less false negatives).Specificity?YES: But to a lesser extend.Prevalence of the disease?YES: Low prevalence: High pre-test probability for negatives. The test will pick up more true negatives.YES: High prevalence: Low pre-test probability for negatives. The test will pick up more false negatives.
39 Negative predictive value of a test according to prevalence and sensitivity NPV (%)
40 Relation between predictive values and sensitivity (Se), specificity (Sp), prevalence (Pr) (1-Se)Pr + Sp(1-Pr)Disease(1-Sp)(1-Pr)Se PrNoYesSe Pr + (1-Sp)(1-Pr)Pr1-PrSp(1-Pr)(1-Se)PrTest+-
41 Calculate PPV and NPV Pr) Sp)(1 (1 Pr Se PPV - + = Pr Se) (1 Pr) -
42 Relation between predictive values and sensitivity / specificity (1PrSePPV-+=Increasing specificity increasing PPVPrSe)(1Pr)-Sp(1NPV+=Increasing sensitivity increasing NPV
43 Relation between predictive values and prevalence Sp)(1(1PrSePPV-+=Increasing prevalence increasing PPVPrSe)(1Pr)-Sp(1NPV+=Decreasing prevalence increasing NPV
44 Example: Screening for acute Q-fever in two settings ELISA IgM testSensitivity = 98%Specificity = 95%Population in low endemic areaPrevalence = 0.5%Patients with atypical pneumoniaPrevalence = 20%10,000 tests performed in each group
45 Example: Screening for acute Q-fever in a population in a low endemic area IgM ELISA test sensitivity = 98%IgM ELISA test specificity = 95%Prevalence = 0.5%Q-feverYesNoTotalIgM ELISA+49497546-19,4539,454509,95010,000PPV = %NPV = %
46 Example: Screening for acute Q-fever in patients with atypical pneumonia IgM ELISA test sensitivity = 98%IgM ELISA test specificity = 95%Prevalence = 20%Q-feverYesNoTotalIgM ELISA+1,9604002,360-407,6007,6402,0008,00010,000PPV = %NPV = %
47 To whom predictive values matters most? Look at denominators!Persons testing positivePersons testing negative► To cliniciansprobability that a individual with a positive test is really sick?probability that a individual with a negative test is really healthy?► To epidemiologists!proportion of positive tests corresponding to true patients?proportion of negative tests corresponding to healthy subjects?
48 Summary Sensitivity and specificity matter to laboratory specialists Studied on panels of positives and negativesIntrinsic characteristics of a testCapacity to identify the affectedCapacity to identify the non-affectedPredictive values matter to clinicians and epidemiologistsStudied on homogeneous populationsDependent on the disease prevalencePerformance of a test in real lifeHow to interpret a positive testHow to interpret a negative test
Your consent to our cookies if you continue to use this website.