Presentation is loading. Please wait.

Presentation is loading. Please wait.

1 Performance of a diagnostic test Based on the Lecture of 2011 by Steen Ethelberg Dagmar Rimek EPIET-EUPHEM Introductory Course 2012 Lazareto, Menorca,

Similar presentations


Presentation on theme: "1 Performance of a diagnostic test Based on the Lecture of 2011 by Steen Ethelberg Dagmar Rimek EPIET-EUPHEM Introductory Course 2012 Lazareto, Menorca,"— Presentation transcript:

1 1 Performance of a diagnostic test Based on the Lecture of 2011 by Steen Ethelberg Dagmar Rimek EPIET-EUPHEM Introductory Course 2012 Lazareto, Menorca, Spain

2 2 Outline 1.Performance characteristics of a test –Sensitivity –Specificity –Choice of a threshold 2.Performance of a test in a population –Positive predictive value of a test (PPV) –Negative predictive value of a test (NPV) –Impact of disease prevalence, sensitivity and specificity on predictive values

3 3 1.Performance characteristics of a test in a laboratory setting

4 4 Population with affected and non-affected individuals Affected Non-affected

5 5 A perfect diagnostic test identifies the affected individuals only Affected Non-affected

6 6 In reality, tests are not perfect Affected Non-affected

7 Sensitivity of a test The sensitivity of a test is the ability of the test to identify correctly the affected individuals Proportion of persons testing positive among affected individuals Affected persons Test result +-+- True positive (TP) False negative (FN) Sensitivity (Se) = TP / (TP + FN) 7

8 Estimating the sensitivity of a test Identify affected individuals with a gold standard Obtain a wide panel of samples that are representative of the population of affected individuals –Recent and old cases –Severe and mild cases –Various ages and sexes Test the affected individuals Estimate the proportion of affected individuals that are positive with the test 8

9 Example: Estimating the sensitivity of a new ELISA IgM test for acute Q-fever Identify persons with acute Q-fever with a gold standard (IgM Immunofluorescence Assay) Obtain a wide panel of samples that are representative of the population of individuals with acute Q-fever –Recent and old cases –Severe and asymptomatic cases –Various ages and sexes Test the persons with acute Q-fever Estimate the proportion of persons with acute Q-fever that are positive with the ELISA IgM test 9

10 Example: Sensitivity a new ELISA IgM test for acute Q-fever Patients with acute Q-fever ELISA IgM test result + True positive (TP)148 - False negative (FN)2 150 Sensitivity = TP / (TP + FN) 148 / 150 = 98.7% 10

11 What factors influence the sensitivity of a test? Characteristics of the affected persons? YES: Antigenic characteristics of the pathogen in the area (e.g., if the test was not prepared with antigens reflecting the population of pathogens in the area, it will not pick up infected persons in the area) Characteristics of the non-affected persons? NO: The sensitivity is estimated on a population of affected persons Prevalence of the disease? NO: The sensitivity is estimated on a population of affected persons Sensitivity is an INTRINSIC characteristic of the test 11

12 Specificity of a test Specificity (Sp) = TN / (TN + FP) The specificity of a test is the ability of the test to identify correctly non-affected individuals Proportion of persons testing negative among non-affected individuals 12 Non-affected persons Test result +-+- False positive (FP) True negative (TN)

13 Estimating the specificity of a test Identify non-affected individuals –Negative with a gold standard –Unlikely to be infected Obtain a wide panel of samples that are representative of the population of non-affected individuals Test the non-affected individuals Estimate the proportion of non-affected individuals that are negative with the test 13

14 Example: Estimating the specificity of a new ELISA IgM test for acute Q-fever Identify persons without Q-fever –Persons without sign and symptoms of the infection –Persons at low risk of infection, negative with gold standard (IgM Immunofluorescence Assay) Obtain a wide panel of samples that are representative of the population of individuals without Q-fever Test the persons without Q-fever Estimate the proportion of persons without Q-fever that are negative with the new ELISA IgM test 14

15 Specificity of a new ELISA IgM test for acute Q-fever Persons without acute Q-fever ELISA IgM test result + False positive (FP)10 - True negative (TN) Specificity = TN / (TN + FP) 190 / 200 = 95% 15

16 What factors influence the specificity of a test? Characteristics of the affected persons? NO: The specificity is estimated on a population of non- affected persons Characteristics of the non-affected persons? YES: The diversity of antibodies to various other antigens in the population may affect cross reactivity or polyclonal hypergammaglobulinemia may increase the proportion of false positives Prevalence of the disease? NO: The specificity is estimated on a population of non- affected persons Specificity is an INTRINSIC characteristic of the test 16

17 17 Disease TN Sp = TN + FP Performance of a test FP TN No ­ TP Se = TP + FN Test TP FN Yes + -

18 To whom sensitivity and specificity matters most? INTRINSIC characteristics of the test To laboratory specialists! 18

19 Quantitative result of the test Distribution of quantitative test results among affected and non-affected people TN Non-affected: Affected: TP Number of people tested Threshold for positive result Ideal situation

20 TNTP FNFP Distribution of quantitative results among affected and non-affected people Non-affected: Threshold for positive result Quantitative result of the test Number of people tested Affected: Realistic situation

21 21 TN TP FN FP Non-affected: Affected: Threshold for positive result Effect of Decreasing the Threshold Number of people tested Quantitative result of the test

22 22 Disease FP TN No Effect of Decreasing the Threshold ­ TN Sp = TN + FP TP Se = TP + FN Test TP FN Yes + -

23 TN TP FN FP Non-affected: Affected: Threshold for positive result Number of people tested Quantitative result of the test Effect of Increasing the Threshold

24 24 Effect of Increasing the Threshold Disease FP TN No ­ TN Sp = TN + FP TP Se = TP + FN Test TP FN Yes + -

25 25 Performance of a test and threshold Sensitivity and specificity vary in opposite directions when changing the threshold (e.g. the cut-off in an ELISA) The choice of a threshold is a compromise to best reach the objectives of the test –consequences of having false negatives? –consequences of having false positives?

26 26 Using several tests One way out of the dilemma is to use several tests that complement each other First use test with a high sensitivity (e.g. screening for HIV by ELISA, or for syphilis by TPHA) Second use test with a high specificity (e.g. confirmation of HIV or syphilis by western blot)

27 27 ROC curves Receiver Operating Characteristics curve Representation of relationship between sensitivity and specificity for a test Simple tool to: –Help define best cut-off value of a test –Compare performance of two tests

28 28 Prevention of blood transfusion malaria: Choice of an indirect IFA threshold IFA Dilutions 1/10 1/20 1/40 1/80 1/160 1/320 1/ Specificity (%): Proportion of false positives Sensitivity (%)

29 Comparison of performance of IFA and ELISA IgM tests for detection of acute Q-fever IFA ELISA Specificity (%) Sensitivity (%) Area under the ROC curve (AUC)

30 30 2.Performance of a test in a population

31 How well does the test perform in a real population? Status of persons AffectedNon-affected Test PositiveTrue +False +A+B NegativeFalse -True -C+D A+CB+DA+C+B+D The test is now used in a real population This population is made of –Affected individuals –Non-affected individuals The proportion of affected individuals is the prevalence 31

32 Predictive value of a positive test The predictive value of a positive test is the probability that an individual testing positive is truly affected Proportion of affected persons among those testing positive 32

33 Positive predictive value (PPV) of a test Status of persons AffectedNon- affected Test PositiveABA+B NegativeCDC+D A + CB+DA+C+B+D PPV = A / (A+B) This is only valid for the sample of specimens tested 33

34 What factors influence the positive predictive value of a test? Sensitivity? YES: To some extend. Specificity? YES: The more the test is specific, the more it will be negative for non-affected persons (less false-positive results). Prevalence of the disease? YES: Low prevalence: Low pre-test probability for positives. The test will pick up more false positives. YES: High prevalence: High pre-test probability for positives. The test will pick up more true positives. Status of persons AffectedNon- affected Test PositiveABA+B NegativeCDC+D A + CB+DA+C+B+D 34

35 Positive predictive value of a test according to prevalence and specificity Specificity PPV (%) 35

36 Predictive value of a negative test The predictive value of a negative test is the probability that an individual testing negative is truly non-affected Proportion of non-affected persons among those testing negative 36

37 Negative predictive value (NPV) of a test Status of persons AffectedNon- affected Test PositiveABA+B NegativeCDC+D A+CB+DA+C+B+D NPV = D / (C+D) This is only valid for the sample of specimens tested 37

38 What factors influence the negative predictive value of a test? Sensitivity? YES: The more the test is sensitive, the more it captures affected persons (less false negatives). Specificity? YES: But to a lesser extend. Prevalence of the disease? YES: Low prevalence: High pre-test probability for negatives. The test will pick up more true negatives. YES: High prevalence: Low pre-test probability for negatives. The test will pick up more false negatives. 38 Status of persons AffectedNon- affected Test PositiveABA+B NegativeCDC+D A+CB+DA+C+B+D

39 Negative predictive value of a test according to prevalence and sensitivity Sensitivity NPV (%) 39

40 Relation between predictive values and sensitivity (Se), specificity (Sp), prevalence (Pr) (1-Se)Pr + Sp(1-Pr) Disease (1-Sp)(1-Pr) Se Pr NoYes Se Pr + (1-Sp)(1-Pr) Pr1-Pr Sp(1-Pr)(1-Se)Pr Test

41 41 Calculate PPV and NPV Pr) Sp)(1 (1 Pr Se Pr Se PPV Pr Se)(1Pr)-Sp(1 Pr)-Sp(1 NPV

42 42 Relation between predictive values and sensitivity / specificity Increasing specificity increasing PPV Increasing sensitivity increasing NPV Pr) Sp)(1 (1 Pr Se Pr Se PPV Pr Se)(1Pr)-Sp(1 Pr)-Sp(1 NPV

43 43 Relation between predictive values and prevalence Increasing prevalence increasing PPV Decreasing prevalence increasing NPV Pr) Sp)(1 (1 Pr Se Pr Se PPV Pr Se)(1Pr)-Sp(1 Pr)-Sp(1 NPV

44 44 ELISA IgM test –Sensitivity = 98% –Specificity = 95% Population in low endemic area –Prevalence = 0.5% Patients with atypical pneumonia –Prevalence = 20% 10,000 tests performed in each group Example: Screening for acute Q-fever in two settings

45 45 Example: Screening for acute Q-fever in a population in a low endemic area Prevalence = 0.5% PPV = 8.97% NPV = 99.98% IgM ELISA test sensitivity = 98% IgM ELISA test specificity = 95% Q-fever YesNoTotal IgM ELISA ­- 19,4539, ,95010,000

46 46 Example: Screening for acute Q-fever in patients with atypical pneumonia Prevalence = 20% PPV = 83.05% NPV = 99.48% IgM ELISA test sensitivity = 98% IgM ELISA test specificity = 95% Q-fever YesNoTotal IgM ELISA +1, ,360 ­- 407,6007,640 2,0008,00010,000

47 To whom predictive values matters most? Look at denominators! –Persons testing positive –Persons testing negative To clinicians – probability that a individual with a positive test is really sick? – probability that a individual with a negative test is really healthy? To epidemiologists! – proportion of positive tests corresponding to true patients? – proportion of negative tests corresponding to healthy subjects? 47

48 48 Sensitivity and specificity matter to laboratory specialists – Studied on panels of positives and negatives – Intrinsic characteristics of a test Capacity to identify the affected Capacity to identify the non-affected Predictive values matter to clinicians and epidemiologists – Studied on homogeneous populations – Dependent on the disease prevalence – Performance of a test in real life How to interpret a positive test How to interpret a negative test Summary

49 Where will you do your rain dance? Here? There?


Download ppt "1 Performance of a diagnostic test Based on the Lecture of 2011 by Steen Ethelberg Dagmar Rimek EPIET-EUPHEM Introductory Course 2012 Lazareto, Menorca,"

Similar presentations


Ads by Google