1. Performance of a diagnostic test Tunisia, 31 Oct 2014
Acknowledgments :Mia Brytting and Georgia Ladbury
EPIET/EUPHEM Introductory Course 2014
Prof Enver Roshi
Faculty of Public Health, University of Medicine, Tirane- Albania

2. Outline Performance characteristics of a test
Sensitivity
Specificity
Choice of a threshold
Performance of a test in a population
Positive predictive value of a test (PPV)
Negative predictive value of a test (NPV)
Impact of disease prevalence, sensitivity and specificity on predictive values

3. Performance characteristics of a test in a laboratory setting

4. Population with affected and non-affected individuals

5. A perfect diagnostic test identifies the affected individuals only
Non-affected

6. In reality, tests are not perfect
Affected
Non-affected

7. Diagnostic sensitivity of a test
The sensitivity of a test is the ability of the test to identify correctly the affected individuals
Proportion of persons testing positive among affected individuals
Affected persons
Test result + -
True positive (TP)
False negative (FN)
Sensitivity (Se) = TP / (TP + FN) 7

8. Estimating the sensitivity of a test
Identify affected individuals with a gold standard
Obtain a wide panel of samples that are representative of the population of affected individuals
Recent and old cases
Severe and mild cases
Various ages and sexes
Test the affected individuals
Estimate the proportion of affected individuals that are positive with the test

9. Example: Estimating the sensitivity of a new ELISA IgM test for acute Q fever
Identify persons with acute Q fever with a golden standard (IgM Immunofluorescence Assay)
Obtain a wide panel of samples that are representative of the population of individuals with acute Q-fever
Recent
Severe and asymptomatic cases
Various ages and sexes
Test the persons with acute Q-fever
Estimate the proportion of persons with acute Q-fever that are positive with the ELISA IgM test

10. Example: Sensitivity a new ELISA IgM test for acute Q-fever
Patients with acute Q-fever
ELISA IgM test result +
True positive (TP)
148 -
False negative (FN) 2
150
Sensitivity = TP / (TP + FN) / 150 = 98.7% 10

11. What factors influence the sensitivity of a test?
Characteristics of the affected persons?
YES: Antigenic characteristics of the pathogen in the area (e.g., if the test was not prepared with antigens reflecting the population of pathogens in the area, it will not pick up infected persons in the area)
Characteristics of the non-affected persons?
NO: The sensitivity is estimated on a population of affected persons
Prevalence of the disease?
Sensitivity is an INTRINSIC characteristic of the test 11

12. Diagnostic specificity of a test
The specificity of a test is the ability of the test to identify correctly non-affected individuals
Proportion of persons testing negative among non-affected individuals
Non-affected persons
Test result + -
False positive (FP)
True negative (TN)
Specificity (Sp) = TN / (TN + FP) 12

13. Estimating the specificity of a test
Identify non-affected individuals
Negative with a gold standard
Unlikely to be infected
Obtain a wide panel of samples that are representative of the population of non-affected individuals
Test the non-affected individuals
Estimate the proportion of non-affected individuals that are negative with the test 13

14. Example: Estimating the specificity of a new ELISA IgM test for acute Q-fever
Identify persons without Q-fever
Persons without sign and symptoms of the infection
Persons at low risk of infection, negative with gold standard (IgM Immunofluorescence Assay)
Obtain a wide panel of samples that are representative of the population of individuals without Q-fever
Test the persons without Q-fever
Estimate the proportion of persons without Q-fever that are negative with the new ELISA IgM test 14

15. Specificity of a new ELISA IgM test for acute Q-fever
Patients with acute Q-fever
ELISA IgM test result +
False positive (TP) 10 -
True negative (TN)
190
200
Specificity = TN / (TN + FP) / 200 = 95% 15

16. What factors influence the specificity of a test?
Characteristics of the affected persons?
NO: The specificity is estimated on a population of non-affected persons
Characteristics of the non-affected persons?
YES: The diversity of antibodies to various other antigens in the population may affect cross reactivity or polyclonal hypergammaglobulinemia may increase the proportion of false positives
Prevalence of the disease?
Specificity is an INTRINSIC characteristic of the test 16

17. Performance of a test + - ­ Disease Test TP FN Yes FP TN No TP Se =
Sp =
TN + FP 17

18. To whom sensitivity and specificity matters most?
INTRINSIC characteristics of the test
► To laboratory specialists! 18

19. Distribution of quantitative test results among affected and non-affected people
Ideal situation
Non-affected:
Threshold for
positive result
Affected:
Number of people tested TN TP 19
Quantitative result of the test

20. Distribution of quantitative results among affected and non-affected people
Realistic situation
Non-affected:
Threshold for
positive result
Affected: TN TP
Number of people tested FN FP
Quantitative result of the test 20

21. Effect of Decreasing the Threshold
Non-affected:
Threshold for
positive result
Affected: FP
Number of people tested TP TN FN
Quantitative result of the test 21

22. Effect of Decreasing the Threshold
Disease
Test TP FN
Yes + - FP TN No TP
Se =
TP + FN TN
Sp =
TN + FP 22

23. Effect of Increasing the Threshold
Non-affected:
Threshold for
positive result
Affected:
Number of people tested TN TP FN FP
Quantitative result of the test 23

24. Effect of Increasing the Threshold
Disease
Test TP FN
Yes + - FP TN No TP
Se =
TP + FN TN
Sp =
TN + FP 24

25. Performance of a test and threshold
Sensitivity and specificity vary in opposite directions when changing the threshold (e.g. the cut-off in an ELISA)
The choice of a threshold is a compromise to best reach the objectives of the test
consequences of having false negatives?
consequences of having false positives? 25

26. When false diagnosis is worse than missed diagnosis
Example: Screening for congenital toxoplasmosis
One should minimise false positives
Prioritise SPECIFICITY 26

27. When missed diagnosis is worse than false diagnosis
Example: Testing for Helicobacter pylori infection
One should minimise the false negatives
Prioritise SENSITIVITY 27

28. Using several tests
One way out of the dilemma is to use several tests that complement each other
First use test with a high sensitivity (e.g. screening for HIV by ELISA, or for syphilis by TPHA)
Second use test with a high specificity (e.g. confirmation of HIV or syphilis by western blot) 28

29. Performance of a test Validity Reproducibility Sensitivity Specificity29

30. 2. Performance of a test in a population

31. Would like to know… As a clinician
probability that a individual with a positive test is really sick?
probability that a individual with a negative test is really healthy?
As an epidemiologist/PH microbiologist
proportion of positive tests corresponding to true patients?
proportion of negative tests corresponding to healthy subjects? 31

32. Positive Predictive Value (PPV)
Probability that an individual testing positive is truly affected
proportion of affected people among those testing positive
Disease
Yes No +
Test TP FP
PPV = TP/(TP+FP) 32

33. Negative Predictive Value (NPV)
Probability that an individual testing negative is truly unaffected
proportion of non affected among those testing negative
Disease
Yes No -
Test FN TN
NPV = TN/(TN+FN) 33

34. What factors influence the predictive values of a test?
Predicted values are not constant, they vary between different populations
Sensitivity
Specificity
Prevalence 34

35. How PPV, NPV, Se, Sp and Pr are inter-related
Disease
Yes No FP TP +
PPV = TP/(TP+FP)
Test - FN TN
NPV = TN/(TN+FN) 35

36. How PPV, NPV, Se, Sp and Pr are inter-related (cont.)
Disease
Yes No +
Se Pr + (1-Sp)(1-Pr)
Se Pr
PPV=
Se Pr
(1-Sp)(1-Pr)
Test -
(1-Se)Pr+ Sp(1-Pr)
Sp(1-Pr)
NPV=
(1-Se)Pr
Sp(1-Pr) Pr
1-Pr 36

37. Relation between predictive values and sensitivity / specificity(1 Pr Se
PPV - + =
Increasing specificity  increasing PPV Pr
Se) (1
Pr) -
Sp(1
NPV + =
Increasing sensitivity  increasing NPV 37

38. Relation between predictive values and prevalence
Sp)(1 (1 Pr Se
PPV - + =
Increasing prevalence  increasing PPV Pr
Se) (1
Pr) -
Sp(1
NPV + =
Decreasing prevalence  increasing NPV 38

39. Example: Testing for acute Q-fever in two settings
ELISA IgM test
Sensitivity = 98%
Specificity = 95%
Population in low endemic area
Prevalence = 0.5%
Patients with atypical pneumonia
Prevalence = 20%
10,000 tests performed in each group 39

40. Example: Testing for acute Q-fever in a population in a low endemic area
Prevalence = 0.5%
IgM ELISA test sensitivity = 98%
IgM ELISA test specificity = 95%
Q-fever
Yes No
Total
IgM ELISA + 49
497
546 ­- 1
9,453
9,454 50
9,950
10,000
PPV = %
NPV = % 40

41. Example: Testing for acute Q-fever in patients with atypical pneumonia
IgM ELISA test sensitivity = 98%
IgM ELISA test specificity = 95%
Prevalence = 20%
Q-fever
Yes No
Total
IgM ELISA +
1,960
400
2,360 ­- 40
7,600
7,640
2,000
8,000
10,000
PPV = %
NPV = % 41

42. What happens in an outbreak situation?
Sensitivity
Specificity
PPV
NPV 42

43. Summary Sensitivity and specificity matter to laboratory specialists
Intrinsic characteristics of a test
Se: capacity to identify sick people as sick
Sp: capacity to identify healthy people as healthy
Predictive values matter to clinicians, epidemiologists and PH microbiologists
Dependent on the disease prevalence
Performance of a test in a real-life population
PPV: how to interpret a positive test
NPV: how to interpret a negative test 43

44. Any questions? Thank you!
Prof Enver Roshi
Faculty of Public Health, University of Medicine, Tirane- Albania