1. Assessing Information from Multilevel (Ordinal) and Continuous Tests ROC curves and Likelihood Ratios for results other than “+” or “-” Michael A. Kohn, MD, MPP 10/6/2005

2. Four Main Points 1) Dichotomizing a multi-level or continuous test by choosing a fixed cutpoint reduces the value of the test. 2) The ROC curve summarizes the discriminatory ability of the test. 3) LR(result) = P(result|D+)/P(result|D-) = slope of ROC curve. 4) Pre-Test Odds x LR(result) = Post-Test Odds

3. Many Tests Are Not Dichotomous Ordinal “-”, “+”, “++”, “+++” for leukocyte esterase on urine dip stick “Normal”, “Low Prob”, “Intermediate Prob”, “High Prob” on VQ scan Continuous Systolic Blood Pressure WBC Count

4. Evaluating the Test --Test Characteristics For dichotomous tests, we discussed sensitivity P(+|D+) and specificity P(-|D-) For multi-level and continuous tests, we will discuss the Receiver Operating Characteristic (ROC) curve

5. Using the Test Result to Make Decisions about a Patient For dichotomous tests, we use the LR(+) if the test is positive and the LR(-) if the test is negative For multilevel and continuous tests, we use the LR(r), where r is the result of the test

6. Clinical Scenario 5-month old boy with fever 39.7. You have the results of a WBC count. How do you use this WBC result to determine whether to treat empirically for possible bacteremia?

7. Why Not Make It a Dichotomous Test? WBC Count (x1000/uL)BacteremiaNo Bacteremia >151092028 0 -14.99186601 Total1278629 Lee GM, Harper MB. Risk of bacteremia for febrile young children in the post- Haemophilus influenzae type b era. Arch Pediatr Adolesc Med. 1998;152(7):624-628.

8. Why Not Make It a Dichotomous Test? Sensitivity = 109/127 = 0.86 Specificity = 6601/8629 = 0.76 LR(+) = 0.86/(1 - 0.76) = 3.65 LR(-) = (1 - 0.86)/0.76 = 0.19 Equivalently LR(+) = P(+|D+)/P(+|D-) = (109/127)/(2028/8629) = 3.65 LR(-) = P(-|D+)/P(-|D-) = (18/127)/(6601/8629) = 0.19

9. Clinical Scenario WBC = 16,000/mL (Demonstrate LR Slide Rule?) Pre-test prob: 0.03 LR(+) = 3.65 Post-Test prob = ?

10. Clinical Scenario WBC = 16,000/mL Pre-test prob: 0.03 Pre-test odds: 0.03/0.97 = 0.031 LR(+) = 3.65 Post-Test Odds = Pre-Test Odds x LR(+) = 0.031 x 3.65 =.113 Post-Test prob =.113/(.113+1) =.10

11. Clinical Scenario WBC = 28,000/mL Pre-test prob: 0.03 LR(+) = ? Post-Test prob =?

12. Clinical Scenario WBC = 28,000/mL Pre-test prob: 0.03 Pre-test odds: 0.03/0.97 = 0.031 LR(+) = 3.65 (same as for WBC=16,000!) Post-Test Odds = Pre-Test Odds x LR(+) = 0.031 x 3.65 =.113 Post-Test prob =.113/(.113+1) =.10

13. Why Not Make It a Dichotomous Test? Because you lose information. The risk associated with WBC=16,000 is equated with the risk associated with WBC=28,000. Choosing a fixed cutpoint to dichotomize a multi-level or continuous test throws away information and reduces the value of the test.

14. Main Point 1: Avoid Making Multilevel Tests Dichotomous Dichotomizing a multi-level or continuous test by choosing a fixed cutpoint reduces the value of the test

15. WBC Count (x1000/uL) BacteremiaNo Bacteremia 30 - 351567 25 - <3012155 20 - <2534469 15 - <20481337 10 - <15152767 5 - <1033291 0 - <50543 TOTAL1278629 Lee GM, Harper MB. Risk of bacteremia for febrile young children in the post- Haemophilus influenzae type b era. Arch Pediatr Adolesc Med. 1998;152(7):624-628.

16. WBC Count (x1000/uL) BacteremiaNo Bacteremia 30 - 3511.8%0.8% 25 - <309.4%1.8% 20 - <2526.8%5.4% 15 - <2037.8%15.5% 10 - <1511.8%32.1% 5 - <102.4%38.1% 0 - <50.0%6.3% TOTAL100%

18. Histogram Does not reflect prevalence of D+ (Dark D+ columns add to 100%, Open D- columns add to 100%) Sensitivity and specificity depend on the cutpoint chosen to separate “positives” from “negatives” The ROC curve is drawn by serially lowering the cutpoint from highest (most abnormal) to lowest (least abnormal).* * Just said that choosing a fixed cutpoint reduces the value of the test. The key issues are 1) the ROC curve is for evaluating the test, not the patient, and 2) drawing the ROC curve requires varying the cutpoint, not choosing a fixed cutpoint.

27. WBC Count (x1000/uL) Sensitivity1 - Specificity > 35 0% > 3011.8%0.8% > 2521.3%2.6% > 2048.0%8.0% > 1585.8%23.5% > 1097.6%55.6% > 5100%93.7% > 0100%

28. Area Under Curve (AUC) = 0.86 30,000/uL 25,000/uL 20,000/uL 15,000/uL 10,000/uL 5,000/uL

33. WBC Cutoff = 12,000/μL

34. Test Discriminates Well Between D+ and D- Test Result D- D+

35. Test Discriminates Well Between D+ and D-

36. Test Discriminates Poorly Between D+ and D- Test Result D- D+

37. Test Discriminates Poorly Between D+ and D-

38. Area Under Curve (AUC) = 0.86 30,000/uL 25,000/uL 20,000/uL 15,000/uL 10,000/uL 5,000/uL Area Under ROC Curve

39. Summary measure of test’s discriminatory ability Probability that a randomly chosen D+ individual will have a more positive test result than a randomly chosen D- individual e.g. randomly choose 1 of the 127 bacteremic children and 1 of the 8629 non-bacteremic children. The probability that the bacteremic child’s WBC will fall in a higher WBC interval than the non-bacteremic child is 0.86

40. Area Under ROC Curve Corresponds to the Mann-Whitney (Wilcoxan Rank Sum) Test Statistic, which is the non-parametric equivalent of Student’s t test. Also corresponds to the “c statistic” reported in logistic regression models

42. “Walking Man” Approach to ROC Curves Divide vertical axis into d steps, where d is the number of D+ individuals Divide horizontal axis into n steps, where n is the number of D- individuals Sort individuals from most to least abnormal test result Moving from the first individual (with the most abnormal test result) to the last (with the least abnormal test result)…

43. “Walking Man” (continued) …call out “D” if the individual is D+ and “N” if the individual is D- Let the walking man know when you reach a new value of the test The walking man takes a step up every time he hears “D” and a step to the right every time he hears “N” When you reach a new value of the test, he drops a stone.

44. WBC Count in 5 Bacteremic Children PatientWBC Count D127 D222 D319 D417 D514

45. WBC Count in 10 Non-Bacteremic Children PatientWBC Count N121 N218 N317 N413 N512 N612 N78 N87 N96 N104

46. BACTEREMIANO BACTEREMIA 27 22 21 19 18 17 14 13 12 8 7 6 4

47. DDNDN(DN)DN(NN)NNNN

49. Main Point 2 ROC Curve Describes the Test, Not the Patient Describes the test’s ability to discriminate between D+ and D- individuals Not particularly useful in interpreting a test result for a given patient

50. ROC Curve Describes the Test, Not the Patient Clinical Scenario WBC count = 16,000 WBC count = 28,000

51. Febrile Child with WBC count = 16,000

52. Area Under Curve (AUC) = 0.86 30,000/uL 25,000/uL 20,000/uL 15,000/uL 10,000/uL 5,000/uL

53. WBC Count (x1000/uL) Sensitivity1 - Specificity > 35 0% > 3011.8%0.8% > 2521.3%2.6% > 2048.0%8.0% > 1585.8%23.5% > 1097.6%55.6% > 5100%93.7% > 0100%

54. Likelihood Ratios LR(+) = Sensitivity/(1 – Specificity) = P(+|D+)/(1-P(-|D-)) = P(+|D+)/P(+|D-) LR(-) = (1 – Sensitivity)/Specificity = (1-P(+|D+))/P(-|D-) = P(-|D+)/P(-|D-)

55. Likelihood Ratios LR(result) = P(result|D+)/P(result|D-) P(Result) in patient WITH disease ------------------------------------------------------ P(Result) in patients WITHOUT disease

56. Likelihood Ratios The ratio of the height of the D+ distribution to the height of the D- distribution 37.8% 15.5% LR = 37.8%/15.5% = 2.4

57. 30,000/uL 25,000/uL 20,000/uL 15,000/uL 10,000/uL 5,000/uL Likelihood Ratio = Slope of ROC Cuve 37.8% 15.5% Slope = 37.8%/15.5% = 2.4

58. Likelihood Ratio WBC Count (x1000/uL) BacteremiaNo BacteremiaLR 30 - <3511.8%0.8%15.2 25 - <309.4%1.8%5.3 20 - <2526.8%5.4%4.9 15 - <2037.8%15.5%2.4 10 - <1511.8%32.1%0.37 5 - <102.4%38.1%0.06 0 - <50.0%6.3%0.00

59. Common Mistake When given an “ROC Table,” it is tempting to calculate an LR(+) or LR(-) as if the test were “dichotomized” at a particular cutoff. Example: LR(+,10,000) = 97.6/55.6 = 1.8 This is NOT the LR of a particular result (e.g. WBC >10,000 and <15,000); it is the LR(+) if you divide “+” from “-” at 10,000.

60. Common Mistake 55.6% 97.6% 10,000/uL 15,000/uL 5,000/uL 20,000/uL 97.6/55.6 = 1.8

61. Main Point 3 Likelihood Ratio P(Result) in patient WITH disease ------------------------------------------------------ P(Result) in patients WITHOUT disease Slope of ROC Curve Do not calculate an LR(+) or LR(-) for a multilevel test.

62. Clinical Scenario WBC = 16,000/mL Pre-test prob: 0.03 Pre-test odds: 0.03/0.97 = 0.031 LR(WBC btw 15 and 20) = 2.4 Post-Test Odds = Pre-Test Odds x LR(16) = 0.031 x 2.4 = 0.074* Post-Test prob = 0.074*/(0.074*+1) = 0.07 *Wrong in handout

63. Clinical Scenario WBC = 28,000/mL Pre-test prob: 0.03 Pre-test odds: 0.03/0.97 = 0.031 LR(28,000/uL) = 5.3 Post-Test Odds = Pre-Test Odds x LR(28) = 0.031 x 5.3 = 0.16 Post-Test prob = 0.16/(0.16+1) = 0.14

64. Clinical Scenario WBC = 16,000/uL Post-Test Prob = 0.07 WBC = 28,000/uL Post-Test Prob = 0.14 (Recall that dichotomizing the WBC with a fixed cutpoint of 15,000/uL meant that WBC = 16,000/uL would be treated the same as WBC = 28,000/uL and post-test prob = 0.10)

65. Main Point 4 Bayes’s Rule Pre-Test Odds x LR(result) = Post-Test Odds What you knew before + What you learned = What you know now

66. Summary 1)Dichotomizing a multi-level or continuous test by choosing a fixed cutpoint reduces the value of the test. 2)The ROC curve summarizes the discriminatory ability of the test. 3) LR(result) = P(result|D+)/P(result|D-) = Slope of ROC Curve (NOTE: Do not calculate an LR(+) or LR(-) for a multilevel test.) 4)Pre-Test Odds x LR(result) = Post-Test Odds

67. WBC Count (x1000/uL) Sensitivity1 - Specificity > 50 0% > 3011.8%0.8% > 2521.3%2.6% > 2048.0%8.0% > 1585.8%23.5% > 1097.6%55.6% > 5100.0%93.7% > 0100.0% Using “ROC Tables” to Get Interval LRs

68. Most abnormal interval (>= to top cutoff): D+ frequency = sensitivity of top cutoff; D- frequency = FPR of top cutoff For each less abnormal interval (between a higher and lower cutoff): D+ frequency = sensitivity of the lower cutoff - sensitivity of the higher cutoff; D- frequency = FPR of the lower cutoff - FPR of the higher cutoff Least abnormal interval (<= lowest cutoff): D+ frequency = 100% - low cutoff sensitivity; D- frequency = 100% - low cutoff FPR.

69. Example 1 Febrile Child with WBC count = 16,000

70. Lee et al. Arch Peds Adol Med 1998;152:624-28 Focus on these

71. Using “ROC Tables” to Get Interval LRs CutoffSensitivitySpecificity1 - Spec >= 150.860.770.23 >= 160.770.810.19 >=170.720.840.16 We will use this row, and … …this row

72. Using “ROC Tables” to Get Interval LRs For the interval >= 15 and <17, P(r|D+) = Sens (>=15) – Sens(>=17) = 0.86 - 0.72 = 0.14 P(r|D-) = FPR(>=15) – FPR(>=17) = 0.23 - 0.16 = 0.07

73. Using “ROC Tables” to Get Interval LRs LR(WBC btw 15-17) = P(r|D+) / P(r|D-) = 0.14/0.07 = 2 For the interval >= 15 and <17, LR(WBC btw 15 and 17) = 2 Child has WBC Count of 16,000 Post-Test Odds = Pre-Test Odds x 2

74. Something to notice The LR we just obtained, for a WBC of 16 (15-17, actually) was 2.0 The LR for the category 15- <20 was 2.4 This makes sense, because 16 is at the low end of the 15 –20 range The LR for a WBC of 19 would be a little higher than 2.5

75. Likelihood Ratio Slide Rule Calculations: Pre-Test Probability: 3% WBC = 16,000 LR(16,000): 2.0

76. Clinical Scenario WBC = 16,000/mL Pre-test prob: 0.03 Pre-test odds: 0.03/0.97 = 0.031 LR(WBC = 16) = 2.0 Post-Test Odds = Pre-Test Odds x LR(16) = 0.031 x 2.0 = 0.062 Post-Test prob = 0.062/(0.062+1) = 0.058

77. ROC Curve when a lower test result is more abnormal Gestational age as a predictor of neonatal morbidity. Trace ROC curve by serially moving cutoff from the lowest level (<24 weeks) up to the highest level (<45 weeks)

78. Gestational Age as Predictor of Neonatal Morbidity/Mortality

79. < 36 weeks

80. Calculating the c Statistic In the “walking man” approach to tracing out the ROC curve, the actual values of the test are not important for the shape of the ROC curve or the area under it--only the ranking of the values. The c statistic for the area under an ROC curve comes out exactly the same as the Wilcoxon Rank Sum statistic (or Mann- Whitney U, which is equivalent). Non-parametric equivalent of the t test statistic comparing two means.

81. BACTEREMIANO BACTEREMIA 27 22 21 19 18 17 14 13 12 8 7 6 4 TEST RESULTS

82. Boxes under Curve = 43.5 Total Boxes = 50 Area Under Curve = 43.5/50 = 0.87

83. BACTEREMIANO BACTEREMIA 1 2 3 4 5 6.5 8 9 10 11 12 13 14 15 S = 21.5 Replace Test Results with Ranks

84. S = 21.5 Smin = d(d+1)/2 = 5(6)/2 = 15 Smax = dn + Smin = 5(10) + 15 = 65 C = (Smax – S) / (Smax – Smin)* = (65 – 21.5) / (65 – 15) = 43.5/50 = 0.87 * Smax – Smin = dn Calculating the C Statistic

85. Logarithms Log 10 (a) = b, where a =10 b Log 10 (100) = 2 Log 10 (10) = 1 Log 10 (1) = 0 Log 10 (0.10) = -1 Log 10 (0.01) = -2

86. Multiplying is Adding Logs Log 10 (xy) = Log 10 (x) + Log 10 (y) Log 10 (10 x 1000) = Log 10 (10) + Log 10 (100) = 1 + 2 = 3

87. Dividing is Subtracting Logs Log 10 (x/y) = Log 10 (x) - Log 10 (y) Log 10 (10/100) = Log 10 (10) - Log 10 (100) = 1 - 2 = -1