1. Critiquing for Evidence-based Practice: Diagnostic and Screening Tests M8120 Columbia University Fall 2001 Suzanne Bakken, RN, DNSc

2. Overview zProbability fundamentals zClinical role of probability revision zCharacterizing new information (test performance) zProbability revision methods –Bayes’ formula –Contingency table zIn class exercises

3. Probability Fundamentals zStrength of belief zA number between 0 and1 that expresses an opinion about the likelihood of an event zProbability of an event that is certain to occur is 1 zProbability of an event that is certain to NOT occur is 0

4. Summation Principle zProbability that an event will occur plus probability that it will not occur equals 1 zProbability of all possible outcomes of a chance event is always equal to 1 –Blood type: What is p[AB] given p[O]=0.46, p[A]=.40, and p[B]=.10? –Fraternal triplets: What is the probability of at least one boy and one girl?

5. Diagramming Probabilities Type O (0.46) Type A (0.40) Type B (0.10) Type AB (0.04) Chance node Sum of probabilities at chance node = 1

6. Conditional Probability zProbability that event A occurs given that event B is known to occur zp[AlB] zp[A,B]=p[AlB] X p[B] zExamples in health care

7. Components of Probability Estimates zPersonal experience zPublished experience - evidence zAttributes of the patient

8. Role of Probability Revision Techniques Abnormal Finding Diagnosis Before Finding After Finding 0 1 Probability of Disease Prior Probability Posterior Probability

9. Role of Probability Revision Techniques Negative Finding Diagnosis After Finding Before Finding 0 1 Probability of Disease Posterior Probability Prior Probability

10. Findings zSigns zSymptoms zDiagnostic tests zProbabilistic relationships between findings and disease basis of diagnostic decision support systems –Dxplain –QMR –Iliad

11. Definitions zPrior probability - the probability of an event before new information (finding) is acquired; pretest probability or risk zPosterior probability - the probability of an event after new information (finding) is acquired; posttest probability or risk zProbability revision - taking new information into account by converting prior probability to posterior probability

12. Review of Conditional Probability p[AlB] p[A and B] p[B] = Probability that an event is true, given that another event is true What is the probability that someone has HIV antibody given a positive HIV test? What is the probability that someone has venothrombosis given a swollen calf? What is the probability that someone has dyspnea given the nurse says dyspnea is present?

13. Characterizing “Test” Performance zCompare test against gold standard (e.g., presence of disease; established test) zIdeal test - no values at which the distribution of those with the disease and without the disease overlap zFew tests ideal so … –TP –TN –FP –FN

14. Test Performance zTrue positive rate (TPR) p[+lD] = probability of an abnormal test result given that the disease is present; the number of persons WITH the disease who have an abnormal test result divided by the number of persons WITH the disease; sensitivity

15. Contingency Table View zDisease present = TPR + FNR = 1 zDisease absent = FPR + TNR = 1

16. Test Performance zFalse positive rate (FPR) p[+lno D] = probability of an abnormal test result given that the disease is absent: the number of persons WITHOUT the disease who have an abnormal test result divided by the number of persons WITHOUT the disease

17. Contingency Table View zDisease present = TPR + FNR = 1 zDisease absent = FPR + TNR = 1

18. Test Performance zTrue negative rate (TNR) p[-lno D] = probability of a normal test result given that the disease is absent; number of persons WITHOUT the disease who have a normal test result divided by number of persons WITHOUT the disease; 1 - FPR or specificity; 100% specificity = pathognomonic

19. Contingency Table View zDisease present = TPR + FNR = 1 zDisease absent = FPR + TNR = 1

20. Test Performance zFalse negative rate (FNR) p[-l D] = probability of a normal test result given that the disease is present; number of persons WITH the disease who have a normal test result divided by number of persons WITH the disease; 1 - TPR

21. Contingency Table View zDisease present = TPR + FNR = 1 zDisease absent = FPR + TNR = 1

22. Sensitivity vs. Specificity zWeighing sensitivity Vs. specificity in setting cutoff level for abnormality in a test zConsequences of FPR vs. FNR –Severity of disease –Availability of treatment –Risk of treatment zSensitivity and specificity are characteristics of a test and a criterion for abnormality

23. Receiver Operating Characteristic (ROC) Curves 0 0.5 1.0 1.0 0.5 FPR (1 - specificity) TPR ( sensitivity ) Increased p[D] Decreased p[D]

24. Contingency Table View zDisease present = TPR + FNR = 1 zDisease absent = FPR + TNR = 1

25. Example of HIV What is the TPR? What is the FPR? What is specificity?

26. Example of HIV TP TPR = TP + FN TPR = sensitivityp[+lD]

28. Example of HIV FP FPR = FP + TN FPR = p[+lno D]

30. Example of HIV TN TNR = TN + FP TNR = specificityp[-lno D]

32. Nurse and Patient Rating of Symptoms zBy definition, patient is gold standard for symptom rating zRN as “test” for presence or absence of symptom zFatigue –RN yes/Pt yes = 50 –RN yes/Pt no = 15 –RN no/Pt yes = 20 –RN no/Pt no = 15 zWhat is sensitivity? What is specificity?

33. Nurse and Patient Ratings of Symptoms: Sensitivity TP TPR = TP + FN

35. Nurse and Patient Ratings of Symptoms: Specificity TN TNR = TN + FP

37. Moving from Test Characteristics to Predictive Value and Posterior Probabilities zPredictive value zForms of Bayes’ –Bayes’ formula –Contingency table view –Likelihood ratio

38. Prevalence zFrequency of disease in the population of interest at a given point in time

39. Predictive Value zSensitivity, specificity, and their complements (FNR & FPR) focus on probability of findings given presence or absence of disease so not in a clinically useful form zPredictive value focuses on probability of disease given findings zPredictive value takes prevalence of disease in study population into account

40. Positive Predictive Value number of persons with disease with abnormal finding PV+ = number of persons with abnormal finding TP PV+ = TP + FP The fraction of persons with an abnormal finding who have the disease

41. Negative Predictive Value number of persons with normal finding WITHOUT disease PV- = number of persons with normal finding TN PV+ = TN + FN The fraction of persons with an normal finding who DO NOT have the disease

42. Nurse and Patient Rating of Symptoms zBy definition, patient is gold standard for symptom rating zRN as “test” for presence or absence of symptom zShortness of Breath –RN yes/Pt yes = 25 –RN yes/Pt no = 10 –RN no/Pt yes = 15 –RN no/Pt no = 50 zWhat is PV+? What is PV-?

43. Nurse and Patient Ratings of Symptoms: PV+ TP PV+ = TP + FP

45. Nurse and Patient Ratings of Symptoms: PV- TN PV- = TN + FN

47. Role of Probability Revision Techniques Abnormal Finding Diagnosis Before Finding After Finding 0 1 Probability of Disease Prior Probability Posterior Probability

48. Calculating Posterior Probability with Bayes’ zWhat is the probability that someone has HIV antibody given a positive HIV test? zCan calculate with Bayes’ if you know: –Prior probability of the disease –Probability of an abnormal test (+) result conditional upon the presence of the disease (TPR) –Probability of an abnormal test (+) result conditional upon the absence of the disease (FPR)

49. Bayes’ Theorem p[Dl+] p[D] x p[+lD] {p[D] x p[+lD]} + {p[no D] x p[+lno D]} = Not a clinically useful form!

50. Deriving Bayes’ Theorem Summation principle: p[Dl+] p[+] p[+,D] = Given definition of conditional probability: p[+] = p[+,D] + p[+,no D] p[Dl+] p[+,D] + p[+, no D] p[+,D] = Thus: 1 2 3 p[+lD] p[D] p[+,D] = p[+lno D] p[no D] p[+,no D] = 4 and Given principle of conditional independence, rearrange expressions above: p[+,D] = p[D] x p[+lD] p[+,no D] = p[no D] x p[+lnoD] 5 p[Dl+] p[D] x p[+lD] +p[no D] x p[+lnoD] p[D] x p[+lD] = Substitute into 1: 6 Given definition of conditional probability

51. Calculating Posterior Probability with Bayes’ zWhat is the probability that someone has HIV antibody given a positive HIV test? zCan calculate with Bayes’ if you know: –Prior probability of the disease –Probability of an abnormal (+) test result conditional upon the presence of the disease (TPR=.98) –Probability of an abnormal (+) test result conditional upon the absence of the disease (FPR=.01)

52. Bayes’ Theorem p[Dl+] p[D] x p[+lD] {p[D] x p[+lD]} + {p[no D] x p[+lno D]} = TPR FPR 1 - p[D]

53. When Abnormal Test Result is Present p[Dl+] p[D] x TPR {p[D] x TPR} + {1 - p[D] x FPR} = A somewhat more clinically useful form

54. When Normal Test Result is Present p[Dl-] p[D] x 1 - TPR {p[D] x 1 - TPR} + {1 - p[D] x 1 - FPR} = A somewhat more clinically useful form FNR TNR

55. Calculating Posterior Probability with Bayes’ zWhat is the probability that someone has HIV antibody given a positive HIV test? zCan calculate with Bayes’ if you know: –Prior probability of the disease (can be prevalence or other information) –Probability of an abnormal test result conditional upon the presence of the disease (TPR=.98) –Probability of an abnormal test result conditional upon the absence of the disease (FPR=.01)

56. The Role of Prior Probability p[Dl+] = Prevalence of HIV Antibody in Homosexual Men in SF in mid1980s =.5 p[Dl+] =

57. The Role of Prior Probability p[Dl+] p[.5] x.98.49 + (.51 x.01) =.4951 = Prevalence of HIV Antibody in Homosexual Men in SF in mid1980s =.5 p[Dl+] =.99

58. The Role of Prior Probability p[Dl+] = Prevalence of HIV Antibody in Female Blood Donors =.0001 p[Dl+] =

59. The Role of Prior Probability p[Dl+] p[.0001] x.98.00098 + (.9999 x.01) =.010979 = Prevalence of HIV Antibody in Female Blood Donors =.0001 p[Dl+] =. 089

60. Likelihood Ratios Likelihood ratio = FPR An even more clinically useful form! TPR Nomogram for interpreting diagnostic test result

61. Diagramming Probabilities Type O (0.46) Type A (0.40) Type B (0.10) Type AB (0.04) Chance node Sum of probabilities at chance node = 1

62. Path Probability Operate Do not operate Disease present Disease absent Disease present Disease absent Survive Operative death Palliate Operative death Survive No cure Cure No Cure No cure Cure p=.10 p=.90 p=.10 p=.90 p=.10 p=.02 p=.98 p=.10 p=.90 p=.10 p=.90 p=.10 p=.01 p=.99 Try for the cure Path probability of a sequence of chance events is the product of all probabilities along that sequence

63. Conditional Independence zTwo findings are conditionally independent if TPR and FPR of one clinical finding do not depend upon the presence of the other finding zAssumption of conditional independence invoked when the same TPR (or FPR) is used in Bayes’ regardless of the prior probability of disease zRelevant in series of tests zMay be invalid in some clinical situations

64. Interpreting Sequence of Tests zPosttest probability of first test used as pretest probability of second test zTPR and FPR of second test used in Bayes’ to calculate posttest probability following second test

65. Test Performance Biases zMost significance source of error in measuring test performance is due to differences between population in which test performance is measured and the population in which the test will be used zSpectrum bias - differences between populations in the spectrum of disease presentation and severity –Test population contains more sick persons than clinically relevant population –Test-referral bias - the composition of the population used to evaluate a diagnostic test is altered when the test is a criterion for referring a patient for the definitive diagnostic procedure zTPR is usually higher in the study population than in the clinically relevant population due to few negatives (FN & TN) referred zFPR is usually higher in the study population than in the clinically relevant population due to few TN (remember FP + TN = 1)

66. Critically Analysis of Report of Diagnostic or Screening Test 4Are the results of the study valid? zWhat are the results? zWill the results help me in caring for my patients?

67. Critically Analysis of Report of Diagnostic or Screening Test zAre the results of the study valid? –Was there an independent, blind comparison with a reference (gold) standard? –Did the patient sample include an appropriate spectrum of patients to whom the diagnostic test will be applied in clinical practice? –Did the results of the test being evaluated influence the decision to perform the reference standard? –Were the methods for performing the test described in sufficient detail to permit replication?

68. Critically Analysis of Report of Diagnostic or Screening Test zAre the results of the study valid? 4What are the results? zWill the results help me in caring for my patients?

69. Critically Analysis of Report of Diagnostic or Screening Test zWhat are the results? –Are likelihood ratios for the test result presented or data necessary for their calculation included?

70. Critically Analysis of Report of Diagnostic or Screening Test zAre the results of the study valid? zWhat are the results? 4Will the results help me in caring for my patients?

71. Critically Analysis of Report of Diagnostic or Screening Test zWill the results help me in caring for my patients? –Will the reproducibility of the test result and its interpretation be satisfactory in my setting? –Are the results applicable to my patient?