1. SC-PM6: Prediction Models in Medicine: Development, Evaluation and Implementation Michael W. Kattan, Ph.D. Ewout Steyerberg, Ph.D. Brian Wells, M.S., M.D.

2. When The Patient Wants A Prediction, What Options Does The Clinician Have? Quote an overall average to all patients Deny ability to predict at the individual patient level Assign the patient to a risk group, i.e. high, intermediate, or low Apply a model Predict based on knowledge and experience

3. How do we typically compute risk? Based on features, we make a crude tree. Most cancer staging systems do this. BT=high H=Agg And DE=E HIGH RISK LOW RISK Y N Y N

4. The problem with crude trees They are very easy to use. But they do not predict outcome optimally. – High risk groups are very heterogeneous. – A single risk factor may qualify a patient as high risk. Other approaches, like a Cox regression statistical model, predict more accurately. Readily applicable presentation essential.

5. Biopsy Gleason Grade  2+  2 3+3  3+  4  2+3  4+ ? Total Points 0 20 40 60 80100120140160180200 60 Month Rec. Free Prob..96.93.9.85.8.7.6.5.4.3.2.1.05 3+  2 Clinical Stage T1cT1ab T2aT2cT3a T2b Points 0 10 20 30 40 50 60 70 80 90100 PSA 0.11236891012163045701107 20 4 Preoperative Nomogram for Prostate Cancer Recurrence Instructions for Physician: Locate the patient’s PSA on the PSA axis. Draw a line straight upwards to the Points axis to determine how many points towards recurrence the patient receives for his PSA. Repeat this process for the Clinical Stage and Biopsy Gleason Sum axes, each time drawing straight upward to the Points axis. Sum the points achieved for each predictor and locate this sum on the Total Points axis. Draw a line straight down to find the patient’s probability of remaining recurrence free for 60 months assuming he does not die of another cause first. Instruction to Patient: “Mr. X, if we had 100 men exactly like you, we would expect between and to remain free of their disease at 5 years following radical prostatectomy, and recurrence after 5 years is very rare.”  1997Michael W. Kattan and Peter T. Scardino Kattan MW et al: JNCI 1998; 90:766-771.

6. Some simple steps that will make a difference 1. Build the most accurate model possible. 2. Take model to bedside – As a nomogram, – In stand-alone software (desktop, handheld, web) – Built into the electronic medical record Doing this will predict patient outcome more accurately, resulting in – better patient counseling – better treatment decision making

7. What is a Nomogram? A prediction device Usually a regression model presented graphically Continuous-based prediction

8. Making a nomogram Usually a regression model (Cox or logistic) – Could consider machine learning techniques (neural nets, optimized trees like CART) Keep continuous variables continuous but relax linearity assumptions P-values for predictors don’t matter No variable selection or univariable screening Bottom line is its predictive accuracy

9. CaPSURE Heterogeneity within Risk Groups Risk Group Nomogram Values by Prostate Cancer Risk Group Preoperative Nomogram Predicted Probability LowIntermediateHigh 1.0 0.9 0.8 0.7 0.6 0.5 0.4 0.3 0.2 0.1 0.0 J Urol. 2005 Apr;173(4):1126-31

10. Kattan MW, et al., J Clin. Oncol., 2000.

11. 10 50 T2c T3c 2 4 6 3 5 3D Conformal Radiation Therapy Nomogram for PSA Recurrence Kattan MW, et al., J Clin. Oncol., 2000.

12. Discrimination on Cleveland Clinic data (N=912) Kattan MW et al., J Clin. Oncol., 2000. 0.5 0.55 0.6 0.65 0.7 0.75 0.8

13. Why Nomograms Matter: A Particular Example Mr. X, from the Cleveland Clinic: – PSA=6, clinical stage = T2c, biopsy Gleason sum=9, planned dose of 66.6 Gy without neoadjuvant hormones Shipley risk stratification: 81% @ 5 yr. Surgery nomogram: 68% @ 5yr. Radiation therapy nomogram: 24% @ 5yr. Kattan MW, et al., J Clin. Oncol., 2000.

14. When The Patient Wants A Prediction, What Options Does The Clinician Have? Quote an overall average to all patients Deny ability to predict at the individual patient level Assign the patient to a risk group, i.e. high, intermediate, or low Apply a model Predict based on knowledge and experience

15. Urologists vs. Preoperative Nomogram 10 case descriptions from 1994 MSKCC patients presented to 17 urologists – In addition to PSA, biopsy Gleason grades, and clinical stage, urologists were provided with patient age, systematic biopsy details, previous biopsy results, and PSA history. Preoperative nomogram was provided. Urologists were asked to make their own predictions of 5 year progression-free probabilities with or without use of the preoperative nomogram. Concordance indices: – Nomogram = 0.67 – Urologists = 0.55, p<0.05 Ross P et al., Semin Urol Oncol, 2002.

16. Nomogram for predicting the likelihood of additional nodal metastases in breast cancer patients with a positive sentinel node biopsy Vanzee K, et al., Ann Surg Oncol., 2003.

17. Breast Cancer Prediction: 17 Clinicians vs. Nomogram on 33 Patients Nomogram AUC 0.72 Clinician AUC 0.54 Sensitivity: Proportion of women with positive nodes predicted to have positive nodes Specificity: Proportion of women with negative nodes predicted to have negative nodes

18. ROC Curves Individual Clinicians and Nomogram Areas 0.75 0.72 Nomogram 0.68 0.65 0.63 0.59 0.58 0.55 0.53 0.52 0.50 0.49 0.47 0.43 0.42 0.40

19. 19 All of these patients received radical prostatectomy, are now experiencing rising PSA, and have not started ADT. AgeRace Clinical Stage Biopsy PSA Biopsy Gleason Sum Adjuvant Radiation Months from Surgery to Today Pathological Gleason SumCap.invECEMarginSVILN PSA at BCR PSA Doubling time (months) If you had 100 patients just like this one, how many do you think would have a positive bone scan 1 year from today if left untreated? (Enter a number between 0 and 100) 67WT2A2.77N16.129.00PPNPP2.53.62 60WT2B12.77N133.097.00PNPNN295911.65 63WT1C20.06N13.197.00PNPPP0.55.11 72WT1C13.27N9.647.00PNPNN0.63.04 64WT2C101.05N25.107.00PPPPP23.24 57WT2B11.14N9.187.00PPPPN6.41.51 54WT2B23.910N7.607.00PPPPP1.51.28 65WT2A13.56N103.167.00PPPNN88.52 65WT1C25.86N8.136.00PPPNN0.58.08 61WT1C13.56N34.907.00PPPNN0.711.58 72WT1C10.17N14.678.00PNPNN0.84.29 67WT1C26.86N10.436.00PPPNN1.43.92 62WT2A4.57N13.397.00PPNNN0.55.17 69WT1C4.77N11.328.00PPNPN3.41.76 67WT1C10.76N44.057.00PPNNN5.47.32 65WT1C7.46N37.507.00PNNNN6.95.79 59WT1C5.07N13.957.00PNNNN0.294.07 57WT1C13.37N3.827.00PNNNN0.52.21 53WT1C14.69N5.829.00PPPPN0.33.36 62WT1C14.68N20.537.00PNPNN1.36.10 62WT1C15.89N16.129.00PNPNN7.43.76 63WT2A7.17N21.817.00PPNNN1.25.61 43WT1C4.67N31.587.00PNPNN18.50 57WT2A4.47N9.057.00PNNNN0.34.84 59WT1C4.27N20.729.00PPPPN0.44.50

20. Nomogram to Predict Bone Scan Positivity (Cont.) Slovin SF, et al. Clin Can Res. 2005;11:8669-8673. Nonogram Used to Predict Patient-Specific Probabilities of Metastasis-free Survival at 1 and 2 Years, and the Median Progression-free Survival Time AUC=0.69 Points bPSA, ng/mL PSADT, mo Gleason Total Points 1-Year PFS 0102030405060708090100 483624126 68-9 7 2-Year PFS T Stage Median PFS 020406080100120140160180 3+ 1-2 0.80.70.50.30.1 0.90.70.50.30.1 1210864 2 0 0.371.02.77.42055150245 20

21. Predictions by Docs and Nomogram, by Patient

22. Survey Results

24. Discrimination accuracy for nomogram and docs

25. Feedback recall, overconfident, hindsight bias, chance Biases in Human Prediction adapted from Hogarth, 1988

26. Conclusions for Part 1 Continuous statistical prediction models offer accuracy advantages over: – Crude risk groups – Clinical judgment Remaining challenges: – How to evaluate accuracy – If accuracy is acceptable, how to deploy/disseminate