Presentation is loading. Please wait.

Presentation is loading. Please wait.

D:/rg/folien/ms/ms-USA-040511.ppt F 1 Assessment of prediction error of risk prediction models Thomas Gerds and Martin Schumacher Institute of Medical.

Published byRodney McKenzie Modified over 8 years ago

Similar presentations

Presentation on theme: "D:/rg/folien/ms/ms-USA-040511.ppt F 1 Assessment of prediction error of risk prediction models Thomas Gerds and Martin Schumacher Institute of Medical."— Presentation transcript:

1 D:/rg/folien/ms/ms-USA-040511.ppt F 1 Assessment of prediction error of risk prediction models Thomas Gerds and Martin Schumacher Institute of Medical Biometry and Medical Informatics University Hospital Freiburg, Germany

2 D:/rg/folien/ms/ms-USA-040511.ppt F 2 –Situation –Measures of prediction error –Application to prediction of breast cancer survival –General conclusion –Considerations for breast cancer risk prediction Outline

3 D:/rg/folien/ms/ms-USA-040511.ppt F 3 Situation (1) –Goal:Assessment of predictions  (t|X i ) based on a comparison with actually observed outcomes T i in a sample of n individuals (i = 1,…,n) predicted probability that an individual will be event- free up to t units of time based on covariate information X available at t = 0 –Prediction:  (t|X) T denotes time to event of interest –Outcome:

4 D:/rg/folien/ms/ms-USA-040511.ppt F 4 Situation (2) can be defined for a fixed time t or for a time range should have the properties of a survival probability function is ideally externally derived but otherwise, can be anything: produced by statistical model building, by machine learning techniques or may constitute expert guesses –Prediction:  (t|X)

5 D:/rg/folien/ms/ms-USA-040511.ppt F 5 Measures of prediction error (1) –General loss function approach E (L (T, X,  )) –Common choices:

6 D:/rg/folien/ms/ms-USA-040511.ppt F 6 Measures of prediction error (2) –Expected quadratic or Brier score "Mean Squared Error of Prediction (MSEP)"

7 D:/rg/folien/ms/ms-USA-040511.ppt F 7 Measures of prediction error (2) –Expected quadratic or Brier score "Mean Squared Error of Prediction (MSEP)" –Decomposition S(t|X) denotes the "true" probability that an individual with covariate X will be event-free up to t

8 D:/rg/folien/ms/ms-USA-040511.ppt F 8 Measures of prediction error (3) –Empirical quadratic or Brier Score "Residual Sum of Squares (RSS)" –MSEP and RSS are time-dependent in survival problems –Graphical tool: plotting RSS over time

9 D:/rg/folien/ms/ms-USA-040511.ppt F 9

10 D:/rg/folien/ms/ms-USA-040511.ppt F 10 Measures of prediction error (3) –Incorporation of censored observations "Weighted Residual Sum of Squares (WRSS)" –Empirical quadratic or Brier Score "Residual Sum of Squares (RSS)"

11 D:/rg/folien/ms/ms-USA-040511.ppt F 11 Application to prediction of breast cancer survival GBSG-2-study (German Breast Cancer Study Group) 686 patients with complete information on prognostic factors Two thirds are randomized, otherwise standardized treatment Median follow-up 5 years, 299 events for event-free survival Prognostic factors considered: age, tumor size, tumor grade, number of positive lymph nodes, progesterone receptor, estrogen receptor Predictions for individual patients are derived in terms of conditional event-free probabilities given the covariate combination by means of the Nottingham Prognostic Index and a Cox regression model with all six prognostic factors

12 D:/rg/folien/ms/ms-USA-040511.ppt F 12

13 D:/rg/folien/ms/ms-USA-040511.ppt F 13 Which benchmark value? –"Naive" prediction  (t|X) = 0.5 for all t and X gives a Brier score value of 0.25 –Common prediction  (t) for all individuals ignoring the available covariate information ("pooled Kaplan Meier")

14 D:/rg/folien/ms/ms-USA-040511.ppt F 14

15 D:/rg/folien/ms/ms-USA-040511.ppt F 15 Which benchmark value? –"Naive" prediction  (t|X) = 0.5 for all t and X gives a Brier score of 0.25 –Common prediction  (t) for all individuals ignoring the available covariate information ("pooled Kaplan Meier")' –Calculation of R 2 -measures for checking various aspects of prediction models

16 D:/rg/folien/ms/ms-USA-040511.ppt F 16

17 D:/rg/folien/ms/ms-USA-040511.ppt F 17

18 D:/rg/folien/ms/ms-USA-040511.ppt F 18 –is the mean squared error of prediction (MSEP) when predictions are made in terms of event(-free) probabilities –allows the assessment of any kind of predictions based on individual covariate values –can be estimated even in the presence of right censoring by a weighted residual sum of squares in a nonparametric way –is a valuable tool to detect overfitting –allows the calculation of R 2 -measures –can be adapted to the situation of competing risks and dynamic updating of predictions General conclusion The quadratic or Brier score

19 D:/rg/folien/ms/ms-USA-040511.ppt F 19 Considerations for breast cancer risk prediction –Intention:Assessment of predictions for t = 5y based on aggregated data published by Costantino et al. JNCI 1999; constant prediction ignoring all covariate information is used as benchmark value predicted probability that a woman will develop breast cancer up to time t based on covariate information including age available at t = 0 (entry into program or study; time when prediction is performed) –Prediction:  (t|X) T denotes time from entry into program to development of breast cancer –Outcome: development of breast cancer before t otherwise

20 D:/rg/folien/ms/ms-USA-040511.ppt F 20 Costantino et al., Journal of the National Cancer Institute, Vol. 91, No. 18, September 15, 1999

21 D:/rg/folien/ms/ms-USA-040511.ppt F 21 Estimated prediction error based on aggregated data Brier ScoreLogarithmic score Age group, ymodel 1const. pred.model 1const. pred.  49 0.034280.03432 0.152890.15367 50 – 59 0.029000.02899 0.134460.13434  60 0.035190.03527 0.156250.15702 All ages 0.032960.03301 0.148340.14897

22 D:/rg/folien/ms/ms-USA-040511.ppt F 22 Estimated relative risk (RR) for predicted risk quintiles (model 1, all ages) Predicted 5-year, % risk < 2.321199301 2.32 – 2.651199331.1 2.66 – 3.281191431.436 3.29 – 4.731184391.316 > 4.731196591.972 No. of women Observed breast cancer RR

23 D:/rg/folien/ms/ms-USA-040511.ppt F 23 "Diagnostic" properties of predicted risk quintiles (model 1, all ages) 2.320.8530.2030.0360.975 2.660.6900.4050.0390.974 3.290.4800.6040.0410.971 4.730.2890.8030.0490.970 SensitivityPos. pred. value SpecificityNeg. pred. value Cutpoint Pred. 5-year risk, %

24 D:/rg/folien/ms/ms-USA-040511.ppt F 24

Download ppt "D:/rg/folien/ms/ms-USA-040511.ppt F 1 Assessment of prediction error of risk prediction models Thomas Gerds and Martin Schumacher Institute of Medical."

Similar presentations

Surviving Survival Analysis

Surviving Survival Analysis
Survival Analysis. Key variable = time until some event time from treatment to death time for a fracture to heal time from surgery to relapse.

Survival Analysis. Key variable = time until some event time from treatment to death time for a fracture to heal time from surgery to relapse.
Survival Analysis In many medical studies, the primary endpoint is time until an event occurs (e.g. death, remission) Data are typically subject to censoring.

Survival Analysis In many medical studies, the primary endpoint is time until an event occurs (e.g. death, remission) Data are typically subject to censoring.
Survival Analysis-1 In Survival Analysis the outcome of interest is time to an event In Survival Analysis the outcome of interest is time to an event The.

Survival Analysis-1 In Survival Analysis the outcome of interest is time to an event In Survival Analysis the outcome of interest is time to an event The.
Introduction to Survival Analysis October 19, 2004 Brian F. Gage, MD, MSc with thanks to Bing Ho, MD, MPH Division of General Medical Sciences.

Introduction to Survival Analysis October 19, 2004 Brian F. Gage, MD, MSc with thanks to Bing Ho, MD, MPH Division of General Medical Sciences.
Departments of Medicine and Biostatistics

Departments of Medicine and Biostatistics
Detecting an interaction between treatment and a continuous covariate: a comparison between two approaches Willi Sauerbrei Institut of Medical Biometry.

Detecting an interaction between treatment and a continuous covariate: a comparison between two approaches Willi Sauerbrei Institut of Medical Biometry.
April 25 Exam April 27 (bring calculator with exp) Cox-Regression

April 25 Exam April 27 (bring calculator with exp) Cox-Regression
Cervical Cancer Case Study Eshetu Atenafu, Sandra Gardner, So-hee Kang, Anjela Tzontcheva University of Toronto Department of Public Health Sciences (Biostatistics)

Cervical Cancer Case Study Eshetu Atenafu, Sandra Gardner, So-hee Kang, Anjela Tzontcheva University of Toronto Department of Public Health Sciences (Biostatistics)
the Cox proportional hazards model (Cox Regression Model)

the Cox proportional hazards model (Cox Regression Model)
Model and Variable Selections for Personalized Medicine Lu Tian (Northwestern University) Hajime Uno (Kitasato University) Tianxi Cai, Els Goetghebeur,

Model and Variable Selections for Personalized Medicine Lu Tian (Northwestern University) Hajime Uno (Kitasato University) Tianxi Cai, Els Goetghebeur,
Chapter 11 Survival Analysis Part 2. 2 Survival Analysis and Regression Combine lots of information Combine lots of information Look at several variables.

Chapter 11 Survival Analysis Part 2. 2 Survival Analysis and Regression Combine lots of information Combine lots of information Look at several variables.
Using time-dependent covariates in the Cox model THIS MATERIAL IS NOT REQUIRED FOR YOUR METHODS II EXAM With some examples taken from Fisher and Lin (1999)

Using time-dependent covariates in the Cox model THIS MATERIAL IS NOT REQUIRED FOR YOUR METHODS II EXAM With some examples taken from Fisher and Lin (1999)
How do we know whether a marker or model is any good? A discussion of some simple decision analytic methods Carrie Bennette on behalf of Andrew Vickers.

How do we know whether a marker or model is any good? A discussion of some simple decision analytic methods Carrie Bennette on behalf of Andrew Vickers.
Measures of disease frequency (I). MEASURES OF DISEASE FREQUENCY Absolute measures of disease frequency: –Incidence –Prevalence –Odds Measures of association:

Measures of disease frequency (I). MEASURES OF DISEASE FREQUENCY Absolute measures of disease frequency: –Incidence –Prevalence –Odds Measures of association:
Sample Size Determination

Sample Size Determination
EVIDENCE BASED MEDICINE

EVIDENCE BASED MEDICINE
Cox Proportional Hazards Regression Model Mai Zhou Department of Statistics University of Kentucky.

Cox Proportional Hazards Regression Model Mai Zhou Department of Statistics University of Kentucky.
Analysis of Complex Survey Data

Analysis of Complex Survey Data
Mapping Rates and Proportions. Incidence rates Mortality rates Birth rates Prevalence Proportions Percentages.

Mapping Rates and Proportions. Incidence rates Mortality rates Birth rates Prevalence Proportions Percentages.

Similar presentations

Ads by Google