Mark Pletcher 6/9/2011 Prognostic and Genetic Tests

An Example Mammaprint Gene expression profiling for Breast CA Grind up the tumor, extract RNA Incubate with a microarray of DNA fragments to estimate expression for each gene 70 previously identified genes predict outcomes

Van de Vijver et al. NEJM 2002;347(25):

An Example Mammaprint Pattern of expression correlates with disease-free and overall survival

Van de Vijver et al. NEJM 2002;347(25):

An Example Mammaprint 10-year probability of: SurvivalFree of mets Good pattern95%85% Bad pattern55%51% Van de Vijver et al. NEJM 2002;347(25):

Outline Prognostic vs. Diagnostic Tests Evaluating a Prognostic Test Accuracy Utility Genetic Tests (very briefly)

Prognostic vs. Diagnostic Tests How is a prognostic test different from a diagnostic test?

Diagnostic TestPrognostic Test Purpose Chance Event Occurs to Patient Study Design Maximum Obtainable AUROC Identify Prevalent Disease Predict Incident Disease/Outcome Prior to TestAfter Test Cross-SectionalCohort <1 (not clairvoyant) 1 (gold standard) Prognostic vs. Diagnostic Tests

Classic prognosis: Prediction of death after diagnosis of a disease

Prognostic vs. Diagnostic Tests Prognosis, broadly speaking: Prediction of any future event Death or recurrence of cancer Stroke after presentation for TIA Peri-operative MI in surgical patients First MI in asymptomatic persons

Prognostic vs. Diagnostic Tests Prognosis vs. Diagnosis: A Spectrum Grey areas Pre-clinical disease: Coronary calcium Reversible disease: Tiny lung CA Irreversible predisposition: Huntingtons gene

Prognostic vs. Diagnostic Tests Prognostication Etiology Risk factor Causes the disease Reducing it may prevent disease Confounding is crucial issue in observational studies Risk marker (i.e., prognostic factor) Predicts the disease Need not be concerned about unmeasured confounders Not all risk markers are risk factors…(e.g., CRP)

Evaluating Prognostic Tests Test Performance Association Discrimination Calibration Reclassification Pitfalls Test Utility

Evaluating Prognostic Tests Association Is the marker associated with development of the disease? Odds ratio, relative risk, hazard ratio Independently associated means after adjustment for other known predictors

Evaluating Prognostic Tests HR adj = 4.6 P<.001 Van de Vijver et al. NEJM 2002;347(25):

Evaluating Prognostic Tests Discrimination Ability to distinguish between people with higher or lower risk of disease Metrics: just like diagnostic tests!? Sensitivity/specificity ROC curves

Evaluating Prognostic Tests Mammaprint Sensitivity = 28/30 = 93% Specificity = 41/83 = 49% Mets <5yr No mets

Coronary artery calcium Predictor of CHD events Adds discrimination AUROC FRS = Framingham Risk Score CACS = Coronary Artery Calcium Score Greenland et al. JAMA 2004;291(2): Evaluating Prognostic Tests

Discrimination Results are specific to a particular time point 5-year risk of metastases or death 90-day risk of stroke

Evaluating Prognostic Tests Discrimination Different results at 5 years….

Evaluating Prognostic Tests Discrimination …than at 10 years

Evaluating Prognostic Tests Discrimination Often 1 time point is most relevant or easily communicated, but information is lost… Can think of a set of discrimination statistics/ROC curves Harells C-Statistic Integrated C-statistic for survival data Similar interpretation as AUROC Harrell et al. Stat Med 1996;15(4):

Evaluating Prognostic Tests Calibration How close is predicted risk to actual risk?

Evaluating Prognostic Tests Prognostic test results are often converted into absolute risk estimates Like post-test probabilities in diagnosis Required for clinical interpretation Estimated directly in a longitudinal study

Evaluating Prognostic Tests But absolute risk estimates can be off When derivation population different than target population, etc Framingham example

DAgostino et al. JAMA 2001;286(2):

Evaluating Prognostic Tests Calibration is orthogonal to discrimination Awful discrimination but good calibration Awful calibration but good discrimination Miscalibration leads to worse errors, but its easier to fix…

Evaluating Prognostic Tests Reclassification How often does the test lead to reclassification across a treatment threshold? i.e., how often might the test lead to a change in treatment? CRP reclassification example

Evaluating Prognostic Tests Reclassification How often does the test lead to reclassification across a treatment threshold? Cook et al. Annals of Int Med 2006;145(1):21-29

Evaluating Prognostic Tests Reclassification metrics Net Reclassification Improvement (NRI) Net % reclassified correctly Depends on specified treatment thresholds/categories

Evaluating Prognostic Tests Pitfalls for prognostic test studies Loss to follow-up and competing risks Especially problematic if loss is differential

Evaluating Prognostic Tests Pitfalls for prognostic test studies Bias if clinician knows the test result e.g. – persons with coronary calcium+ are: More likely to get revascularization More likely to get referred to ED if they have chest pain

Evaluating Prognostic Tests Pitfalls for prognostic test studies Overfitting Test will perform best in sample from which it is derived More variables and choices more danger of overfitting Gene expression arrays, proteomics

Evaluating Prognostic Tests Clinical Utility Does it improve health?

Evaluating Prognostic Tests Test Result Better patient understanding of disease/risk Healthier patient behaviors Better clinical decisions Better health Pletcher et al. Circulation 2011;123;

Evaluating Prognostic Tests Clinical Utility Cannot be estimated from test performance metrics alone Need to understand downstream consequences, including Benefits and harms of interventions based on test result Harms from test itself Quality and length of life Costs

Evaluating Prognostic Tests Clinical Utility Can be estimated directly… Randomized trial of test-and-treat strategy …or indirectly Decision analysis/cost-effectiveness modeling Same issues for diagnostic tests, and especially important when screening apparently healthy people… Pletcher et al. Circulation 2011;123;

Genetic Tests Potentially useful for mechanistic insight Prognostic implications across individuals in a family Otherwise, must meet same standards for prognostic utility as other tests Single gene studies often disappointing

Key concepts For prognostic tests, an element of time and chance remain (perfect test impossible) Discrimination vs. Calibration Reclassification indices help us understand how often a test might change management Clinical utility depends on accounting for net benefits and harms (and costs)