1 SCreening for Occult REnal Disease (SCORED) Simple Algorithms to Predict Kidney Disease: ready to be used in the real world? Heejung Bang, PhD & Madhu Mazumdar, PhD Division of Biostatistics and Epidemiology Department of Public Health Weill Medical College of Cornell University

2 Overview  Background  Objectives  Methods: model development and validation  Results  Discussion

3 Background Prevalence of Kidney Disease ( ) Stages 1 and 2 with kidney damage

4 Background End-Stage Renal Disease (ESRD) Counts

5 Background Total Cost of Medicare for ESRD (in billions)

6 Background  Chronic kidney disease (CKD) is a global health problem. Low-awareness and late detection are common problems.  It is progressive disease. Yet, most affected individuals are asymptomatic with known risk factors and are not routinely tested.  Identifying individuals with CKD should be ‘simple’ with serum creatinine concentration that is widely available and inexpensive ($10-20), in combination with urinalysis.  Systematic methods to predict disease in other chronic conditions such as cardiovascular disease (e.g., Framingham, Reynolds scores, stroke instrument), cancer (e.g., Gail model), diabetes exist but not for CKD.

7 Objectives  To develop risk prediction model for prevalent CKD  Important prerequisites in our investigation: Easy to use but accurate Cumulative effects of concurrent risk factors Demographic + medical history + modifiable risk factors  To test the validity of the model internally as well as using independent large databases (i.e., external validation)  To compare the performance of the model with the current clinical practice guidelines  To develop risk prediction model for incident CKD

8 Kidney Early Evaluation Program (KEEP) by the National Kidney Foundation if a persons is ≥ 18 years old and has one or more of the following: 1. diabetes 2. high blood pressure 3. a family history of diabetes, high blood pressure or kidney disease

9 SCreening for Occult REnal Disease (SCORED) Bang et al. (2007)

10 Methods  Cross-sectional analysis of a nationally representative population based survey, the National Health and Nutritional Examination Surveys (NHANES)  Adult subjects only (≥20 years old)  Potential risk factors searched from literature  Endpoint: CKD stage 3 or higher, i.e., glomerular filtration rate (GFR) < 60 ml/min/1.73m 2 (using the MDRD formula)

11 Methods (Cont’d)  Split-sample method to create a development and validation dataset using a 2:1 ratio.  Standard diagnostic characteristics: # at high risk, sensitivity, specificity, positive & negative predictive values, area under ROC curve  Multiple logistic regression model (with proper weighting and complex survey design) e.g., proc surveylogistic in SAS.

12 Methods (Cont’d)  ‘Categorical scoring system’ derived by assigning an integer for the regression coefficients  ‘Continuous probability’ of having CKD from the fitted regression model  External validation using the Atherosclerosis Risk in Communities (ARIC) Study, Cardiovascular Health Study (CHS) and NHANES  Comparison between SCORED vs. KEEP using standard diagnostic measures  A number of sensitivity analyses (e.g., missing info, different definitions) --- important to be used in the real world!

13 Results  NHANES gave 10,291 individuals  After exclusions (based on unmeasured or missing data, etc.), dataset included 8,530 observations  A total of 601 individuals had CKD (5.4% weighted proportion)

14 Final SCORED model in development data (N= 5,666, AUC=0.88)

15 Diagnostic characteristics of SCORED in internal validation dataset (N=2,864) (cutpoint ≥4 to define high risk group)

16 Event rate by risk score

17 Fitting SCORED model to ARIC dataset (N= 12,038, AUC=0.71)

18 Sample questionnaire

19 Advantages of SCORED  Estimate the cumulative likelihood of having disease with multiple risk factors  Accuracy and high sensitivity.  Simple to use (implemented by the pen & pencil method) so foresee a variety of uses e.g., mass screenings public education initiatives, health fair medical emergency departments web-based medical information sites patient waiting room in clinics

20 Limitations of SCORED  Inability to assess family history of kidney disease -- many large national and community studies do not enquire about history of kidney disease.  For prevalent disease, not incident disease (a new risk score is needed, later in this talk)  Some variables may be commonly missing (e.g. proteinuria)  Low PPV (but prediction is HARD!)  Kidney disease: multiple definitions, different stages

21 Diagnostic performance of SCORED vs. KEEP using external validation data (Bang, Mazumdar et al. 2008) Screening guidelines % high risk SensitivitySpecificityPPVNPVAUC SCORED NHANES ARIC/CHS ARIC/CHS * ARIC/CHS * KEEP NHANES NHANES * ARIC ARIC/CHS * some sensitivity analyses

22 A simple algorithm to predict incident kidney disease (aka, SCORED II) by Kshirsagar, Bang et al. In Press

23 Prediction is very hard, especially about the future - Yogi Berra

24 Background  Another important issue is to predict a new disease in disease-free population.  In many asymptomatic diseases, both prevalent and incident diseases are important. (in contrast, for hard outcomes such as heart attack, only incident disease makes sense)  Incident disease is less urgent so less user-friendliness is acceptable different models developed: 1) best-fitting continuous, 2) best-fitting categorical, 3) simplified categorical.  Beyond AUC. We also used AIC/BIC.

25 Background (Conti’)  We need prospective studies to develop the models.  Internal validation only using Split-sample, no external validation.  Same logistic regression --- so observed outcome among survivors.  Cutpoint for high risk group might be less important.

26 Simplified categorical model (AUC=0.69, AIC=6295, BIC=6374) Covariate Beta coefficient (standard error) Odds Ratio (95% Cl) P value Assigned score Age (0.12)1.9 (1.5, 2.4)< (0.12)3.8 (3.0, 5.8)< or older1.46 (0.14)4.3 (3.3, 5.6)< Female0.13 (0.07)1.1 (1.0, 1.3)0.051 Anemia0.48 (0.20)1.6 (1.1, 2.4)0.021 Hypertension0.55 (0.07)1.7 (1.5, 2.0)< Diabetes mellitus0.33 (0.10)1.4 (1.2, 1.7) History of cardiovascular disease 0.26 (0.10)1.3 (1.1, 1.6) History of heart failure0.50 (0.25)1.6 (1.0, 2.7)0.041 Peripheral vascular disease0.41 (0.13)1.5 (1.2, 1.9)0.0021

27 Risk prediction table for up to 10 years Total scoreEstimated Risk (%) ≤1≤1≤5≤ ≥8≥8≥50

28 Discussion  Evidence-based medicine = Science (theory) + Data + Statistics.  Risk score = Statistics + Art + Reality --- SCORED is a good example. ☺  Performed well in a variety of different settings.  Seems to provide the enhanced guidelines upon the current clinical practice guidelines.  It started be utilized in the ‘real world’.  SCORED II yet to be validated but strong consistency/ similarities observed in SCORED I and II.

29 Discussion (Conti’)  Categorization can be a bad idea (Royston et al. 2005; Greenland 1995) but is crucial for risk scoring algorithms to be useful in the real world.  More than 1 model may be justified and we can let consumers/users to choose because All models are wrong, but some are useful ---George Box  Relying on only 1 measure (e.g., AUC) can be problematic (Cook et al. 2006; Cook 2007).  Trade-offs between accurate vs. easy medical terms.  Risk scores for internet vs. physician’s office vs. Walmart can be different.

30 Current and future research  Evaluation of SCORED in vascular patients because detection of CKD in patients with or at increased risk of CVD was emphasized by a science advisory from the American Heart Association and National Kidney Foundation (2006).  Relationships SCORED with other risk scores  Testing SCORED in community settings

31 References  Bang  Bang, Vupputuri, Shoham et al. (2007). SCreening for Occult REnal Disease (SCORED). A simple prediction model for chronic kidney disease. Archives of Internal Medicine.  Bang  Bang, Mazumdar, Kern et al. (2008). Validation and Comparison of a novel prediction rule for kidney disease: KEEPing SCORED. Arch Int Med. Bang  Kshirsagar, Bang, Bomback et al. A simple algorithm to predict incident kidney disease. In Press. Arch Int Med.  Bang  Bang, Mazumdar, Newman et al. Screening for kidney disease in vascular patients. Submitted.  Building and Using Disease Prediction Models in the Real World. Roundtable discussion led by H. Bang at JSM, Utah, Slides at: Building and Using Disease Prediction Models in the Real World

32 Exposed to and used by public  Covered by the CBS Early Show (on World Kidney Day 2007)  SCORED questionnaire is posted in some health information websites  Distributed by ESRD network, KidneyTrust, Am Kidney Fund, UK Dept of Health, and UNC Kidney Center for Kidney Education Outreach Program  “Research Highlights” in Nature Clinical Practice Nephrology (2007)  Lead Story in Physician’s Weekly (2007)