Gall C, Katch A, Rice T, Jeffries HE, Kukuyeva I, and Wetzel RC STANDARDIZED MORTALITY RATIOS FOR THE MOST FREQUENT CARDIAC SURGICAL PROCEDURES UTILIZING PEDIATRIC INDEX OF CARDIAC SURGICAL INTENSIVE CARE MORTALITY Gall C, Katch A, Rice T, Jeffries HE, Kukuyeva I, and Wetzel RC

OBJECTIVES Verify the performance (including calibration and discrimination) of PICSIM on the most frequently performed cardiac surgical cases in the VPS database. Use PICSIM to evaluate post-operative mortality outcomes by calculating Standardized Mortality Ratios.

BACKGROUND The Pediatric Index of Surgical Intensive Care Mortality (PICSIM) score was developed to support robust risk-adjustment for the pediatric cardiac surgical population cared for in pediatric Intensive Care Units (ICUs) [1]. While PICSIM performs well in a general pediatric cardiac ICU surgical population, and outperforms PIM 2, PRISM 3, and STAT severity of illness scores overall [2], its performance in predicting post-operative outcomes for individual diagnostic categories is unclear.

BACKGROUND cont. Development of a risk-adjustment tool that enables a consistent comparison of mortality outcomes both for the overall pediatric cardiac surgical sample as well as for cohorts of children receiving the same cardiac surgical interventions will enable surgical techniques, pre-, peri-, and post-operative care strategies to be effectively evaluated. To explore the use of PICSIM to evaluate the ICU outcomes of children following cardiac surgery and post-operative ICU care, data from the Virtual PICU Systems, LLC (VPS) database were selected to inform this question.

BACKGROUND cont. VPS, LLC is the largest network in pediatric critical care. Formed by the National Association of Children’s Hospitals and Related Institutions (NACRI, now CHA), Children’s Hospital Los Angeles, and Children’s Hospital of Wisconsin to develop a tool for prospective data collection. VPS database has nearly 1,000,000 ICU encounters. Trained clinicians collect and enter VPS data. VPS performs quarterly inter-rater reliability (IRR) testing & quality validation IRR of VPS data is consistently above 90%

METHODS This study was waived for approval by Seattle Children’s IRB. Cardiac surgical patients were defined as patients with a cardiac diagnosis that had an STS (Society of Thoracic Surgeons) registered surgical procedure. Inclusion criteria for PICSIM: ICUs that managed postoperative pediatric cardiac surgical patients. Patients that have estimates for both the Pediatric Index of Mortality 2 (PIM 2) and the pediatric risk of mortality (PRISM 3) scores. Cardiac surgical patients that had surgery either before or after admission to the ICU. Patients with a STAT score that could be derived from the index procedure.

METHODS Step 1 Find the most frequently performed cardiac surgical procedures. We focused on the 25 most frequently performed cardiac surgical procedures by volume. These procedures represented 66% of all cardiac surgical procedures in the VPS database.

METHODS cont. Step 2 Evaluate PICSIM model accuracy via AUC. One way to evaluate model accuracy is by finding the area under the curve (AUC) of the Receiver Operating Characteristic (ROC) curve. The Receiver Operating Characteristic Curve shows how well the model correctly and incorrectly classifies patient deaths. Accuracy is measured by the Area Under the ROC Curve: an area of 1 represents perfect classification; an area of 0.5 shows a classification made via random classification of patients into the two categories.

METHODS cont. Step 3 Evaluate PICSIM model calibration via Hosmer-Lemeshow. One way to assess model calibration is via Hosmer-Lemeshow (HL). HL evaluates how close the observed and predicted values are, for a given group of patients. A specified number of (equally-sized) groups is chosen a priori and patients are assigned to the groups according to their predicted mortality rates.

METHODS cont. Step 4 Evaluate PICSIM model calibration via Standardized Mortality ratios (SMRs). Another way to determine how well a model is calibrated is to analyze the SMRs and confidence intervals of the 25 surgical procedures. An SMR is a ratio of the total number of observed deaths for a given procedure to the total number of expected deaths for the same procedure. Across procedures, we expect the SMRs to fluctuate around 1. If a majority of the SMRs deviate from 1, this suggests that the model has demonstrated 'drift' and is not precise. SMRs were calculated to compare observed deaths with expected deaths, using PICSIM for prediction. SMRs were computed for each of the top 25 cardiac surgical index procedures by volume, as well as over all top 25 procedures. Confidence Intervals (CI) at the 95% significance level were calculated for each procedure and results were plotted (in the figure)

RESULTS 66 VPS ICUs participating (July 2009 - March 2014) were included in the analysis (n=20,076). PICSIM showed satisfactory discrimination (AUC=0.89) & calibration via HL (HL: P=0.181) and SMR criteria. The combined SMR for all top 25 procedures was 0.94 (95% CI: 0.86-1.02). The SMRs and confidence intervals for each of the top 25 most frequently performed cardiac surgical procedures by volume is shown in the figure; procedures were ranked by most to least frequent, by volume. Two procedures had no associated deaths; SMRs could not be generated. 18 of the remaining 23 procedures and the aggregate CIs included 1.0.

CONCLUSIONS The results demonstrate that PICSIM’s capacity to predict mortality for the most common surgical categories is robust. PICSIM SMRs represent a reasonable approach for comparison of mortality among ICUs for select cardiac surgeries. Further development of PICSIM performance will enable us to assess the impact of site, unit and care delivery team characteristics on pediatric ICU mortality for children and adults requiring cardiac surgical intervention..

REFERENCES Jeffries HE, Soto-Campos G, Katch A, Gall C, Rice TB, Wetzel RC: Pediatric Index of Cardiac SurgicalIntensive Care Mortality (PICSIM) risk score for pediatric cardiac critical care. Pediatric Critical Care Medicine. 2014. In review. Wetzel RC, Jeffries HE, Katch A, Gall C, Kukuyeva I, Rice TB. Validation of the Pediatric Index of Cardiac Surgical Intensive Care Mortality (PICSIM) score. Accepted for poster presentation at the Pediatric Cardiac Intensive Care Society Annual Meeting in Miami, FL, December 11-12, 2014. LaRovere JM, Jeffries HE, Sachdeva RC, Rice TB, Wetzel RC, Cooper DS, Bird GL, Ghanayem NS, Checchia PA, Chang AC, Wessel DL. Databases for assessing the outcomes of the treatment of patients with congenital and pediatric cardiac disease – the perspective of critical care. Cardiol Young 2008;18(Suppl 2): 130-36

PICSIM SMRs for VPS Cardiac Procedures