Presentation on theme: "Development and Evaluation of CMS-HCC Concurrent Risk Adjustment Models Presented by Eric Olmsted, Ph.D. Gregory Pope, M.S. John Kautter, Ph.D. RTI International."— Presentation transcript:
1 Development and Evaluation of CMS-HCC Concurrent Risk Adjustment Models Presented by Eric Olmsted, Ph.D. Gregory Pope, M.S. John Kautter, Ph.D. RTI International Presented at Academy Health June 26, 2005411 Waverley Oaks Road ■ Suite ■ Waltham, MA
2 Concurrent Risk Adjustment Introduction OverviewRisk Adjustment/HCC ModelConcurrent v. ProspectiveProject Goals and ChallengesModel DevelopmentModel EvaluationSummary and ConclusionOverview – what is population based risk adjustment, what are some of its potential uses?
3 Overview Risk Adjustment Introduction Population Risk Adjustment:The process by which the health status of a population is taken into consideration when setting capitation rates or evaluating patterns or outcomes of practiceRisk adjustment is used to create “apples to apples” comparisonsRisk adjustment removes the effect of health status differencesReduces or eliminates the problem of selection
4 Overview Risk Adjustment Model Model calibrated on 5% national sample of Medicare fee-for-service beneficiariesExpenditures are regressed on HCC (& demographic) risk markers to estimate incremental impact of each diagnosis on expendituresAnnualized Expenditures = Σαi + Σβi + Єiαi = demographic markersβi = HCC markersRisk markers are used to predict health expenditures.Additive not categorical model.
5 Overview HCC Model Full model contains 184 HCCs CMS-HCC model contains 70 HCCsCMS-HCCs:Cover a broad spectrum of health disordersHave well-defined diagnostic criteriaExclude highly discretionary diagnosesInclude conditions with significant expected health expendituresDemographic MarkersAge, Gender, Medicaid, & Originally Disabled StatusEnsure means for demographic populations correctly estimatedThus, the CMS-HCC model is a "selected significant diseases" model that focuses on adjusting for risk associated with selected high-cost diagnoses; it does not incorporate all diagnoses.
6 Overview Concurrent vs. Prospective Prospective risk adjustment uses current year diagnoses to predict next year’s expendituresChronic conditions are more importantConcurrent risk adjustment uses current year diagnoses to predict this year’s expendituresAcute conditions are more importantConcurrent also known as ‘retrospective’.
7 Overview Concurrent vs. Prospective AMI:Prospective Coefficient = $1,838Concurrent Coefficient = $12,21163% of HCC coefficients with >$1,000 differenceR-squared:ConcurrentProspective70 Total Coefficients23% with greater than $5,000 difference63% with >1,000 difference
8 Project Goals Concurrent Risk Adjustment Project Goals: Develop payment model for Pay-for-Performance demonstrationDevelop model for use in profiling physiciansMake model consistent with prospective CMS-HCC model that is being used for MA payment, and its data collection requirementsImprove prediction across the spectrum of patient costMA = Medicare Advantage
9 Concurrent Modeling Challenges Applied standard HCC modelResulted in negative predictions and coefficientsConcurrent HCC coefficients fit high-cost beneficiariesThis forces age-sex coefficients down and they sometimes become negativeAge-sex coefficients reflect the average beneficiaryNegative age-sex coefficients can lead to negative predictions
14 Project Goals Model Selection Criteria for Model SelectionAvoid negative predictions, which lack face validityAvoid negative coefficientsMaintain correct age-sex means to prevent age and sex selection by providersPrefer simple models to complex modelsSelect model with good ‘performance’ among model evaluation measuresCMS and physicians evaluate models based on ease of use and interpretability.
15 Model Development Sample Statistics 1.4 million FFS Medicare beneficiaries with mean expenditures of $5,214Beneficiaries with at least one CMS-HCC represent 61% of the population, but provide 94% of all Medicare expendituresBenes with one subset of HCCs represent 24% of population but account for 72% of expenditures.
16 Model Development Standard Models Full HCC Model184 HCCs & demographicsCMS-HCC Model70 HCCs & demographicsInteraction and Topcoding ModelsCreated disease and demographic interactions to tease out high-expense beneficiariesCreated topcoded models to reduce impact of outliers
17 Model Development Alternative Models Nonlinear ModelsLog modelSquare root modelSplit Sample ModelsDesigned separate models for populations with different expected expendituresCommunity/InstitutionalHigh Cost/Low Cost HCCCatastrophic HCCMulti-stage models including two-part and four-part logit modelsSimple two-stage model with demographic multipliersSegmentation
18 Model Evaluation Standard Model Results Full HCC model suffers not only from 30% negative predictions, but also contains negative HCC coefficientsCMS-HCC model explains 92% of the variation that the Full HCC model explainsCMS-HCC model eliminates negative HCC coefficientsCMS-HCC model has only 10% negative predictionsInteraction and Topcoding ModelsDid not sufficiently reduce negative predictionsIntroduce the models.Note: Full HCC model suffers not only from 30% negative predictions, but also contains negative HCC coefficients.Note: CMS-HCC model explains 92% of the variation that the Full HCC model explains. HMOs are currently collecting the information for the CMS-HCC model. CMS-HCC model eliminates negative HCC coefficients.Notice that R-Squared within .04 for all modelsCPM showed similar results
19 Model Evaluation Alternative Model Results Nonlinear ModelsLog model and square root model did not produce reasonable predictionsSplit Sample ModelsSplitting sample by community/institutional did not eliminate negative predictionsSplitting sample by disease burden eliminated negative predictionsLog Model – Top 1% predicted expenditures = 1.6 millionSquare Root Model - First three deciles predicted expenditures twice as much as actual.
20 Model Evaluation Measures of Model Performance R2 within .04 for all modelsR2 did not differentiate modelsPredictive Ratio = Average of model’s predictionsAverage of actual expendituresWhere each of the two averages is taken over the individuals in the subgroupPredicted expenditure decilesNumber of HCCs for a beneficiaryTo evaluate the models across the full spectrum of beneficiaries
21 Model Evaluation Predictive Ratios by Expenditure Percentile Closer to 1.00 equals a better model fit.High-low cost model predicts well.As does four part model.
22 Model Evaluation Predictive Ratios by Expenditure Percentile 1.0 is ideal.Model 8 tracks well, followed by Model 12.CMS-HCC model performs poorly for lower deciles.
23 Model Evaluation Predictive Ratios by # of HCCs Again, closer to 1.00 equals a better model fit.High-low cost model predicts well.As does four part model.
24 Model Evaluation Predictive Ratios by # of HCCs Again, closer to 1.00 equals a better model fit.
25 Concurrent Model Evaluation Model Summary High Cost & Catastrophic Models performs wellSome face validity problems with splitting HCCs into “high-cost” and “low-cost”Still has negative predictionsFour Part Model also performs wellComputationally advanced and hard to interpret intuitivelyNo negative predictionsSample Segmentation Model performs very wellAlso computationally advancedTwo-Stage Multiplier Model performs adequatelyNo face validity problems
26 Concurrent Model Evaluation Conclusion Nonlinearities cause difficulties in concurrent risk adjustment model calibrationNegative coefficients and predictionsThese difficulties can be addressed with:Nonlinear modelsSplit sample modelsBut nonlinear/split sample models add complexityDifficult to estimateDifficult to interpretAdds instabilityTwo-Stage Multiplier ModelGood face validity, avoids negative coefficients and predictionsSimpler to estimate and interpret