1. Proposed Changes to the OPTN Transplant Program Outcomes Review System
OPTN/UNOS Membership and Professional Standards Committee

2. What problem will the proposal solve?
OPTN method for reviewing transplant program outcomes is thought to have contributed to members’ increasingly conservative behavior
Transplant programs are becoming more selective
Organs accepted to transplant
Patients listed for transplant
The OPTN’s current method for monitoring transplant program outcomes is thought to have contributed to members becoming increasingly selective in the organs they will accept to transplant and the patients that are listed for transplant. This is particularly detrimental to patients who remain on the waitlist, or who aren’t added to the waitlist at all, and who would be better served by a transplant.
If questions about the current system, please refer to Additional Information Slide #12
The MPSC currently reviews a transplant program if it has a higher hazard ratio of mortality or graft failure than would be expected for that transplant program. The criteria used to identify programs with a hazard ratio that is higher than expected will include either of the following:
1. The probability is greater than 75% that the hazard ratio is greater than 1.2.
2. The probability is greater than 10% that the hazard ratio is greater than 2.5.
The lines that define these criteria are shown here. Plotting a program’s estimated hazard ratio versus the expected number of events, as show on this slide, the MPSC will review a program if it in the shaded area (above either threshold).
Additionally, and not captured on this slide- For programs performing 9 or fewer transplants in a 2.5 year period, the MPSC will review a transplant program if the program has one or more events in a 2.5 year cohort.
OPTN Bylaws Appendix D.11.A (Transplant Program Performance) contain these details and further explain the OPTN’s transplant program performance review system.

3. What are the proposed solutions?
Tier 1: Expanded Program Performance Review
>60% probability that hazard ratio > 1.75
100% MPSC engagement
Tier 2: Routine Program Review
>60% probability that hazard ratio > 1.25 & the program is not included in Tier 1
50% MPSC engagement
Tier 3: Routine Program Review
Hazard ratio > 1.0 & the program is not included in Tier 1 or Tier 2
10% MPSC engagement
 Tier 4: Performing As Expected or Better
Hazard ratio ≤ 1.0
0% MPSC engagement
To address these problems, the MPSC is proposing a new, four-tiered review system that relies on the analysis of hazard ratios that compare the program’s observed and expected post-transplant graft and patient outcomes:
The top tier, called “Expanded Program Performance Review Tier 1” is defined by a greater than 60% probability that the program’s graft or patient survival hazard ratio is greater than The MPSC will review 100% of the programs in this tier.
The next tier, “Routine Program Review Tier 2” is defined by a greater than 60% probability that the hazard ratio is greater than 1.25, and the program is not included in Expanded Program Performance Review Tier 1. The MPSC will randomly select fifty percent of the programs in this tier for review.
“Routine Program Review Tier 3” is defined by a hazard ratio that is greater than 1.0, and the program is not included in Expanded Program Performance Review Tier 1 or Routine Program Review Tier 2. Ten percent of the programs in this tier will be randomly selected for MPSC review.
The bottom tier is called “Performing as Expected or Better Tier 4” and is defined by an observed hazard ratio of 1.0 or less. This tier will not prompt MPSC review.

4. What are the proposed solutions?
Here is a graph that illustrates each tier, and the “review space” established by each.

5. What are the proposed solutions?
Here is a graph that illustrates each tier, and the “review space” established by each. It is truncated to focus on the top three tiers that will prompt MPSC review.
In these figures, the y-axis represented the program’s observed hazard ratio and the x-axis represents the program’s number of expected graft loss events; “x” represents an expanded program performance review; “+” represents a routine program review; and “o” represents programs that would not be reviewed through this system. Please note that there are a few programs shown with a “+” in tier 4; this happens because the figure displays results for the graft survival outcome and these programs had higher levels of underperformance for patient survival. In these few instances, and because a program may be reviewed based on its patient survival outcomes or its graft outcomes, the patient survival outcomes caused their program component to receive a higher-level review.
As Expected or Better Tier
0% Probability of MPSC Review
HR ≤ 1

6. What are the proposed solutions?
Organ
New CMS (O/E>1.85)
New CMS Condition-Level
Current MPSC
Post MPSC filters
Expanded Reviews Tier 1
Post MPSC filters
Routine Reviews Tier 2
Routine Reviews Tier 3
Total Routine Reviews
Total Reviews- All Tiers HR 6 2 12 7 1
4.5
3.7
11.5
5.7
13.5
6.7 KI 17 8 32
9.5
8.1
25.1
16.5
33.1
18.5 LI 3 14 4 10 5
4.9
14.9
15.9 LU
6.5
1.4
8.5
5.4
ALL 29 13 70 30
40.5
20.5
19.5
16.6 60
37.1 72
40.1
Using data from the December 2015 program specific reports, this table reflects approximately how many programs would be expected in each tier. If you are having trouble seeing this table on the screen, please reference the proposal which also includes these data.
This table includes the approximate number of reviews that the MPSC would actually undertake after other operational rules (filters) are applied. Additionally, for the sake of comparison, this table also includes the number of programs that may be reviewed as a function of CMS thresholds, and the number of programs the MPSC would review under the system currently in the Bylaws.
If questions about operational rules filter:
• Is the program active?
• Is the program already under review?
• Was the program released in the last 2 meeting cycles?
• Has the program had a death or graft failure since the most recent release from review?

7. What are the proposed solutions?
MPSC and Member Actions
Expanded Program Performance Review Tier 1 – members will be expected to complete an “expanded survey”
Routine Program Review Tiers 2 & 3 – members that are randomly selected will be expected to complete an “initial survey”
Programs demonstrating ongoing and appropriate quality improvement efforts will be released from further MPSC review
This proposal also modifies the MPSC’s response to programs identified for outcomes review. Programs identified in Expanded Program Performance Review Tier 1 will be expected to complete an “expanded survey.” This survey is commonly reserved currently for programs that remain under MPSC review for an extended period of time (usually longer than a year). With this approach and using a higher hazard ratio threshold, the MPSC hopes it will be able to better identify those programs having real difficulties with their transplant outcomes, and that it will be able to help those programs more quickly discover and address the root of those issues.
Programs that are randomly selected in Routine Program Review Tiers 2 and 3 will be requested to complete an “initial survey” that will be similar to the initial reviews currently undertaken by the MPSC, including a focus on the program’s quality improvement efforts. The random selection component of Tier 2 and Tier 3 is intended to address concerns in the community about the ability to assess the true level of program underperformance at these lower hazard ratio thresholds. Although the mathematics of the model used to review transplant program outcomes necessarily means that a hazard ratio greater than 1.0 indicates that the program experienced a greater number of events than what would be expected, the transplant community has indicated concerns with the model’s ability to discriminate significant differences between programs plotted close together as a result of these analyses and the lack of available data to describe all patient risk profiles.
As OPTN Bylaws require that all transplant hospitals, “develop, implement and maintain an ongoing, comprehensive and data-driven QAPI program,” reviews driven by the random selection of programs in these tiers should require minimal resources to respond as programs will primarily be reporting on ongoing efforts at their institutions. Additionally, quality improvement efforts will be a primary focus of the MPSC’s reviews that are prompted by the random selection of programs within these tiers. Programs that are able to demonstrate ongoing and appropriate quality improvement efforts will be released from further MPSC review.

8. How will members implement this proposal?
No major changes to members’ role in this process
Transplant programs are expected to be responsive to the MPSC if engaged in an outcomes performance review
Although the system to identify transplant programs for an OPTN outcomes review is completely changing, there are no major changes to the members’ role in this process. If identified for an OPTN outcomes performance review, transplant programs are excepted to be responsive to the MPSC’s request.

9. How will the OPTN implement this proposal?
Board consideration – December 2016
Programming – No
Implementation- If approved, the MPSC will use this method to identify programs for outcomes review in the June 2017 MPSC Outcomes Reports
Will require SRTR to revise its process for running the MPSC Outcomes Reports used to identify programs for outcomes review
Could be implemented together or separately from Transplant Program Performance Measure proposal
Also sponsored by MPSC this public comment cycle
Developed independently from this proposal
If public comment is favorable, this proposal is slated to be considered by the OPTN/UNOS Board of Directors during its December 2016 meeting. If the Board adopts this proposal, the MPSC will use the system outlined in this proposal to identify programs for outcomes review in the June 2017 MPSC Outcomes Reports. This will require SRTR to revise its process for running the MPSC Outcomes Reports, which it produces at the same time as the public program specific reports.
It is important to note that the MPSC is sponsoring another proposal out for public comment this cycle that proposes different changes to the OPTN’s outcomes review system- Transplant Program Performance Measures (Outcome Measures). These proposals can be implemented together to supplement the impact each proposal may have, but as they are not interconnected, there is nothing in either proposal that would prohibit them from being implemented separately or in isolation. Additionally, both of these proposals are also separate from the COIIN pilot project currently being undertaken by UNOS.

10. Questions? Jeffrey Orlowski, MS, CPTC Committee Chair
Chad Waller
Committee Liaison

11. Additional Information

12. Current MPSC Review Space
The MPSC currently reviews a transplant program if it has a higher hazard ratio of mortality or graft failure than would be expected for that transplant program. The criteria used to identify programs with a hazard ratio that is higher than expected will include either of the following:
1. The probability is greater than 75% that the hazard ratio is greater than 1.2.
2. The probability is greater than 10% that the hazard ratio is greater than 2.5.
The lines that define these criteria are shown here. Plotting a program’s estimated hazard ratio versus the expected number of events, as show on this slide, the MPSC will review a program if it in the shaded area (above either threshold).
Additionally, and not captured on this slide- For programs performing 9 or fewer transplants in a 2.5 year period, the MPSC will review a transplant program if the program has one or more events in a 2.5 year cohort.
OPTN Bylaws Appendix D.11.A (Transplant Program Performance) contain these details and further explain the OPTN’s transplant program performance review system.
OPTN Bylaws Appendix D.11.A (Transplant Program Performance)

13. Alternative Monitoring Program
Scope of Work
Community Engagement
Outcomes/ Goals
Time Frame
Task 18 – Collaborative Improvement Innovation Network (COIIN) for Transplant Hospitals
“Can we pilot an alternative approach to performance monitoring (what, when, how)?”
Pilot collaborative with ~15 voluntary kidney transplant programs to potentially redefine the methodology for transplant program review – focusing on outcomes, process, structure, and relationships. Pilot focuses on kidney, but could eventually expand to all organs.
COIIN Advisory Council
a) Increase transplantation by identifying effective practices to support increased usage of moderate to high risk kidneys.
b) Support a learning network/collaborative across participating hospitals.
c) Use real time data monitoring (dashboards, other tools) to support improvement using multiple measures of performance including outcomes, process, structural, and relationship measures.
d) Participation offers potential waiver from current MPSC monitoring and CMS mitigating factors consideration.
3 year Pilot
Year 1 – Design
Year 2 – Deployment
Year 3 - Evaluation
MPSC Outcomes Work Group
“Can we provide a solution to encourage the use of high risk kidneys by removing disincentives created by current review of 1 year graft and patient survival outcomes?”
Develop proposal to exclude high risk kidneys from MPSC evaluation of 1 year graft and patient survival in order to increase use of transplantable kidneys that currently have high discard rates. Focusing initially on kidney, but could eventually expand to other organs.
MPSC Work Group. Includes representation from MPSC and various stakeholder societies.
To evaluate if a proposal will result in an increased number of kidney transplants being performed, particularly from high risk donors.
Public feedback during spring 2016 public comment
Discuss at June 2016 Board Meeting
Public comment fall 2016
Board consideration in December 2016
Task Force to Reduce Disincentives to Transplantation
“Can we reduce the number of programs that are flagged by identifying a more clinically relevant threshold?”
Wide focus for all organs to modify the MPSC process for identifying programs for outcome review using a significant “clinical threshold” to reduce number of hospitals currently flagged.
OPTN/UNOS Board of Directors Task Force. Includes representatives of the Board, MPSC, and stakeholder societies.
To identify more clinically appropriate thresholds in order to reduce the number of currently flagged programs.
Submit findings to MPSC and present proposal to June 2016 Board meeting for action.