1. Performance Improvement Projects: Validating Process, Tips, and Hints Eric Jackson, MA Research Analyst October 19, 2009

2. Overview of the Training Meeting Logistics Validation Process CMS Protocol Activities Tips and Hints Resources Questions and Answers Other Training Topics

3. Meeting Logistics Using GoToWebinar™ How to ask questions to team during the presentation –Through the questions section in the GoToWebinar application –You can use the GoToWebinar raise your hand option We will review submitted and additional questions at the end of the presentation

4. First the Basics of GoToWebinar: The GoToWebinar Attendee Interface 1. Viewer Window 2. Control Panel

5. Validation Process for PIP Request desk materials –All PIPs currently in progress or completed since the last EQR should be sent –Example: NCQA Activity Forms After receiving desk materials, three PIPs are selected for validation Perform CMS Protocol Activities by reviewing the desk materials we received

6. CMS Protocol Activities Activity One: Assessing the Study Methodology Activity Two: Verify Study Findings (optional) Activity Three: Evaluating Overall Validity and Reliability of Study Results

7. Activity One: Assessing the Study Methodology Ten steps with related review questions and scoring –27 components / questions and 116 possible points Specific questions can be found in the CCME PIP Validation Overview document

8. Activity One: Assessing the Study Methodology (cont) Each component is scored by the degree they meet the protocol requirements –MET: component fully meets the criteria without any issues –PARTIALLY MET: component meets some but not all of the criteria –NOT MET: component fails to meet most or all the criteria –NA: component does not apply to the project being reviewed

9. Activity One: Step One Review the Selected Study Topic(s) –How was the topic of the study selected –Is the topic appropriate 3 questions with 7 possible points

10. Activity One: Step Two Review the Study Question(s) –Is the study question documented Single question with 10 possible points

11. Activity One: Step Three Review the Selected Study Indicator(s) –Are the study indicators objective and clearly defined –Are the study indicators appropriate 2 questions with 11 possible points

12. Activity One: Step Four Review the Identified Study Population –Is the study population well defined –Is the population being captured correctly 2 questions with 6 possible points

13. Activity One: Step Five Review the Sampling Methods –Applies only if sampling was used in the project –Is a valid sampling method being used –Is the sample large enough 3 questions with 20 possible points

14. Activity One: Step Six Review the Data Collection Procedures –Are data sources clearly specified –Was a data analysis plan established in the documentation –Did the instruments used for data collection provide consistent, accurate data 6 questions with 18 possible points

15. Activity One: Step Seven Assess Improvement Strategies –Are reasonable interventions being planned and implemented Single question with 10 possible points

16. Activity One: Step Eight Review the Data Analysis and Interpretation of Study Results –Was the data analysis plan followed –Are numerical results presented accurately and clearly 4 questions with 17 possible points

17. Activity One: Step Nine Assess Whether Improvement is “Real” Improvement –Is there any documented, quantitative improvement –Was the same methodology used for baseline and repeated measurement 4 questions with 12 possible points

18. Activity One: Step Ten Assess Sustained Improvement –Is sustained improvement demonstrated in the repeated measurements of the study Single question with 5 possible points

19. Activity Two: Verify Study Findings Optional Activity Requires the plans to produce the data that generated the results of the PIP Review would try to mimic the documented results from the data received from the plan

20. Activity Three: Evaluating Overall Validity and Reliability of Study Results Scores are summarized Validation Finding is calculated o VF = (score project received / total possible points) o Multiply by 100 to report as a percentage Final Audit Designation is assigned

21. Activity Three: Evaluating Overall Validity and Reliability of Study Results (cont) Score ranges for the Final Audit Designation AUDIT DESIGNATION POSSIBILITIES High Confidence in Reported Results Little to no minor documentation problems or issues that do not lower the confidence in what the plan reports. Validation findings must be 90%–100%. Confidence in Reported Results Minor documentation or procedural problems that could impose a small bias on the results of the project. Validation findings must be 70%–89%. Low Confidence in Reported Results Plan deviated from or failed to follow their documented procedure in a way that data was misused or misreported, thus introducing major bias in results reported. Validation findings between 60%–69% are classified here. Reported Results NOT Credible Major errors that put the results of the entire project in question. Validation findings below 60% are classified here.

22. Tips and Hints Remember the Study Questions in your project documentation! Document the numerators and denominators along with the rates Double check rate calculations

23. Tips and Hints (cont) Check your terminology o Percent change is not the same as percentage point change Example: Baseline = 55% Re-measurement = 95% Percent Change = (95% - 55%) / 55% = 73% change Percentage Point Change = 95% - 55% = 40 percentage point change

24. Resources Go to: www.thecarolinascenter.orgwww.thecarolinascenter.org –Search for SC EQR –Training materials should be one of the top results CCME PIP Validation Overview document CMS PIP Protocol o “Validating Performance Improvement Projects: A protocol for use in conducting Medicaid external quality review activities”

25. Question and Answers

26. Additional Training Topics

27. Please Remember the Evaluation! It will display after you end the webinar.