1. Conducting Successful Program Evaluations: Researchers Reveal Their Insights Sue Palsbo, PhD Thilo Kroll, PhD NRH Center for Health & Disability Research

2. Why Do an Evaluation? Does the program work? Why? How? Who benefits? Who loses? Who pays? How does it compare to other programs?

3. Evaluation Process Analyze data Determine the Outcome Collect data State the Performance Measure

4. Who Cares? Interested in different outcomes Different outcomes require different methods Different methods require different data sources

5. Funding Considerations Public or private? Contracts or grants? Single source or multiple? Lead time Scope and /or size Funding organization’s priorities

6. Diverse Stakeholders Diverse interests for outcomes Requires diverse set of measures quantitative qualitative Blended together for total picture

7. Evaluation Types Formative –Immediate feedback allows program modifications and mid-course corrections Summative –Post-program perspective –Comprehensive overview of outcomes –Total impact of the intervention or program

8. Design Considerations Unit of analysis –Person, Provider, State Funding for the study Single vs. multi-method approach Short-term vs long-term outcomes Legacy impacts e.g. sustainability and duplication

9. How to Select the Evaluators Your evaluation identifies the questions and methods; these indicate the skill set you need In-house or external? Sole source contracting Competitive bidding

10. Example

11. Stakeholders

12. Evaluation Funding

13. Evaluation Design Consortium model Prospectively designed Contemporaneous (formative) Highly leveraged 3-year cumulative

14. MnDHO Goals 1. Create and maintain satisfaction 2. Promote overall well-being of enrollees 3. Meet cost and utilization goals.

15. Measurement Domain 1. Satisfaction 2. Quality of care 3. Utilization and patterns of care 4. Costs and rate setting

16. Measurement Selection Criteria 1. Use existing data 2. Avoid duplication of information. 3. Avoid recall bias. 4. Minimize respondent burden.

17. Preliminary Findings From Mixed-Methods Evaluation Longitudinal CAHPS®-based survey –At MnDHO enrollment –1 year post-enrollment Focus groups –MnDHO enrollees –Eligible people who chose not to enroll

18. Survey Content General satisfaction Care coordination experience Access to a wide variety of health and long- term-care services Self-directed care Quality of interactions with health care providers Quality of interactions with MA or AXIS staff

19. Longitudinal Survey 100 MnDHO participants – 35 enrollees have completed baseline and follow-up surveys. Baseline –Health care experiences in the year before enrollment. Follow-up –Health care experiences in MnDHO.

20. 4 Focus Groups October, 2002 2 MnDHO enrollees 2 fee-for-service Medical Assistance program

21. Care Coordination. In the year before / after you enrolled in AXIS/UCare Complete, did anyone help manage the health care services you received from different doctors, nurses, therapists, PCAs,or equipment providers?

22. Focus Group Statements The MnDHO / AXIS Experience: –“She’ll (health coordinator) line me up with my appointments and stuff. She’ll find my therapists and my physical equipment…, all that stuff.” –“My stress level has been relieved somewhat. I’m able to focus on more vocational and future issues, as opposed to the day-to-day healthcare issues……”

23. Focus Group Statements The fee-for-service experience: “I spend most of my time on the phone calling people, setting up appointments, or trying to get services myself, and that is very tiring. I get exhausted because I do have MS.” “Our experience was so bad we had to hire our own coordinator. It seems like if the person is stable……, people don’t return their calls, people don’t seem to care……”

24. Self-direction in health care. In the year before / after you enrolled in AXIS/UCare Complete, were you involved as much as you wanted in making decisions about your health care?

25. Focus Group Statement “My healthcare coordinator, she always involves me. It is our lives. They can give us all the information, but ultimately it comes down to us, what we want to do…... it comes to us making the final decision…... The healthcare coordinators are always there to give you the pros and cons.”

26. Recommendations Its worth the time and money to do a program evaluation Do a formative evaluation so you can make changes in real time Important lessons from a summative evaluation can tell you what worked and what didn’t, to improve the next effort Mixed methods provide more information together than a single method