1. Performance Evaluation: From Grant Application and Fed eral Regulations to Program Performance Results Carol Bergquist, Ph.D. Hannahville Indian Community Project VISIONS 906-723-2580 Carol.Bergquist@HICVisions.org Michael Shoemaker, M.A., CRC, LVRC, CPM Utah State Office of Rehabilitation CANAR Annual Conference Salt Lake City, Utah November 2015

2. What is Program Evaluation? A Reflective and Systematic Process… that focuses on questions of interest or concern regarding activities, characteristics or outcomes of a program… where information is collected, analyzed and interpreted… to assist in making data informed decisions.

3. Program Evaluation Program Evaluation is everybody’s business. It is inquiry: an act of asking for information. It should lead to ways to change and improve the program. It is likely something you are already doing and are not aware of it. (improving client showing to appointments). The Truth: You may be the best evaluator because nobody knows your program better than you. There is a time and place for both internal and external evaluation.

4. Purposes of Program Evaluation Demonstrate program effectiveness to funders Improve the implementation and effectiveness of programs Better manage limited resources Document program accomplishments Justify program funding Satisfy ethical responsibility to clients to demonstrate positive and negative effects of program participation (accountability and transparency) Document program development and activities to help ensure successful replication

5. Program Evaluation is Part of Continuous Quality Improvement Stewhart’s PDCA Cycle For Change

6. PROGAM EVALUATION MODEL

7. Program Evaluation Activities Customer Satisfaction Surveys Case File Reviews Needs Assessments Economic Impact Return on Investment Process Evaluation Authorizations and Payments Reviews Impact and Outcome Evaluation

8. Performance Measures of Quality WIAWIOA Entered Employment Rate Employment Rate (Q2 post- exit) Now Employed Q-2 after exit Employment Retention Rate Employment Rate (Q4 post- exit) Now Employed Q-4 after exit Average Earnings Median Earnings Median earnings Q-2 after exit only Credential Rate Will require follow-up for 1 yr after exit Measurable Skills Gain Real Time Measure, note exit based. How to document skills gain?

9. Performance Measures of Quality Rehabilitation Rates (percent rehabilitated vs. those that were not rehabilitated) Employment Serious Vocational Impediment Timeliness of Determination Timeliness of Evaluation and Planning Phase Accuracy of Determination, Service, Program Compliance and Fiscal Transactions

10. Common Areas for Performance Measures CUSTOMER SERVICE FINANCIAL POLICY AND PROCEDURE EMPLOYEES

11. What Program Evaluation Activities are Required by RSA for Tribal VR Projects? Funded Grant Application Financial Draw Downs 6 Month Report to RSA Year 1 Annual Reports to RSA Years 1 – 5 5 Year Report to RSA at End of Grant MOU with State VR Agencies New Requirements under WIOA Optional but Best Practice Participation in Monthly RSA Conference Calls Communication with Project Officer Participation in Annual and Mid-Year CANAR Conferences Participation in Annual Project Directors’ Conference

12. A Detailed Look at Project Visions AIVRS Grant Application – Identify program evaluation activities as described in your application and others you are currently doing Project VISIONS Program Evaluation Description Logic Model Management Work Plan – excerpt

13. Client Outcome Annual Reports DataOps Electronic Client Database DataOps Ad Hoc Reporting - Disability Report Example

14. Satisfaction Surveys Client Satisfaction Survey Client Satisfaction Survey Instrument Client Satisfaction Report Excerpt Partner Satisfaction Survey Partner Email Invitation Partner Satisfaction Report Excerpt

15. Success Stories Bryan – Client Success Story Betty and Al – Client Success Story Joe – Client Success Story 3 Fires Supportive Housing – Project Initiative Success Story

16. Components to Inform Planning for 2016-20 Grant Application – External Evaluator Client In-person Interviews – Power of VR Services Abstract Invitation to Stakeholders for Phone Interview Interview Questions for Stakeholders Stakeholder Interview Abstract Stakeholder Interview Results & Recommendations Evaluation of Operational Procedures and Staff Roles Abstract

17. Additional Methods Instrumented Group Interview (IGI) Case File Review – RSA Corbridge Model Project VISIONS Abstract CANAR Conference Presentation Description Coffee Café Model Focus Groups and Talking Circles Client Long Term Follow-up “Where have all our clients gone?”

18. Guiding thoughts about Program Evaluation “Data are our friends” “What gets measured get’s done” “ Innovation or New Idea - Pilot test and measure results” “Given little or no information - people make up the answers”

19. Resources Suzanne Malson – New Beginnings Brochure DataOps – Information – Catherine and George Laguna

20. SMALL GROUP DISCUSSION  How will you apply what you learned today?  Did you think about a program evaluation method or measure you hadn’t considered before?  Will you look more deeply at a particular project component? If so, what component?  Will you develop a plan of how you can use something from today’s presentation?

21. “Megwetch” Thank You for Participating in our Presentation

22. A Community of Practice The Summit Group http://vocational-rehab.com/http://vocational-rehab.com/ Free Email Listserv with 360 VR professionals The 9th Annual Summit on Vocational Rehabilitation Program Evaluation, Fall 2016, Richmond, Virginia The Rehabilitation Program Evaluation Network http://vocational-rehab.com/rpen/what-is-rpen/