1. Rethinking Homelessness Their Future Depends on it!

2. Program Evaluation Homeless Education State Coordinators Meeting February 3, 2009 Key Bridge Marriott Arlington, VA

3. Program Evaluation “Program Evaluation” and “Performance Evaluation” are comprehensive terms encompassing two critical components:  Project Design  Project Evaluation

4. Project Design - Outline  Review performance evaluation purposes and processes  Create sample outcomes and activities with criteria and measures  Assess/create your outcome measures with SMART processes

5. Performance Evaluation Performance measurement: a process that systematically evaluates whether your efforts are making an impact/change on/to the clients you are serving or the problem you are targeting.

6. Why should programs be interested in performance measurement? We are all in the business of helping people; we need to… understand whether current activities are working to achieve intended results.

7. We are all in the business of helping people; we need to… understand whether current activities are working to achieve intended results. drive program improvement and share information on effective practices with others. Why should programs be interested in performance measurement?

8. We are all in the business of helping people; we need to… understand whether current activities are working to achieve intended results. drive program improvement and share information on effective practices with others. acknowledge that high-performing programs receive credibility/funding especially in competitive processes.

9. Creating SMART Objectives With Measurable Outcomes S pecific M easurable A ctivity-oriented R ealistic/feasible T ime-oriented

10. Building Blocks of Performance Measurement  Inputs  Activities  Outputs  Outcomes

11. Building Blocks of Performance Measurement  Inputs include resources dedicated to, or consumed by the program— e.g., money, staff and staff time, volunteers and volunteer time, facilities, equipment and supplies.

12. Building Blocks of Performance Measurement  Activities are what the program does with the inputs to fulfill its mission– e.g., providing school supplies or staff training, conducting a tutoring or summer program.

13. Building Blocks of Performance Measurement  Outputs are the direct products of program activities, usually represented in terms of the volume of work accomplished— e.g., number of students served and the number of staff trained— and are often confused for outcomes.

14. Building Blocks of Performance Measurement  Outcomes are benefits or changes among clients during or after participating in program activities. Outcomes relate to measurable change in student knowledge, behavior, skills, conditions, or other attributes. Outcomes impact the target you are addressing, which for us in generally the kids we serve.

15. Building Blocks of Performance Measurement  Outcomes (continued) How does the activity that is engaged in impact on the target?

16. Performance Measurement Process Activities Training Tutoring Summer program Field trips School supplies Outputs # staff trained on MV # of students in tutoring # of students in summer program # of students receiving school supplies Outcomes 30% more identified 40% increase TAKS 25% incrs reading retention 50%increased attendance How do we document our efforts? What did our efforts achieve? Inputs Programs Infrastructure Staff Partners Supplies Should we adjust how we spend our resources? Should we add or change use of resources to expand our impact?

17. Outputs vs. Outcomes An outcome is: An output is:  Focused on what the student will gain/how will change  Focused on what the program will do (activity) to achieve the outcome.  A way to measure the student-level impact with clear targets and methods for measuring change.  A way to quantify the frequency and intensity of the activity.  Reasonably attributable (a result of) to an output or outputs  Specific to the activity described  Meaningful and attainable.  Feasible and attainable. Be mindful to distinguish between outputs and outcomes. If outcomes show the program works, then outputs are needed to understand how to replicate results

18. Outcome, Output or Neither?  School supplies distributed to 150 campuses  90% of homeless students pass the state assessments  75% of all LEAs’ staff will be MV trained  40% of homeless students improve reading by one grade level  Homeless student attendance improves by 50% Examples Answers Output Outcome Output Outcome

19. Linking Outputs to Outcomes Outputs Outcomes 75% of homeless students who attend after school tutoring will increase reading by one grade level Homeless students’ attendance will increase by 50% statewide 100 % of students in homeless situations are enrolled immediately 95% of homeless students in LEAs attend after school tutoring 100 percent of homeless students receive school supplies and uniforms LEA staff statewide staff are MV trained

20. Collecting the Necessary Data Can you collect the data you need in order to prove your outcome?

21.  What data will you need to collect?  How will you collect it?  What system will you put in place to collect the data?  What criteria will tell you that you are on the right track?  When do you need to begin collecting data? Collecting the Necessary Data

22. Budget Support and Alignment  Do large categories and line items support activities?  How will you know if your money is spent in the most effective way?  How will you stay on top of anticipated expenses?  What will you do to build a relationship with your business office?  Are there business office practices that make it difficult for you to work from current figures, or know what invoices are paid?

23. Project Design: Summary 1.Need (with baseline reference) 2.Objective(s) 3.Anticipated Successful Target Outcome(s) 4.Timeline(s) 5.Inputs 6.Activities 7.Anticipated Outputs

24. Project Design: Summary See sample schedule for converting this project design into a planning and implementation document.

25. Project Evaluation: Summary Objective(s) Actual Outcome compared to Anticipated Target Outcome Activity(ies) Criteria used to measure activities

26. Project Evaluation: Summary Actual Outputs compared to Anticipated Outputs Analyze Status (mid-year and end of the year) What worked? What didn’t? Changes What will you change to more closely align anticipated outcomes to actual outcomes – mid year or for next year?

27. Project Design and Evaluation: Developing M-V Outcomes Objective: What do you hope to accomplish? Outcome (measurable): How will you know you’ve accomplished it? What will tell you that you have been successful?

28. Project Design and Evaluation: Developing M-V Outcomes Activities: What activities will you put in place to accomplish your outcome? Criteria: What criteria will you look at to see if the activities are being completed?

29. Project Design and Evaluation: Developing M-V Outcomes Measure/Output: What will be the measure (output) of each criteria/activity to tell you that you that you are on track to meet your outcome?

30. Project Design and Evaluation: Developing M-V Outcomes See sample Project Design and Project Evaluation handout developing a statewide M-V outcome for Florida.

31. Project Design and Evaluation: Group Activities See sample M-V outcome activity sheets, or develop outcomes selected by the group.

32. Special Thanks To: Brooke Spellman & Michelle Abbenante For sharing their knowledge and expertise regarding program evaluation, and for granting their permission for to use/revise much of their presentation delivered at the NAEH Conference on Ending Family Homelessness - February 8, 2008

33. Nothing will work if YOU don’t work!