1. Using HMIS for Program Evaluation September 13-14, 2005 St. Louis, Missouri Sponsored by the U.S. Department of Housing and Urban Development Daria Zvetina, Zvetina & Associates, Inc. Christine Mirasy-Glasco, Los Angeles Homeless Services Authority Charlene Flaherty, Arizona Dept. of Econ. Security

2. Using HMIS for Program Evaluation September 13-14, 2005 St. Louis, Missouri Sponsored by the U.S. Department of Housing and Urban Development Daria Zvetina Zvetina & Associates, Inc.

3. September 13-14, 2005 St. Louis, Missouri Sponsored by the U.S. Department of Housing and Urban Development 3 What Is Program Evaluation? Program evaluation, also called performance measurement, is a systematic effort to assess whether your program is making an impact in the areas that you intend.

4. September 13-14, 2005 St. Louis, Missouri Sponsored by the U.S. Department of Housing and Urban Development 4 Why Is It Important? Allows you to determine whether you are achieving your intended results Allows you to modify your approach or activities if you are not achieving your intended results Allows you to use outcome data to communicate the effectiveness of your services to funders and your community

5. September 13-14, 2005 St. Louis, Missouri Sponsored by the U.S. Department of Housing and Urban Development 5 And, program evaluation is a necessity in the increasingly performance/outcomes driven funding environment. Why Is It Important?

6. September 13-14, 2005 St. Louis, Missouri Sponsored by the U.S. Department of Housing and Urban Development 6 You can measure program performance at several levels Process evaluation – are we on track to achieve our outcome targets? Outcome evaluation – did we achieve our outcomes to the extent and within the time frame anticipated?

7. September 13-14, 2005 St. Louis, Missouri Sponsored by the U.S. Department of Housing and Urban Development 7 How Do We Get There? Dare I say, “logic model”?

8. September 13-14, 2005 St. Louis, Missouri Sponsored by the U.S. Department of Housing and Urban Development 8 Outcome Logic Models Logic models are a simple, straightforward way to summarize your resources, efforts, and end. Inputs  Activities  Outputs  OUTCOMES

9. September 13-14, 2005 St. Louis, Missouri Sponsored by the U.S. Department of Housing and Urban Development 9 Why Use a Logic Model? Logic models provide a framework to help you: Organize your thoughts and plans Be intentional about your efforts and allocation of resources Ensure a common understanding exists among all partners, staff, and consumers about what you’re attempting to achieve and how you intend to do it Clearly communicate to others (e.g., funders, the public and media, other stakeholders) the purpose and value of your efforts Regularly evaluate whether your resources, activities, and program approach are aligned and sufficient to achieve your intended results

10. September 13-14, 2005 St. Louis, Missouri Sponsored by the U.S. Department of Housing and Urban Development 10 InputsActivitiesOutputs OUTCOMES Inputs include resources dedicated to or consumed by the program, such as money, staff and staff time, volunteers and volunteer time, facilities, equipment and supplies. Activities are what the program does with the inputs to fulfill its mission, such as sheltering and feeding homeless families or training and counseling adults to help them prepare for jobs. Outputs are the direct products of program activities. They usually are measured in terms of the volume of work accomplished, such as the numbers of participants served and the units of service. Outcomes are benefits or changes for individuals or populations during or after participating in program activities. Outcomes may relate to knowledge, attitudes, values, skills, behaviors, conditions, or other attributes.

11. September 13-14, 2005 St. Louis, Missouri Sponsored by the U.S. Department of Housing and Urban Development 11 Outcomes An outcome should be: Focused on what the individual will gain from the program. Measurable with clear targets, specified timeframes, and methods for measuring change. Specific to the program and directly attributable to it. Attainable. Understandable to someone outside of the program.

12. September 13-14, 2005 St. Louis, Missouri Sponsored by the U.S. Department of Housing and Urban Development 12 Sample Outcome for HUD Permanent Housing Outcome Domain HUD Program Goal: Obtain and remain in permanent housing Sample outcome for an SHP rolling stock transitional housing program: 100% (30) of program families will be placed in subsidized housing at program entrance; at least 90% (27) will remain in this housing for 6 months; 80% (24) will remain in this housing at program exit or move to comparable or better housing; and 75% (23) will still be stably housed 6 months later. Measurement strategy: HMIS data gathered at program entrance, exit, and 6-month follow-up

13. September 13-14, 2005 St. Louis, Missouri Sponsored by the U.S. Department of Housing and Urban Development 13 Sample Outcomes for HUD Skills/Income Outcome Domain HUD Program Goal: Increase skills and income 90% (24) of program participants will increase their income from program entrance to exit via access to additional benefits and/or employment; 70% (21) will have achieved another increase in income within six months of program exit. 70% (21) of participants will have obtained livable wage employment at program exit; 70% (15) of those employed will be employed at equal or greater pay six months after program exit. Measurement strategy: HMIS data gathered at program entrance, exit, and 6-month follow-up

14. September 13-14, 2005 St. Louis, Missouri Sponsored by the U.S. Department of Housing and Urban Development 14 Potential Benefits of Using HMIS to Collect and Analyze Program Outcome Data Information is systematically gathered Easy to extract (theoretically) Is potentially available in real time Can be used to complete reports for funders (e.g., HUD APR) and other constituents Permits richer, more nuanced analysis (e.g., allows you to examine whether your program is working better for some participants than others) Permits examination of data trends and cross year or multi-year analysis

15. September 13-14, 2005 St. Louis, Missouri Sponsored by the U.S. Department of Housing and Urban Development 15 Challenges to Using HMIS for Program Evaluation Programs must enter information routinely, accurately, and reliably (e.g., data must be coded consistently within and across programs) Many providers will have to conquer their technology anxiety (akin to math anxiety?) To ensure use, providers must have access to IT help desk To garner ANY of the benefits listed previously, participating agencies/programs MUST have DIRECT ACCESS to their data

16. September 13-14, 2005 St. Louis, Missouri Sponsored by the U.S. Department of Housing and Urban Development 16 HMIS Data Elements Universal Data Elements: data elements that must be collected by ALL agencies serving persons who are homeless Program Specific Data Elements: elements that must be collected by all HUD funded agencies required to complete the APR

17. September 13-14, 2005 St. Louis, Missouri Sponsored by the U.S. Department of Housing and Urban Development 17 HMIS Universal Data Elements 2.1 Name 2.2 SSN 2.3 DOB 2.4 Ethnicity & Race 2.5 Gender 2.6 Veteran Status 2.7 Disabling Condition 2.8 Residence Prior to Program entrance 2.9 Zip Code Last Permanent Address 2.10 Program Entry Date 2.11 Program Exit Date 2.12 Unique Person ID # 2.13 Program ID # 2.14 Household ID #

18. September 13-14, 2005 St. Louis, Missouri Sponsored by the U.S. Department of Housing and Urban Development 18 Program Specific Data Elements 3.1 Income and Sources 3.2 Non-Cash Benefits 3.3 Physical Disability 3.4 Developmental Disability 3.5 HIV/AIDS 3.6 Mental Health 3.7 Substance Abuse 3.8 Domestic Violence 3.9 Services Received 3.10 Destination 3.11 Reasons for Leaving 3.12 Employment 3.13 Education 3.14 General Health Status 3.15 Pregnancy Status 3.16 Veteran’s Information 3.17 Children’s Education

19. September 13-14, 2005 St. Louis, Missouri Sponsored by the U.S. Department of Housing and Urban Development 19 Using HMIS Data Elements to Gather Housing Outcome Data Permanent Housing Outcome: 100% (30) of program families will be placed in subsidized housing at program entrance; at least 90% (27) will remain in this housing for 6 months; 80% (24) will remain in this housing at program exit or move to comparable or better housing; and 75% (23) will still be stably housed 6 months later. Relevant HMIS data elements: 2.10 Program Entry Date3.10 Destination 2.11 Program Exit Date3.11 Reasons for Leaving To assess housing status, 6-months post-exit an additional data collection timeframe would need to be added.

20. September 13-14, 2005 St. Louis, Missouri Sponsored by the U.S. Department of Housing and Urban Development 20 Using HMIS Data Elements to Gather Income Outcome Data Increased Income/Skills Outcome: 90% (24) of program participants will increase their income from program entrance to exit via access to additional benefits and/or employment; 70% (21) will have achieved another increase in income within six months of program exit. 70% (21) of participants will have obtained livable wage employment at program exit; 70% (15) of those employed will be employed at equal or greater pay six months after program exit. Relevant HMIS data elements: 2.10 Program Entry Date 3.12 Employment 2.11 Program Exit Date 3.2 Non-Cash Benefits 3.1 Income and Sources

21. September 13-14, 2005 St. Louis, Missouri Sponsored by the U.S. Department of Housing and Urban Development 21 Value Added HMIS Program Evaluation Capabilities Beyond determining whether housing and income/skill outcome targets have been achieved, the universal and program-specific data elements afford programs the potential to ask whether housing or income/skill outcomes vary systematically according to: Participant characteristics or circumstances such as: Disability status and type Health status Education Employment status at intake or exit As a result of specific program features such as: Length of time in the program Services received

22. Using HMIS for Program Evaluation September 13-14, 2005 St. Louis, Missouri Sponsored by the U.S. Department of Housing and Urban Development Christine Mirasy-Glasco, Los Angeles Homeless Services Authority

23. September 13-14, 2005 St. Louis, Missouri Sponsored by the U.S. Department of Housing and Urban Development 23 Background Los Angeles Homeless Services Authority’s (LAHSA) primary role is to coordinate and manage funding for homeless programs in the Los Angeles Continuum of Care. Challenge: Fund programs that demonstrate success in ending homelessness. Tool: HMIS Pilot Program Experience

24. September 13-14, 2005 St. Louis, Missouri Sponsored by the U.S. Department of Housing and Urban Development 24 Summary Mindset shift – from activities to outcomes (results) Necessary prerequisites Critical data elements Measures of success – verification and evaluation Benefits of new approach Lessons learned

25. September 13-14, 2005 St. Louis, Missouri Sponsored by the U.S. Department of Housing and Urban Development 25 Mindset Shift Traditional approach – provision of services measured by frequency or intensity of program activities. Data collected to report to other entities i.e. grantor or funder Outcome approach – provision of services measured by verifiable results. Meaningful data is the cornerstone of implementing a successful program. Hence, data collected should be primarily used to inform the program implementers as well as grantors. Bottom line: Both funder and program implementer need and want to know how to ask and answer the result question.

26. September 13-14, 2005 St. Louis, Missouri Sponsored by the U.S. Department of Housing and Urban Development 26 Necessary Prerequisites Understanding the funder’s intent, i.e. ending homelessness, increasing affordable housing stock etc. Identifying the goals of the program, i.e. obtain and maintain permanent housing; increase skills and/or income; and achieve greater self-determination. Establishing performance measures that express specific changes in participant condition or behavior as it relates to identified goals.

27. September 13-14, 2005 St. Louis, Missouri Sponsored by the U.S. Department of Housing and Urban Development 27 Critical Data Elements What question(s) are you answering? What data elements do you need to answer the question(s)? Example Question: How successful is my program in assisting participants in obtaining and maintaining permanent housing?

28. September 13-14, 2005 St. Louis, Missouri Sponsored by the U.S. Department of Housing and Urban Development 28 Critical Data Elements Performance measure: 45% (54) of participants will be placed in PH when they exit the program. 30% (16) of those participants placed in PH will remain in housing for 4 months after they exit the program. Data elements required: # of participants placed in PH Types of PH i.e. market rate apartment, shared housing, Section 8 housing. Program exit date. Follow-up mechanism with dates at 1 month, 2 months, 3 months and 4 months.

29. September 13-14, 2005 St. Louis, Missouri Sponsored by the U.S. Department of Housing and Urban Development 29 Measures of Success Verification – establishing the achievement of a result by documentation. This is obtained by generating reports from HMIS that query specific data elements established in advance. It provides a quick yes or no response to the question “did we achieve our targets?”

30. September 13-14, 2005 St. Louis, Missouri Sponsored by the U.S. Department of Housing and Urban Development 30 Measures Of Success Evaluation – at the end of the day, both funder and program implementer want to know whether the program is effective. Therefore, the question being asked here is “what have we learned?” The information obtained through this exercise should be used to change or modify program aspects that are not producing results. he information is also useful for funders in making funding and policy decisions.

31. September 13-14, 2005 St. Louis, Missouri Sponsored by the U.S. Department of Housing and Urban Development 31 Benefits of Outcome Approach Ability to detect the need for correction in real time throughout program implementation. Immediate feedback on program success to key stakeholders. Early evidence of program effectiveness that can be used to help build a case for continued funding.

32. September 13-14, 2005 St. Louis, Missouri Sponsored by the U.S. Department of Housing and Urban Development 32 Lessons Learned What worked Dispelling the misconception that the more clients served the better. Emphasis should be on what the clients achieved and whether it is consistent with the program’s goals. HMIS is a potent tool for program implementers to manage programs in a proactive manner. It is not a tool for funders to penalize program implementers who are not meeting their targets. Realization that program implementers and funders are real partners.

33. September 13-14, 2005 St. Louis, Missouri Sponsored by the U.S. Department of Housing and Urban Development 33 Lessons Learned What did not work Use of HMIS as a daily activity. As a result, very limited information could be retrieved to measure program progress. Overall agency buy-in. The benefits of HMIS were not well communicated to executive staff, line staff and clients. As a result, making changes to the program using information from HMIS was difficult.

34. September 13-14, 2005 St. Louis, Missouri Sponsored by the U.S. Department of Housing and Urban Development 34 Next Implementation Phase Deeply involve agency staff from all levels in the implementation process from the beginning. Invest time in explaining the benefits of performance targets and milestones/benchmarks to program implementers. Agree on protocol of checking utilization of the system by users.

35. Investing in Outcomes: The Arizona Homeless Program Evaluation Project September 13-14, 2005 St. Louis, Missouri Sponsored by the U.S. Department of Housing and Urban Development Charlene Moran Flaherty Arizona Department of Economic Security

36. September 13-14, 2005 St. Louis, Missouri Sponsored by the U.S. Department of Housing and Urban Development 36 Objectives Background Partners Tools/Methodology Timeline Next Steps

37. September 13-14, 2005 St. Louis, Missouri Sponsored by the U.S. Department of Housing and Urban Development 37 What Is It? Tools and methodology for… Evaluating the effectiveness of homeless programs Improving outcomes in order to make a real difference in the lives of homeless people Making informed decisions regarding investments Identifying best practices

38. September 13-14, 2005 St. Louis, Missouri Sponsored by the U.S. Department of Housing and Urban Development 38 Why Is It Important? Conscious shift from focus on outputs to investing in outcomes HUD McKinney-Vento grant State Performance Measures Private Investors

39. September 13-14, 2005 St. Louis, Missouri Sponsored by the U.S. Department of Housing and Urban Development 39 Goal Reduce the gap between knowledge and practice Which programs are effective? What interventions produce results? Where do we target investment?

40. September 13-14, 2005 St. Louis, Missouri Sponsored by the U.S. Department of Housing and Urban Development 40 Partners Arizona Dept. of Economic Security Arizona Dept. of Housing Continua of Care Virginia G. Piper Charitable Trust HUD

41. September 13-14, 2005 St. Louis, Missouri Sponsored by the U.S. Department of Housing and Urban Development 41 Provider Engagement Data to self-evaluate programs System for obtaining feedback Celebrate best practices Potential for technical assistance

42. September 13-14, 2005 St. Louis, Missouri Sponsored by the U.S. Department of Housing and Urban Development 42 Developing the System Statewide Steering Committee Provider Meetings Surveys of Providers Testing of Pilot Outcomes Training

43. September 13-14, 2005 St. Louis, Missouri Sponsored by the U.S. Department of Housing and Urban Development 43 Project Components Self-Sufficiency Matrix Annual Progress Report data Housing placement success Provider-developed outcomes

44. September 13-14, 2005 St. Louis, Missouri Sponsored by the U.S. Department of Housing and Urban Development 44 Self-Sufficiency Matrix DomainScore12345 Income no incomeInadequate income and/or spontaneous or inappropriate spending Can meet basic needs with subsidy; appropriate spending Can meet basic needs and manage debt without assistance Income is sufficient, well managed; has discretionary income and benefits Employment No jobTemporary, part-time or seasonal; inadequate pay, no benefits. Employed full time; inadequate pay; few or not benefits Employee full time with adequate subsidized housing. Maintains permanent employment with adequate income and benefits. Housing Homeless or threatened with eviction In transitional, temporary or substandard housing; and/or current rent/mortgage payment is unaffordable (over 30%of income) In stable housing that is safe but only marginally adequate. Household is in safe, adequate, subsidized housing. Household is safe, adequate, unsubsidized housing. Food No food or means to prepare it. Relies to a significant degree on other sources of free or low-cost food. Household is on food stamps Can meet basic food needs, but requires occasional assistance. Can meet basic food needs without assistance. Can choose to purchase any food household desires. Childcare Needs childcare, but none is available/accessible and/or child is not eligible. Child case is unreliable or unaffordable, inadequate supervision is a problem for child care that is available Affordable subsidized childcare is available, but limited. Reliable, affordable childcare is available, no need for subsidies Able to select quality childcare of choice SafetyHome or residence is not safe; immediate level of lethality is extremely high; possible CPS involvement Safety is threatened/temporary protection is available; level of lethality is high Parenting Skills There are safety concerns regarding parenting skills

45. September 13-14, 2005 St. Louis, Missouri Sponsored by the U.S. Department of Housing and Urban Development 45 Self-Sufficiency Matrix, cont. Children's Education One or more eligible children not enrolled in school. One or more eligible children enrolled in school, but not attending classes. Enrolled in school, but one or more children only occasionally attending classes. Enrolled in school and attending classes most of the time. All eligible children enrolled and attending on a regular basis Adult Education Literacy problems and/or no high school diploma/GED are serious barriers to employment. Enrolled In literacy and/or GED program and/or has sufficient command of English to where language is not a barrier to employment. Has high school diploma/GED Needs additional education/training to improve employment situation and/or resolve literacy problems to where they are able to function effectively in society. Has completed educational/training needed to become employable. No literacy problems Legal Current outstanding tickets or warrants. Current charges/trial pending, noncompliance with probation/parole. Fully compliant wit probation/parole terms. Has successfully completed probation/parole within past 12 months, no new charges filed. No active criminal justice involvement in more than 12 months and/or no felony criminal history Health Care No medical coverage with immediate need. No medical coverage and great difficulty accessing medical care when needed. Some household members may be in poor health. Some members (eg Children on AHCCCS All members can get medical care when needed, but may strain budget. All members are covered by affordable, adequate health insurance. Life Skills Unable to meet basic needs such as hygiene, food, activities of daily living. Can meet a few but not all needs of daily living without assistance. Can meet most but not all daily living needs without assistance. Able to meet all basic needs of daily living without assistance Able to provide beyond basic needs of daily living for self and family. Mental Health Danger to self or others; recurring suicidal ideation; experiencing severe difficulty in day- to-day life due to psychological problems. Recurrent mental health symptoms that may affect behavior, but not a danger o self/others; persistent problems with functioning due to mental health symptoms. Mild symptoms may be present but are transient; only moderate difficulty in functioning due to mental health problems. Minimal symptoms that are expectable response to life stressors; only slight impairment in functioning Symptoms are absent or rare; good or superior functioning in wide range of activities; no more than every day problems or concerns.

46. September 13-14, 2005 St. Louis, Missouri Sponsored by the U.S. Department of Housing and Urban Development 46 Pilot Data Collection 21 programs committed to participate 13 programs provided pre- data 25% emergency 50% transitional 25% permanent supportive

47. September 13-14, 2005 St. Louis, Missouri Sponsored by the U.S. Department of Housing and Urban Development 47 Pilot Data Collection, cont. 9 programs provided both pre- and post- data Attrition pre- and post- 12% emergency 42% transitional 99% permanent supportive

48. September 13-14, 2005 St. Louis, Missouri Sponsored by the U.S. Department of Housing and Urban Development 48 Scoring Systems System #1 Independence Dysfunction Child Functioning System #2 Self-Sufficiency Community Connectedness

49. September 13-14, 2005 St. Louis, Missouri Sponsored by the U.S. Department of Housing and Urban Development 49 Level of Functioning Permanent Supportive Housing Transitional Shelter Emergency Shelter

50. September 13-14, 2005 St. Louis, Missouri Sponsored by the U.S. Department of Housing and Urban Development 50 Results More challenging clients show greatest increase in self-sufficiency Lower expectations; greatest gains Less challenging clients show smallest increase in self-sufficiency Higher expectations; smaller gains

51. September 13-14, 2005 St. Louis, Missouri Sponsored by the U.S. Department of Housing and Urban Development 51 Timeline Outcomes Training – March 2005 Integrate Matrix w/ HMIS – June 2005 Provider Training on Matrix – July 2005 Statewide Rollout – August/September 2005 DES projects HUD projects

52. September 13-14, 2005 St. Louis, Missouri Sponsored by the U.S. Department of Housing and Urban Development 52 Vertical Integration Service Integration Homelessness State Plan Implementation

53. September 13-14, 2005 St. Louis, Missouri Sponsored by the U.S. Department of Housing and Urban Development 53 Next Steps Data collection Analysis/Mathematical modeling Develop data warehouse Identify and replicate best practices State reports Provider reports