1. System Level Performance Measurement: Using HMIS to Drive Change
Darlene Mathews, The Community Partnership
Lianna Barbu, Community Shelter Board
September 23, 2008

2. Learning Objectives Learn the fundamentals of performance measurement.
Learn how to develop and use HMIS-based performance measures to support local system planning and policymaking on homelessness.
Identify strategies to use HMIS-based performance measures to improve program and system performance.
2008 HMIS Training: Setting the Standard - U.S. Department of Housing and Urban Development

3. Why HUD is Interested in Performance Measurement
Accountable to Congress, OMB,
the White House, and taxpayers
HUD’s overall objective: to “end chronic homelessness and to move homeless families and individuals to permanent housing”
At least 71.5% of formerly homeless individuals will remain housed in HUD permanent housing projects for at least 6 months
At least 63.5% of homeless persons will move from HUD transitional housing into permanent housing
At least 19% of persons exiting HUD homeless assistance projects will be employed
CoC’s accountable for
achieving & documenting results
Performance Measurement
2008 HMIS Training: Setting the Standard - U.S. Department of Housing and Urban Development

4. Performance Measurement Fundamentals
2008 HMIS Training: Setting the Standard - U.S. Department of Housing and Urban Development

5. What is Performance Measurement?
Performance measurement is a process that systematically evaluates whether your Continuum of Care’s efforts are making an impact on the clients you are serving.
2008 HMIS Training: Setting the Standard - U.S. Department of Housing and Urban Development

6. Multiple Levels of Performance Measurement
Local Service
Provider
1. Program Level
APR or Other Performance Measurement Tool
CoC
2. CoC/System Level
CoC Application or Other Performance Measurement Tool
3. State Level
State-Specific Performance Measurement Tool
4. National Level
GRPA and PART Reviews
2008 HMIS Training: Setting the Standard - U.S. Department of Housing and Urban Development

7. Purpose of System Measurement
Are actions achieving intended goals?
What is the purpose of the system? Goals?
Does the system work?
If yes, what makes it work?
If no, what part doesn’t work, and how do you fix it to make it work?
Note that you may have system goals that only relate to certain types of clients or parts of the system (e.g., different goals for severely disabled persons)
2008 HMIS Training: Setting the Standard - U.S. Department of Housing and Urban Development

8. Performance Measurement Building Blocks
Inputs include resources dedicated to, or consumed by, the system—e.g., money, programs, facilities.
Activities are what the system does with the inputs to fulfill its mission, such as providing shelter, feeding the homeless, or providing job training.
Outputs are the direct products of system activities. They usually are presented in terms of the volume of work accomplished—e.g., number of participants served, number of service engagements, number of shelter nights.
Outcomes are benefits or changes among clients during or after participating in system activities. Outcomes may relate to change in client knowledge, attitudes, values, skills, behaviors, conditions, or other attributes.
2008 HMIS Training: Setting the Standard - U.S. Department of Housing and Urban Development

9. Example: Single Adult Shelter System’s Impact Over Time
Outputs
Outcomes
Short-term
400 single adults will receive shelter and access to services.
350 clients will receive individualized housing information and referral assistance.
75% of clients will have a plan to obtain and maintain stable housing.
50% of clients who exit will obtain stable housing.
180 clients will receive weekly check-up calls and problem solving assistance, as needed, after exiting to stable housing.
85% of clients who obtain stable housing will not return to shelter within 12 months (43% of all exits).
Long-term
2008 HMIS Training: Setting the Standard - U.S. Department of Housing and Urban Development

10. Framework for Converting System Goals into Outcomes
Who is the base for measuring results?
Step 1
What do you hope to achieve with this population?
Step 2
Within the base, how many persons achieved it?
Step 3
Who is the base population for measuring results?
Within the base, how many persons achieved it?
Step 4
Outcome (%)
2008 HMIS Training: Setting the Standard - U.S. Department of Housing and Urban Development

11. Converting System Goals into Outcomes: Example
Program Goal: Exiting single adult shelter
clients into “stable” housing
Step 1
Step 2
Step 3
Persons who exited during the past year (N= 400)
Obtain stable housing
200 persons obtained stable housing
400
200
Step 4
50% obtained
stable housing
2008 HMIS Training: Setting the Standard - U.S. Department of Housing and Urban Development

12. Look Out for Ambiguous Concepts
Developing and measuring performance outcomes often invites ambiguous concepts into the process.
For example, what do we mean by…
…obtaining stable housing?
…obtaining employment?
…increasing income?
…accessing services?
…becoming more self-sufficient?
Which data elements and responses will count?
2008 HMIS Training: Setting the Standard - U.S. Department of Housing and Urban Development

13. Key Underlying Assumptions
Outcomes can be calculated consistently for all clients if, and only if:
Staff are collecting the required information at entry and exit for each client.
Staff are recording this information into the HMIS in a timely fashion (e.g., 4th business day of every month for prior month).
The target population is clearly defined.
The program goals are clearly defined for the target population.
2008 HMIS Training: Setting the Standard - U.S. Department of Housing and Urban Development

14. Developing Performance Measures: Key Elements
For each measure (output or outcome):
Purpose: identifies the reason for the measure
Application: identifies the programs and/or system for which a measure applies
Definition: definition for the measure
Goal-Setting Methodology: used by program or system to set goals around the measure
Reporting Methodology: detailed description of how measure is calculated
Use HMIS data elements
Develop and test before implementation
2008 HMIS Training: Setting the Standard - U.S. Department of Housing and Urban Development

15. Example: Successful Housing Outcome
a) Purpose: Indicates program’s success in ending homelessness by linking households to appropriate next step housing which includes transitional and permanent housing. A higher number and rate are considered positive.
b) Programs: All
c) Definition: the number of distinct households that exit (i.e., latest exit for households with multiple stays during report period) to successful housing as defined in Table X and the percentage this represents of total distinct households exited during the report period.
d) Goal-setting methodology: Meet or exceed CoC goal or prior performance. Multiply the percentage goal by the projected number of exited households (i.e., latest exit for households with multiple stays during the report period).
e) Reporting methodology: Calculate the total number of distinct household exits and the total number of destinations that are considered successful housing outcomes. Divide the number of successful housing outcomes by the number of total exits during the report period.
2008 HMIS Training: Setting the Standard - U.S. Department of Housing and Urban Development

16. Considerations When Developing Measurement Methodology
Specify all parts of query/calculation:
Programs
Across a program type (e.g., all outreach, all PSH programs)
All programs
Programs that serve a specific subpopulation
Clients (de-duplicated across programs)
All Exiters
All participants
All currently enrolled
Those who stayed > 6 months
Data Elements
Same HMIS elements as program measures
Need rules to reconcile data across programs if client was enrolled in more than one program
Timeframe
Define date range
Specify date field (entry date, exit date, placement date, assessment date, etc.)
You have to define all parts of your query
2008 HMIS Training: Setting the Standard - U.S. Department of Housing and Urban Development

17. Wherever Possible Use HMIS to Define Your Concepts
Based on the Destination HMIS data element, we can define stable housing (narrowly) using the following response categories:
Emergency shelter
Apartment or house that you own
Transitional housing
Staying/living with family
Permanent housing
Staying/living with friends
Substance abuse facility or detox center
Hotel or motel voucher paid for without ES voucher
Hospital (non-psychiatric)
Foster care home or group home
Jail, prison, or juvenile detention center
Place not meant for human habitation
Room, apartment, or house that you rent
Other
Don’t Know
Refused
2008 HMIS Training: Setting the Standard - U.S. Department of Housing and Urban Development

18. Performance Measurement Methods
2008 HMIS Training: Setting the Standard - U.S. Department of Housing and Urban Development

19. Performance Measurement Methods
Measure Interim Outcomes by Applying Program Measures System-wide
Use Program Performance Results to Change or Manage the System
Measure System Results with HMIS-Based Impact Outcomes
2008 HMIS Training: Setting the Standard - U.S. Department of Housing and Urban Development

20. 1st System Performance Measurement Method
Measure Interim Outcomes by Applying Program Measures System-wide
What: Examine how the system (across all programs or across part of a system) performs on a specific measure
Example: What percentage of all homeless persons have greater incomes when they leave the homeless system?
Example: What percentage of all homeless single men leave the shelter system for transitional or permanent housing?
Use: Mid-term (e.g., annually)
To assess and communicate the overall system’s success in helping clients achieve intended outcomes
To indicate interim success towards overall impact goal
To reinforce or adjust current expenditures and support new funding requests
2008 HMIS Training: Setting the Standard - U.S. Department of Housing and Urban Development

21. Establishing System Performance Measures
Adopt system goals
Past performance data
Account for state and federal goals
Account for local goals related to achievement of long-term impact
Establish program performance expectations based on system goals
Negotiate with agencies
Incorporate into CoC/funding agreements
Include quarterly, semi-annual, annual goals
Response to low performance
Performance reports
Set frequency
Determine audience/distribution
Use data that meets quality standards
System and program level performance
2008 HMIS Training: Setting the Standard - U.S. Department of Housing and Urban Development

22. Case Study: Columbus, Ohio Performance Measures
Prevention
Outreach
Emergency Shelter
Resource Specialists
Rapid
Re-Housing
Permanent Supportive Housing
Number Served 
Successful Housing Outcomes
Average Length of Stay
Average Length of Participation
Recidivism
Successful Income Outcomes
(Tier II shelter)
Direct Client Assistance Utilization
Occupancy
Housing Stability
Housing Retention
2008 HMIS Training: Setting the Standard - U.S. Department of Housing and Urban Development

23. Case Study: Columbus, OH Performance Reporting
Annual Evaluation
System & program measures compared to period goal
Used to help determination annual funding
Data mostly derived from HMIS
Programs scored as:
High: no less than one not achieved
Medium: half or more achieved
Low: less than half achieved
Quarterly System & Program Indicator Reports
System & program performance measures compared to period goal
Used to identify system(s) and program(s) “of concern”, need for intervention
Annual Community Report on Homelessness
Annual and trend data
Point-in-time count data
Reports issued to CSB Board of Trustees, CoC Steering Committee, funders,
2008 HMIS Training: Setting the Standard - U.S. Department of Housing and Urban Development

24. Case Study: Washington, DC
Performance Measures
Clients Served (all)
Clients Served (chronic)
Facility Occupancy
Rate
Clients Engaged in Case Mgmt
Exits
(Positive & Negative)
Clients w/ Income Upon Exit
Length of Stay
Self Sufficiency
Low Barrier & Severe Weather 
Temporary Shelters
Transitional Housing
Permanent Supportive Housing
Low Barrier and Severe Weather Shelters are not required to set measures for the baseline Performance Indicators listed above because they are not indications on the quality of services offered. However, like all other programs, they are responsible for ensuring that their data is accurate so that we may calculate Occupancy, Clients Served and Chronically Homeless numbers per site.
Self Sufficiency Temporary, Transitional and Permanent Supportive Housing Programs required to submit a Self Sufficiency Indicator will have to choose from Substance Abuse, Education, Mental Illness Assistance or Employment.
2008 HMIS Training: Setting the Standard - U.S. Department of Housing and Urban Development

25. Case Study: Washington, DC Performance Measures (con’t)
Outputs/Efficiency Measures
Clients Served
Chronically Homeless Served
Occupancy: the rate at which program was used
Interim Outcome Measures
Permanent Housing
Positive client destinations at exit (TH programs)
Retain clients for 6+ months (PSH programs)
Income: the amount of income or sources obtained
Self-Sufficiency: change in substance use, education, mental illness or employment
Measures apply differently to each program type and are supplemented with qualitative data for ranking purposes
2008 HMIS Training: Setting the Standard - U.S. Department of Housing and Urban Development

26. 2nd System Performance Measurement Method
Use Program Performance Results to Change or Manage the System
What: Compare performance results on selected program measures from one program to another. (Risk adjust for client differences.)
Our system is not as effective as we want in increasing client incomes… which programs are working best or least well?
Use: Mid-term (e.g., annual)
To help understand whether individual programs are working
To help decide how to allocate dollars or rank programs for funding
To identify best practices or agencies that need TA
To determine how performance expectations should differ for different populations, and how to refer clients most appropriately
Theory: Programs with the same goals can be compared to understand and improve performance of each program individually and the system as a whole
2008 HMIS Training: Setting the Standard - U.S. Department of Housing and Urban Development

27. Caveats of System Performance Measurement & Analysis
May require a lag in time even beyond the client’s completion of program involvement
Measuring recidivism requires a certain period of time (12 months?) after program exit in which return to the system might occur
If the HMIS only includes data from a subset of programs, the system analysis will provide an incomplete picture
Perceived decline in recidivism may only reflect absence from participating providers as opposed to not being homeless anymore
2008 HMIS Training: Setting the Standard - U.S. Department of Housing and Urban Development

28. Apples to Apples: Risk Adjustment
Problem: Comparing program results can encourage programs to “cream” to ensure strong results
Solution: Risk adjustment allows differences to be accounted for in client populations when comparing results across programs
Results can be adjusted on the basis of…
Client characteristics, such as demographics, family size, disability
Client history, such as past eviction, criminal background
Client functionality or level of engagement/commitment to change
To adjust for client differences, programs need to collect consistent data on clients to use during analysis of program results
Agree on these standards beforehand
Develop an analysis plan for how to adjust for client differences
Engage a researcher to help develop the plan
Even if results not formally adjusted, acknowledge that different programs may have different outcome expectations based on differences in clients targeted and/or served
2008 HMIS Training: Setting the Standard - U.S. Department of Housing and Urban Development

29. 3rd System Performance Measurement Method
Measure System Results with HMIS-based
Impact Outcomes
What: Define a process to track the impact of the system’s efforts on the need/problem.
Is the system reducing the time that people stay homeless, and therefore reducing the point-in-time prevalence?
Use: Long-term (e.g., three years)
To determine if you need to continue actions, expand current efforts (add resources to the current system), tweak current efforts, or fundamentally change the structure and approach of your homeless system
To evaluate whether past actions (expanding the system or adding money) helped address the issue
To communicate the overall system’s success (such as, when justifying funding requests)
Theory: In the long-term, important to demonstrate that the collective actions of the system solve the problem.
2008 HMIS Training: Setting the Standard - U.S. Department of Housing and Urban Development

30. Sample Impact Measures
Incidence of homelessness - Is homelessness declining?
Incidence of street or CH - Is street or chronic homelessness declining?
Length of stay in system, across all homeless programs - Do people stay homeless for shorter periods of time?
Rates of Recidivism – Are repeat occurrences of homelessness avoided or declining?
Cross-tabulate results by core characteristics to understand if/how results vary for different subpopulations
Provide a one-pager on data/steps to calculate each of these for a CoC.
2008 HMIS Training: Setting the Standard - U.S. Department of Housing and Urban Development

31. Some notes of caution…
There is more to performance measurement than conducting the analysis
Educate, train, obtain buy-in
Be careful about how data is interpreted and used
Jump in, but don’t be careless in how results are used
Look at the results within the context of all the outputs, interim measures and impact measures to validate the interpretation that’s being made
Vet the results before publicly releasing anything
Appropriately caveat the limitations of the data and analysis
2008 HMIS Training: Setting the Standard - U.S. Department of Housing and Urban Development

32. Look before you leap….
It appears that the rate of family homelessness is going down.
2008 HMIS Training: Setting the Standard - U.S. Department of Housing and Urban Development

33. …Lesson Learned
Reality: Programs couldn’t accept new families as beds were still occupied by existing clients who had dramatically longer LOS.
2008 HMIS Training: Setting the Standard - U.S. Department of Housing and Urban Development

34. Value of Performance Measurement
Communicates more accurate description of population (vs. individual, anecdotal needs)
Creates higher level of understanding of community problem (builds community support)
Informs resource allocation (annual funding)
Informs program development/improvement (CQI)
Informs policy options (more shelter vs. housing)
2008 HMIS Training: Setting the Standard - U.S. Department of Housing and Urban Development

35. Additional Resources
What Gets Measured Gets Done: A Toolkit on Performance Measurement for Ending Homelessness, National Alliance to End Homelessness,
The Community Partnership for the Prevention of Homelessness (Washington, DC)
Community Shelter Board (Columbus, OH)
2008 HMIS Training: Setting the Standard - U.S. Department of Housing and Urban Development

36. Questions & Discussion