1. Longitudinal Systems Analysis: Digging Deeper into System Performance by Populations and Geography
Julia Brown, Abt Associates
Erol Fetahagic and Ana Rausch, Houston CoC
Steven Rocha, Los Angeles CoC

2. Discussion Goals Understand the LSA Submission process
The role of your HMIS
The role of the HDX 2.0
Understand how you can use your LSA data locally
How can you get your LSA dataset for analysis?
What can you do with your LSA dataset?
The Real World Houston: Looking at Race and Ethnicity Disparities in Demographics
The Real World Los Angeles: Looking at Geographic Differences in Performance
Understand the tools HUD is working on to help

3. Submitting Your LSA HDX 2.0 Display HDX 2.0 Upload CoC-Level HMIS Data
Demographics by household type for people experiencing sheltered homelessness, using RRH, and using PSH
System Use by household types and population groups
HDX 2.0 Upload
CoC-Level HMIS Data
Demographics
Length of Time Homeless and in the System
Housing Outcomes
Returns to Homelessness
In April 2018, HUD issued LSA Report Specifications and Tools to support HMIS software providers in programming the HMIS LSA report file. These materials have been updated several times to correct bugs identified by vendors during the programming process. We really appreciate the software providers’ ongoing feedback as they prepare this complex report.
The LSA report generated from the HMIS is a .zip file comprising 10 CSV files with potentially thousands of rows of numerical output. Instead of reviewing these files directly, CoCs will need to upload the file to the HDX 2.0 and review the results within the HDX 2.0 Summary Data Display. There, results are calculated by the HDX 2.0 and displayed by reporting category and system use. In short, the HDX 2.0 will be the way CoCs are expected to “see” their LSA results.
This process is a shift from the current norm where the report from the HMIS is the end product. Now, we will want people to think of downloading a report from the HMIS and then uploading to HDX 2.0 as a single, seamless action necessary to review their LSA.
Vendors have not been asked to create an LSA report within the HMIS for CoCs to review the data prior to submission in HDX 2.0. This is because doing so would replicate the tens of thousands of calculations contained in the HDX 2.0 and introduce the potential for discrepancies between the official HDX 2.0 display and the vendor calculations.
If you find issues in your upload, you will need to go back and actually correct client data in your HMIS and re-upload the corrected files. You can upload as many LSA files in the HDX 2.0 as you need to until you are satisfied that they accurately reflect the CoC.
Built-in Data Visualization Tools (Future)
Detailed Downloadable Analysis Tables
Guided System Modeling Tools (Future)

4. Warnings in the HDX 2.0 Notably high missing data rate (>17%).
The number of people per household is [greater than 5/less than 2, depending on household type]
Children ages 5 and under that are unaccompanied by anyone older.
All of the [adults/children] fall into one gender category.
More people/households served in the last 1 year than in the last 3 years.
More enrollments/exits in the last 1 year than in the last 3 years.
There are basically only a small handful of warnings in the system right now. For each, you are asked to either leave a note or correct the issue and re-upload the dataset.
If you get these, it’s not a “ding” or a bad grade, it’s merely intended to help us ALL navigate to the problem.
It may be that it’s a bit odd, but actually true. You will be able to leave a note to that effect.
It may be that there is a data quality in your HMIS. If you can correct it, great. If you can’t, you will be able to leave a note.
It may be a programming issue with your vendor.
It may be a programming issue in the HDX 2.0.

5. Data Quality Results in the HDX 2.0
Count of invalid SSNs (e.g. consecutive zeros, non-numeric, shared SSNs, multiple SSNs)
Invalid values as a percentage of the total counts:
% of all clients with missing/invalid DoB, Gender, Race, Ethnicity
% of all adults with missing/invalid Veteran Status
% of all enrollments with missing/invalid Relationship to HoH, Disabling Condition
% of HoH and other adult enrollments with missing/invalid data fields, Domestic Violence
% of all exits with missing/invalid Destinations
% of all household enrollments with no or more than 1 HoH
The “Data Quality View” displays the data quality portions of the LSA uploaded files. These values are calculated and hard-coded as part of the LSA Report export from your HMIS. They are displayed on screen in the HDX 2.0 to help the CoC and the data reviewer understand how representative the uploaded data are of the CoC as a whole. The LSA data quality content includes system-wide information at a much broader level than actual LSA reporting. For example, the data quality columns look at race, ethnicity, etc. for children who are not heads of household, even though demographics for those children are not included in the LSA report.
HUD does not expect data quality to be perfect, nor does the appearance of a record in the Data Quality tables necessarily even mean that the data are not accurate. Instead, a data quality report is always going to be an imperfect algorithm designed to guide us to likely problems in the HMIS source data. For example, the LSA Report Data Quality tables include a Date of Birth record for any client that is 105 years or older. In general, a large percentage of such records is likely to indicate a data quality issue worth checking into. However, it may be the case that your CoC actually does have a large number of extremely elderly clients that were served during the reporting period. This does not indicate a problem in your system, nor in the content report. It simply means that, in your case, you can provide a comment that you have verified that the data are accurate, and that information will be included in the data review.

6. Dedicated Beds (HMIS + non-HMIS) Counts of People in Pop Groups
Inventory Data in the HDX 2.0
Based on your PDDEs, the HDX 2.0 will calculate these summary data for each of the 9 reporting categories
Overall
Dedicated Beds (HMIS + non-HMIS)
Counts of People in Pop Groups
Measurement Type
People
HMIS Beds
Utiliz.
Non-HMIS
HMIS Coverage
Vets
Youth CH
Unduplicated People
n/a
Avg. Per Night
Adjusted Avg. Per Night
Oct. 31
Jan. 31
Apr. 30
Jul. 31
Avg. Nights/ Person
Based on your PDDEs, the HDX 2.0 will calculate these summary data for each of the 9 reporting categories: ES/SH/TH – AO, ES/SH/TH – AC, ES/SH/TH – CO, RRH – AO, RRH – AC, RRH – CO, PSH – AO, PSH – AC, PSH – CO.
You will also be able to get these same data points at the project level so you can pinpoint problems

7. Submission Timeline 10/31 11/30 Late December Late January
HDX 2.0 Opens for Official Submissions
Within 10 days, log in and attempt an upload
11/30
“Submit” deadline
Late December
Outreach from Liaisons
Work together to resolve issues
Late January
Liaison marks “complete”
Within 3 days, CoC marks “confirmed”
Late February
Data usability
Review timeline with special emphasis:
What does “submit” mean? Don’t forget that there will be several months of review with the team of data liaisons following submission. Consider the “submit” deadline to be the starting point for in-depth data review.
What will review be like? Here, liaisons are going to be working with CoCs one-on-one to assess whether the dataset really reflects your individual CoC and will be running data checks. These checks will be iterative and collaborative and may reveal issues in various points in the process that we may need to tackle:
We may pinpoint the source of a problem within the calculations in HDX 2.0. That won’t require any action on the part of CoCs or vendors, we will fix those and let you all know that corrections have been made. (give example)
We may pinpoint the source of the problem with a specific vendor, which would suggest a programming issue. We would work with the vendor and make sure there is adequate time and support to the affected CoCs to be able to re-run and re-upload their results.
Or we may pinpoint the source of the problem as a data issue within the CoC. This may mean we work with you to support correcting and resubmitting your data, or it may impact the usability of your data.
Once you and your liaison get to a point where the data submitted are as clean as you can get it, it will be marked as “complete.” CoCs will then have 3 days for the CoC primary to mark the data as “confirmed” in the system. This should be a formality – you will already have been communicating extensively – but it will be a more serious, critical deadline than it has been in the past, because we need your ongoing engagement even after submission.

8. Submission Timeline
I will be around to answer questions about submission…
But let’s look into the future… beyond just submitting your data to HUD.... Let’s talk about how YOU can use the LSA in your CoC!

9. Introduce Erol and Ana.

10. LSA Data by Population Group
A Closer Look at the Houston CoC
Erol Fetahagic & Ana Rausch

11. Cities that Fit into Houston
TX-700 CoC = 3,739 sq. miles
From: Knudson,LP

12. TX-700 CoC – 3,739 sq. miles

13. Houston HMIS Overview
The Coalition for the Homeless has been the HMIS Lead Agency for our CoC (TX-700) since 2004
The TX-700 CoC includes Harris, Fort Bend, & Montgomery Counties in Texas (over 3700 sq miles)
Current system size:
78 organizations
241 projects
778 end users
24,888 annual homeless clients
251,815 total client records

14. Race & Ethnicity – U.S. Census

15. Race & Ethnicity – HMIS Data

16. Some Questions to Consider
Is Race/Ethnicity breakout different for general population vs. HMIS client data?
Do different HMIS project types serve different population groups?
Do all clients have equal access to permanent housing?
What are the contributing factors that cause over/underrepresentation of a certain racial/ethnic group in the homeless services system?

17. LSA Raw Data
10 CSV files exported from HMIS

18. LSA in HDX 2.0
Upload CSV files to HDX 2.0
Export Summary Data

19. Race & Ethnicity in LSA Summary Data

20. LSA Race & Ethnicity by Project Type

21. LSA Race & Ethnicity by Household Type

23. LSA in HDX 2.0
As Erol described, you can access your Summary data tables for download. You can also choose to download your “Full Analysis File.” This takes a few minutes to run, but it will export a CSV document of the full set of the roughly 60,000 calculations that the raw LSA upload can be used to generate (actually, there are probably more than that, but these are the ones HUD wanted to be able to create). Currently, these are only available for export.

24. LSA Full Analysis Tables
The full set of updated table shells that you would be able to populate from this file is available on the HUD Exchange:
These are detailed demographics, length of time homeless, length of system use, exit, and returns data for AO, AC, CO households….

25. LSA Full Analysis Tables
All Households
Households with Adults & Children
Adult-Only Households
age 18+
Child-Only Households
<age 18
Unaccompanied Youth (18-21, 22-24, and 18-24)
Unaccompanied Children
Parenting Youth (HoH age 18-24)
Parenting Children
Non-Veterans (all age 25+)
Veteran
Veteran
Chronically Homeless
Fleeing DV
Fleeing DV
Fleeing DV
…as well as household breakouts like “parenting young adults,” or “households in which every member is 55+,” etc – all that are shown here.
Key
All Demos; All Sys. Use Data
All Demos; Ltd. Sys Use
Ltd. Demos; Ltd. Sys Use
Disabled
Disabled
Disabled
Large Families (3+ children)
Older Adult (Age 55+)
1st Time/Returner
1st Time/Returner
1st Time/Returner
Race/Ethnic Groups
Race/Ethnic Groups
Race/Ethnic Groups
Quick-Returners
Quick-Returners
Quick-Returners
Long-stayers
Long-stayers
Long-stayers

26. Introduce Steven.

27. A Closer Look at the Los Angeles CoC Steven Rocha
LSA Data by Geography
A Closer Look at the Los Angeles CoC
Steven Rocha

28. Los Angeles County HMIS Overview
The Los Angeles Homeless Services Authority has been the lead agency for the Los Angeles Continuum of Care (CA-600) since 1993.
LA County is comprised of four different CoCs: Los Angeles, Glendale, Pasadena, and Long Beach. Glendale and Pasadena are part of the LA HMIS collaborative, while Long Beach uses a separate HMIS.
Current system size for the LA HMIS Collaborative:
138 organizations
1209 projects
3,000+ end users
59,000+ homeless clients (HUD FY 2017)
250,000+ total client records (since 10/1/2012)

29. Geography of Los Angeles
Los Angeles is the largest urban county in the nation at 4,083 square miles.
LA is split into several geographic areas, each requiring unique reporting:
4 Continuum of Cares
8 Service Planning Areas
82 Geo Codes
5 Supervisorial Districts
16 Senate Districts
19 Assembly Districts
18 Congressional Districts
15 Council Districts
2000+ Census Tracts
86 Cities
100+ Communities

30. LSA Reporting by Geography
Per HUD specs, the LSA exports from HMIS can be generated at the agency and program level.
As part of the HIC process, LA compiles lists of agencies and projects by region, which can then be used to identify sets of regional projects to run LSA exports.
What metrics would be beneficial to see by geographic region?

31. Length of Time Homeless and Other System Use
System path groups
Shows effectiveness (minimal length of stay) for different programs by type
Are there certain regions that have longer length of stay prior to RRH move in? Compare to other regions that have shorter length of stays, and see how programs in the longer length of stay region may need to be adjusted.
Population groups
Breaks down various demographics by length of time homeless for certain populations
Are there some program types within a region where certain races have longer lengths of stays than others? How would that compare to other regions, and how can that be addressed?

32. PSH and RRH Utilization Detail
PSH Turnover rates
Shows stayers and leavers in PSH to get turnover rates.
Which regions have the highest turnover? What does returns to homelessness look like for those regions?
RRH move-ins
Identifies how long it’s taking to place participants into housing (if at all), and how long assistance is provided.
Are there regions where the time in RRH prior to move-in is significantly longer than others? Does population type have an affect on this?

33. Successful Housing Placements and Retention
Exit destinations by system path groups
Helps to gauge which system path leads to the highest number of permanent placements.
Is there an outlying region that has a high number of PH system exits for a particular system path? What is that region doing right that can be applied to other regions?
Exit destinations by population groups
Breaks down various demographics by exit destination
Similar to length of time homeless, is there a region where certain races have lower PH exits than others? How would that compare to other regions, and how can that be addressed?

34. Returns After a Previous System Exit
Timeframe for returns based on prior destination
Can gauge how successful certain destinations appear
Does a region have a high number of PH placements when compared to others, but also has a higher rate of returns from those PH exits? This can show a case where other regions may not benefit from this region’s example.
Timeframe for returns based on system path groups
Shows how effective each system path may be by showing number of returns.
Is there a particular system path that results in high returns? How does that relate to the regions resources, and can that model be applied to other regions?

36. Data Visualization (Future)
We’re just getting started on the LSA process, so it feels a little overwhelming right now, but over the next year, we’ll be working to bring new tools and resources online to help you explore the opportunities of such a rich dataset. In early 2019, we will add a Performance Module to the HDX 2.0. This module will provide data visualizations of your system’s performance on the three main performance measures, as well as demographics and inventory. All the data visualizations will be broken down by all of the available household types, and comparison tools will be baked into the interface.
So, for the population group analysis that Erol and Ana described, you could turn to this tool and choose to look at the different race and ethnic breakdowns for all households, families versus individuals… for child-only households… for young adults on their own… for parenting young adults… for veteran households… and so on… The tool will respond dynamically so you can choose and view right on screen and you can look at demographics on their own, or look at the system performance differences between population groups within each of the system measures.

37. Data Visualization (Future)
We haven’t touched on it as much here, but in addition to population groups, the LSA organizes system use data into the pathways that clients use, or combinations of project types. The Performance Module’s system map will depict system utilization patterns or pathways for eleven common project type combinations. The map will also show performance by pathway for days homeless, exits and returns. This map and the other visualizations in the SPIST will be available for all the people served in your system and further broken down by all the household types we’ve already talked about.
This information will help users understand how different parts of their system and different groups of people are served by their crisis response and housing system and how these utilization patterns translate into overall system performance.
The module will also be programmed to flag performance outliers for the user, such as a notably longer length of time homeless for one household type versus another. These “insights” can be added to a Performance Improvement Action Plan that will be included in the module.

38. Back to the Present… For now: Bookmark the HUD Exchange HDX page
Focus on data quality, upload, reviewing your warnings, and clicking submit
Everything else is for the future…
Bookmark the HUD Exchange HDX page
Updated HDX 2.0 User Guide posted shortly
Beta guide available now
HUD will host two “Office Hours” style webinars to answer FAQs
2PM eastern
PM eastern
All materials and list serv messages related to the HDX 2.0 and LSA are housed on the HUD Exchange HDX page. Any updated materials related to the LSA report or the HDX 2.0 interface aimed at Sys Admins and HMIS Leads will be posted there, so review it carefully and perhaps even bookmark the page.