1. The VR-ROI Project: Vocational Rehabilitation Return on Investment for Four State VR Agencies Dr. Kirsten Rowe, Virginia Department for Aging and Rehabilitative Services Dr. Robert M. Schmidt, University of Richmond Dr. Steven Stern, University of Virginia 7th Annual Summit on Performance Management in Vocational Rehabilitation Louisville, KY: September 8, 2014

2. Introduction Roadmap of Presentation – History of Project – Goal & Issues – Early work on ROI model development – Current activities and findings

3. Project Overview NIDRR-funded 3-year grant to U. of Richmond VR agency partners include: – Virginia General (DARS) – Virginia Blind (DBVI) – Maryland Combined (DORS) – Oklahoma Combined (DRS) Project consultants from U. of Virginia

4. Innovations Longitudinal administrative data on: – VR services provided – Employment and earnings (pre-, during, post-VR) “Crack the black box” of VR services Use applicant cohorts, not closure cohorts Include everyone who applies for VR Control for selection bias using “instruments” Focus on “Rate of Return” vs. ROI

5. Longitudinal Data on Employment, VR Services, Labor Markets, and SSDI/SSI  Earnings & Employment data from state Unemployment Insurance program records – 3 years prior through 5-10 years post VR application date using quarterly data for all VR applicants in SFY 2000/2007  VR Service Provision – Longitudinal VR services (up to 10 years) to account for multiple cases over time, purchased services and in-house/fixed costs  DI/SSI data from Social Security Administration – 3 years prior through 5-10 years post; monthly receipt & dollar amounts  Local labor market data from U.S. Bureau of Economic Analysis

6. Categories of Purchased Services First model: DTERMO – Diagnosis & evaluation: purchased services for assessing eligibility and developing IPE, plus medical diagnostics – Training: expenditures for OJT, GED program tuition and fees, etc. – Education: post-secondary education and vocational training – Restorative: medical/healthcare services plus AT – Maintenance: transportation, clothing, vehicle/home modifications, etc. – Other

7. Account for Variation in VR Consumers and Types of Services Provided  VR consumers by type of impairment – We estimate separate impacts by types of impairment (mental illness, cognitive impairments, physical impairments, learning disabilities)  VR services – We allow for different impacts of the VR service categories (DTERMO)  We can calculate Rate of Return by disability type or VR service category, as well as agency- wide

8. Model & Estimation Service receipt equation: binary service receipt as a function of demographic characteristics and DRS office and counselor propensities (multivariate probit) Employment equation: binary employment as a function of demographic characteristics, local labor market conditions, and service receipt (probit)

9. Model & Estimation Quarterly Earnings: log quarterly earnings as a function of demographic characteristics, local labor market conditions, and service receipt SSI/DI receipt: binary receipt as a function of demographic characteristics and service receipt Division of time into 3 segments: before application; first two years after application (short-run); and more than two years after application (long-run)

10. Model & Estimation Estimation by maximum simulated likelihood estimation: all parameters estimated jointly with allowance for rich covariance structure across equations

11. Measuring “Rate of Return” versus “Return on Investment”  ROR & ROI both use net earnings impacts and cost of service provision to calculate a measure of VR service efficacy  ROI requires the arbitrary selection of an interest rate, the choice of which becomes more important the longer the earnings time horizon  ROR for VR can be readily compared to rates of return such as the 10% annual ROR for long-term U.S. stock market performance

12. 2000 VA Results: ID

13. 2000 VA Results: MI

14. 2000 VA Results: PI

15. 2000 VA Work in Progress Working on LTESS – Important to control for variation in vendor services Working on LD – Race-based differentials in LD diagnosis and how it changes over time causes surprising results Working on Autism – Data is constructed; estimation will begin soon

16. Service Aggregation for 2007 DTERMPS – Diagnosis & evaluation – Training – Education – Restorative services – Maintenance – Placement – Supported Employment

17. Maryland 2007 Data Description (1 of 4)

18. Maryland 2007 Data Description (2 of 4)

19. Maryland 2007 Data Description (3 of 4) MI Transition Probabilities Initial Service EmploymentNot Employment Subsequent Service Initial Service0.8620.0500.0870.000 Employment0.0020.8420.1440.012 Not Employment0.0020.0390.9510.009 Subsequent Service0.0000.0570.0960.847

20. Maryland 2007 Data Description (4 of 4)

21. Maryland 2007 Special Issues and Progress Federal employees Cross-border employment Very high correlation of physical impairments and mental illness MI estimation is proceeding, and we will have results soon

22. Oklahoma 2007 Data Issues and Progress Enough Native Americans in sample to estimate effects of VR on that group Data construction is mostly done, and estimation is ready to start

23. VA DBVI 2007 Modeling and Data Issues The model that applies to MI, CI, and PI does not work as well for people who are blind because the nature of service provision, the aggregation of services, and (possibly) the measurement of outcomes are different. We are working with blind agencies to make progress on this issue. There are some issues associated with observation of services in this data. But, we are relatively confident the service data exist, so we should be able to solve this problem.

24. 2007 Analysis Timeline Maryland: estimation for MI group running now; other groups will follow; completion over next 6 months Oklahoma: working through last data issues; estimation will start soon; completion over next 6 months VA DBVI: significant modeling and data issues to resolve; completion over next year

25. 2007 Data and Beyond: Big Picture Potential expansion to other states Potential improvements in efficiency in collecting administrative data Other useful data – SSA – Federal employment data – Other states’ UI data