1. 2015 STC Summit Ongoing STC Research and Evaluation Projects Scott Gibbons & Christina Yancey Chief Evaluation Office, USDOL

2. 2015 STC Summit Presentation Overview 1.Summary of a recently completed study 2.Summary of 2 active studies 3.What are the approaches being tested? – How are we measuring their effectiveness? – How can the findings from these studies help you? 4.Some Discussion Topics and Questions 2

3. 2015 STC Summit Recently Completed Employer Survey Slides will be available in conference materials Key Findings (KS, MN, RI, WA) – High levels of satisfaction, some barriers to participation Topics and Areas Investigated – Characteristics of Employer Participation in STC Example: Industry, Size, Tenure, – Employer Opinions & Perceptions of STC – Employer Awareness of STC – Policy Recommendations to Increase Usage 3

4. 2015 STC Summit Ongoing STC Research and Evaluation STC Demonstration Project on Employer Uptake (to be completed in 2017) – Random Control Trial (RCT) and Quasi-Experimental Design (QED), Iowa and Oregon – Testing approaches to increase employer uptake – 12 months of experiment complete in fall 2015, now follow up Supplemental STC Report (to be completed in early CY 2016) – Mandated by Middle Class Tax Act – Synthesis of recent work, best practices, challenges – Interviews with employers, State staff (CT, RI, MI, TX, OH, WA, OR, FL) 4

5. 2015 STC Summit Increasing Employer Uptake of STC Demonstration project in Iowa and Oregon – Can we increase employer uptake of STC? – Can we determine what the impact of different kinds of treatments is on uptake of STC? An RCT design to test controllable approaches A QED design for less-controllable approaches The QED design reflects the kinds of things many states did or anticipate doing with their STC grants 5

6. 2015 STC Summit 6

7. What Approaches Increase Employer Uptake? Increasing information about STC – Direct mailings to employers: Special mailing, notice of claim, tax notices, others – Advertising, “Banners” and links on web sites – Links up on Job listing portal, tax and wage filing, employer area of UI web site Direct outreach to employers – Focus group presentations – Media – Discussions through industry or affinity groups – Educating politicians and representatives – Materials provided during 1-on1 or group meetings 7

8. 2015 STC Summit How Do We Measure What Works? Extract UI claims data (STC, non-STC) and employer agreement data for analysis – Analyze trends and patterns in STC usage – Extent of usage, user characteristics, industry, time period – Analyze changes in numbers of employer agreements Track responses to outreach – Track web site traffic linked from different forms of outreach, hits indicate responses to the outreach – Track outreach from mailings, personal contacts Perform surveys of employers, state staff – Satisfaction, awareness, perceptions of benefit 8

9. 2015 STC Summit How Do We Measure What Works? Many of these approaches can be used now, in your states, to assess important characteristics of STC usage: What industries are most common among STC users? Where, geographically, are the establishments located? Are increases in agreements or decreases in utilization linked to economic factors or conditions such as declining industry employment or output? What are the characteristics of the individuals filing STC claims? What are the trends in patterns in counts of claimants, claims and of employer agreements, now and over time? Do STC claims and regular UI claims increase and decrease at the same time or different times, what factors influence this? Monitoring for people who transition from STC to regular UI Monitoring to see if regular UI claims, benefit payments decrease when employers begin STC 9

10. 2015 STC Summit Planning and Logistics: Ensuring Studies have Resolving Sufficient Power 10 Each data point on the graph represents the number of STC plans initiated in the indicated quarter and the prior three quarters for employers in the proposed treatment and control regions in Oregon.

11. 2015 STC Summit 11 StateStudy Number of eligible employers Number of new STC employers Proportion of STC use Control and comparisonTreatment Control and Comparison (predicted) Treatment (control + MDE) Control and comparison (predicted) Treatment (control + MDE) IowaRCT14,329 8220.00060.0016 Oregon RCT (Portland) 12,33112,33026470.00180.0033 QED14,37315,05920410.00140.0027 Predicted control and comparison group STC usage and minimum detectable effect estimates for treatment group STC usage, assuming STC usage during 12-month study period will be one-half of current usage StateStudy Number of employers Number of new STC employersProportion of STC use ControlTreatment Control and comparison (predicted) Treatment (control + MDE) Control and comparison (predicted) Treatment (control + MDE) IowaRCT14,329 17350.00120.0024 OregonRCT (Portland)12,33112,33051800.00360.0056 QED14,37315,05940680.00280.0045 Predicted control and comparison group STC usage and minimum detectable effect estimates for treatment group STC usage, assuming usage during 12-month study period will be equal to current usage

12. 2015 STC Summit 12 Motivation Interventions Results GoalsStrategies Intervention ActivitiesOutcomes Avert lay-offs Minimize unemployment Maintain business operations Retain employer–employee attachments Awareness Desire Knowledge Ability Reinforcement RCT Emails Postal mailings Quarterly wage reports—banner Employer job postings—banner Notice of claims— letters Webinars (OR) QED (OR) RCT plus Employer group presentations One-on-one employer presentations Supplementary emails from business organization Short Term Phone inquiries Email inquiries Web hits QR web hits (OR) Long Term Plan applications Plan participation Use of STC relative to Unemployment Compensation External Factors a.Economic climate in state, county, industry, and firm; financial costs to state and employers b.State political and social climate (legislation; Federal grants)

13. 2015 STC Summit Considerations When Reviewing or Applying Research Findings Are the results generalizable? What factors would effect your success – Things in your control versus out of your control – Spatial or temporal variance in quality, intensity, frequency of treatment? – Changing conditions in economy, industry? – Similar dynamics between employers and State? – Finding a rationale that resonates 13

14. 2015 STC Summit Discussion Topics Applying evaluation principles to your work – Has your agency attempted to increase uptake and measure whether the efforts were successful? Leveraging DOL’s evaluations – What aspects of this work would you like to know more about (e.g., research design, approaches, data, measures)? Disseminating findings – How can we share the results of these studies in a way helpful to you? 14