WIOA Performance Accountability: A Focus on Youth Performance September 17, 2018 WIOA Performance Accountability: A Focus on Youth Performance October 25, 2017

Sara Hastings Unit Chief /Division of Youth Services DOL/ETA

Our Journey Together TA Series Two Day Kick-off Webinars Series Kick-Off: Where we’ve been, where we are now and where we’re going! Tuesday, October 24, 2017 at 11:30 AM ET WIOA Youth Eligibility Live Question and Answer Session Tuesday, October 24, 2017 at 2:30 PM ET Coming Together at the Table--The Power of Youth Committees to Convene, Coordinate and Collectively Impact Youth Lives Wednesday, October 25, 2017 at 11:30 AM ET   WIOA Youth Performance Accountability   Wednesday, October 25, 2017 at 2:30 PM ET

Evan Rosenberg Cesar Acevedo US Department of Labor, Employment and Training Administration, Division of Youth Services Cesar Acevedo US Department of Labor, Employment and Training Administration, Office of Policy Development and Research

To provide state and local youth practitioners with what they need to know about performance accountability Definitions of participant and exit Primary indicators of performance for WIOA core programs and other workforce programs Statistical Adjustment Model and Negotiations

Quiz Time Which of the following statements is true? A. If an in-school youth becomes a participant on June 15, 2017, they are included in the credential indicator for Program Year 2016. B. If an in-school becomes a participant on June 15, 2017, they are included in the measurable skill gains indicator for Program Year 2016. C. A career readiness certificate counts as a positive outcome in the credential indicator. D. A youth in postsecondary education in the 2nd quarter after exit, who also works a part-time job 6 hours a week during the second quarter after exit, is excluded from the median earnings indicator.

Guidance TEGL 10-16 Change 1, comprehensive, joint performance guidance published Dec 2016 TEGL 26-16, use of supplemental wage information guidance TEGL 3-17, WIOA Annual Performance Report Submission Data Validation guidance (Joint) to be published by end of 2017

Youth Participant Definition of a Participant: reportable individual who receives services other than self-service or information-only services and has satisfied all applicable programmatic requirements including eligibility determination For Youth this means: Eligibility determination Objective assessment Individual service strategy Received 1 of 14 youth program elements Participation date isn’t until the last of these items occurs

Exit Program exit occurs when the participant no longer receives services for 90 days and has no additional services scheduled The date of program exit is determined after 90 days of no services have elapsed and no future services are planned. At that point the date of program exit is applied retroactively to the last date of service. The following services do not delay an exit: Follow-up services Self-service Information-only services Common exit is encouraged but not required across DOL programs (no common exit with ED programs)

Types of Youth Follow-Up Services Post-Exit There are five WIOA youth program elements that may be provided as follow-up services If any of these 5 services are provided after exit, they do not trigger new eligibility determination/ new participation and may be coded as follow-up services Supportive services Mentoring Financial literacy services Services that provide labor market information Postsecondary preparation activities

Primary Indicators of Performance Six Primary Indicators of Performance: Employment Rate 2nd Quarter after Exit -(Education/Employment for youth) Employment Rate 4th Quarter after Exit Median Earnings in the 2nd Quarter after Exit Credential Attainment Rate Measurable Skill Gains Effectiveness in Serving Employers

Employment/Education/Training Rate 2nd Quarter after Exit Percentage of participants in education or training activities, or in unsubsidized employment during the second quarter after exit. Successful outcomes include unsubsidized employment, secondary education, postsecondary education, and occupational skills training in 2nd quarter after exit Adult indicator only recognizes unsubsidized employment as outcome Unlike WIA, WIOA success in this indicator is only based on status in the 2nd quarter after exit regardless of status upon program entry.  For example, if ISY is in high school at program entry and still in high school in the 2nd quarter after exit, they are a success in the indicator even though they were already in high school at program entry.

Employment/Education/Training Rate 4th Quarter after Exit Percentage of participants who are in unsubsidized employment during the fourth quarter after exit from the program. Successful outcomes include unsubsidized employment, secondary education, postsecondary education, and occupational skills training in 4th quarter after exit Adult indicator only recognizes unsubsidized employment as outcome Success based solely on status in 4th quarter regardless of status at participation or in 2nd quarter – not a retention measure

Median Earnings in the 2nd Quarter after Exit The median earnings of participants who are in unsubsidized employment during the second quarter after exit from the program. Only includes those in unsubsidized employment in 2nd quarter after exit Median is the number that is in the middle of the series of numbers, so that there is the same quantity of numbers above the median as there are below the median

Credential Attainment Rate Percentage of participants enrolled in education or training program (excluding on the job training (OJT) and customized training) who attain a recognized postsecondary credential or secondary school diploma within 1 year after program exit. Only includes those who received training or education (excluding OJT or Customized Training) Credential can be obtained during the program or within 1 year (365 days) following exit If participant obtains secondary school diploma or equivalent, they must also be employed or in an education/training program leading to a postsecondary credential within 1 year after exit

Who is Included in Credential Attainment Indicator for WIOA Youth What is considered education or training varies based on specific WIOA program For WIOA Youth: All ISY are included Not all OSY are included; only those in the following are included: Occupational skills training program element Secondary education (at or above 9th grade level) Postsecondary education Adult Education (at or above 9th grade level) YouthBuild Job Corps

Credential Attainment Parameters Only included in numerator and denominator of credential indicator 1 time regardless of number of credentials attained Even if obtain secondary school diploma and another credential, only count 1 time in indicator

Types of Credentials Secondary School diploma or recognized equivalent Associate’s degree Bachelor’s degree Occupational licensure Occupational certificate, including Registered Apprenticeship and Career and Technical Education educational certificates Occupational certification Other recognized certificates of industry/occupational skills completion sufficient to qualify for entry-level or advancement in employment

Postsecondary Credential Definition Highlights Awarded in recognition of an individual’s attainment of measurable technical or industry/occupational skills necessary to obtain employment or advance within an industry/occupation Technical or industry/occupational skills based on standards developed or endorsed by employers or industry associations Neither certificates awarded by workforce development boards nor work readiness certificates are included because neither document the measurable technical or industry/occupational skills Must recognize technical or industry/occupational skills for specific industry/occupation rather than general skills related to safety, hygiene, etc., even if general skills certificates are broadly required to qualify for entry-level employment or advancement in employment

Secondary School Diploma Definition A secondary diploma (or alternate diploma) is one that is recognized by a State and that is included for accountability purposes under the Elementary and Secondary Education Act of 1965 (ESEA), as amended by the Every Student Succeeds Act (ESSA). A secondary school equivalency certification signifies that a student has completed the requirements for a high school education. The types of recognized equivalents, for those not covered under ESEA, are those recognized by a State. Further examples of secondary school diplomas/equivalents can be found in TEGL 10-16.

Examples – Credentials that Count Examples of Credentials that Meet the Definition: Certified Nursing Assistant (CNA) License Example of Occupational Licensure Automotive Service Excellence (ASE) Certification Example of Occupational Certification States must follow definition – not feasible to rule on each individual credential

Examples – Credentials that DO NOT count Examples of common certificates that do not meet the credential definition: Occupational Safety and Health Administration (OSHA) 10 Hour Course that provides awareness of job-related common safety and health hazards Work/Career Readiness Certificates Completion of an Assistive Technology training program (e.g., screen reading software)

Measurable Skill Gains Indicator Percentage of program participants who, during a program year, are in an education or training program that leads to a recognized postsecondary credential or employment and who are achieving measurable skills gains, defined as documented academic, technical, occupational, or other forms of progress, towards such a credential or employment. Only count in numerator and denominator 1 time in a reporting period (i.e., program year) regardless of how many skill gains are attained unless the individual has more than 1 period of participation in the program year The measurable skill gains indicator measures progress in a program year; it is not exit-based

Measurable Skill Gains Note: Participants (who are in education or training) are included in the indicator regardless of how long they have participated in the program year Even if an individual is enrolled in June, s/he is still included in the indicator, if the individual meets the parameters for inclusion in the indicator Programs should not delay enrollment or services to participants until a new program year even if programs believe there is insufficient time for the participant to make any type of measurable skill gain by the end of that program year.

Who is Included in Measurable Skill Gains for WIOA Youth What is considered education or training varies based on specific WIOA program For WIOA Youth: All ISY are included Not all OSY are included; only those in the following are included: Occupational skills training program element Secondary education (at or above 9th grade level) Postsecondary education Adult Education (at or above 9th grade level) YouthBuild Job Corps Note: same Youth participants included as in credential indicator – this is different than in the Adult program

Five Types of Measurable Skill Gains Achievement of at least one educational functioning level, if receiving instruction below postsecondary education level The three ways to measure EFL gains are in TEGL 10-16 Attainment of secondary school diploma or equivalent Secondary or postsecondary transcript for sufficient number of credit hours Secondary: transcript or report card for 1 semester Postsecondary: at least 12 hours per semester or, for part- time students, a total of at least 12 hours over 2 completed semesters during a 12 month period

Five Types of Measurable Skill Gains (continued) Satisfactory progress report toward an established milestone from an employer or training provider Passage of an exam required for an occupation or progress attaining technical/occupational skills as evidenced by trade-related benchmarks

Operational Parameters for Measurable Skill Gain for WIOA Youth Measured by the achievement of any of the 5 types of measurable skill gains No specific measurable skill gain types required for specific Youth participants Type of skill gain should be based on youth’s Individual Service Strategy

Effectiveness in Serving Employers System measure reported jointly across all core programs (i.e., youth program does not have a separate employer measure) (States choose 2) Retention with the same employer in the 2nd and 4th quarters after exit Employer Penetration Rate Repeat Business Customer Rate

WIOA Primary Indicators of Performance Data Availability WIOA Primary Indicators of Performance PY 2016 PY 2017 PY 2018 PY 2019 PY2020 Oct. 2017 Oct. 2018 Oct. 2019 Oct. 2020 Oct. 2021 % Of Data Available on Report Delivery Date Employment Rate 2nd Quarter After Exit 0% 100% Employment Rate 4th Quarter After Exit 50% Median Earnings 2nd Quarter After Exit Credential Attainment Rate Measurable Skill Gains ESE - Retention With the Same Employer ESE - Repeat Customer* ESE - Market Penetration *ESE - Repeat Customer indicator will build to a 3 year cohort. PY 2017 will be based on repeat customers in the last year. PY 2018 will be based on repeat customer in the last 2 years. PY 2019 will be the first year the indicator will be based on repeat customers in the last 3 years.

Overview of Negotiations Requirements WIOA 116(b)(3)(A) requires the establishment of levels of performance for each performance indicator: (iv) Negotiations shall take place in two year intervals, for the 1st two years (PY2016 and PY2017), and then 3rd and 4th years. (v) Consideration of four factors when negotiating levels of performance for each indicator; (viii) The statistical model will be used to make adjustments to the negotiated State levels of performance for actual economic conditions and participant characteristics.

Negotiations Requirements The negotiated levels for each indicator are based upon: Comparison with adjusted performance levels of other States; Adjustments of levels using an objective statistical model; Continuous improvement of each State; and The requirements of the Government Performance and Results Act (GPRA).

Statistical Model Requirements The Model must consider States’ actual economic conditions, including; Differences in Unemployment Rate Differences in job losses or gains in particular industries The Model must also consider the characteristics of participants at the time of enrollment; Indicators of poor work history Low levels of literacy or English proficiency Lack of work experience Disability status Lack of educational or occupational skills attainment Homelessness Dislocation from high-wage or high-benefit employment Ex-offender status Welfare dependency

Elements of the Model The model is required to include two types of elements: Participant Characteristics – variables are presented as percentages of the total exiters for the particular cohort. Economic Conditions – variables are presented as percentages of total employment level by industrial sector and the State unemployment rate.

Elements of the Model (cont’d) In addition to required elements, other relevant characteristics of participants served are captured, including: Gender Education Level Age Employment Status at Participation Race/Ethnicity For reference, details of the variables can be found here: https://www.doleta.gov/performance/guidance/docs/WIOA_Statis tical_Model_Methodology_Report-6-24-2016.pdf https://wdr.doleta.gov/directives/attach/TEGL/TEGL_26-15- Attachment-II_Acc.pdf

Implementing the Model: Baseline Indicators Those indicators where there are insufficient data yet to support the statistical model have been designated as Baseline Indicators. Baseline Indicators, by core program title, include: For Title I: Measurable Skill Gains, Effectiveness in Serving Employers, Median Earnings (Youth program only) For Title III: Effectiveness in Serving Employers

Determining Success or Failure Is this my final answer? The model must be applied at the end of the program year to adjust the negotiated State levels of performance for: Actual participant characteristics, and Actual economic conditions

Key Takeaways on Negotiation and the Statistical Adjustment Model for the Youth Indicators Keep in mind that many of the indicators are still in their baseline period Keep in mind that characteristics of hard to serve youth are factored into the statistical adjustment model When negotiating levels keep in mind: Median Earnings impact on youth who work part- time while in postsecondary or training Measurable Skill Gains impact for those who enroll late in the program year

Workforce Innovation and Opportunity Act, Pub. L. 113-1128 Workforce Innovation and Opportunity Act Statute https://www.gpo.gov/fdsys/pkg/PLAW-113publ128/pdf/PLAW-113publ128.pdf WIOA Joint Rule for United and Combined State Plans, Performance Accountability, and the One-Stop System Joint Provisions, 81 FED. REG. 55791 (Aug. 19, 2016) This Joint WIOA Joint Final Rule provides guidance for State and local workforce development systems https://www.gpo.gov/fdsys/pkg/FR-2018-08-19/pdf/2016-15977.pdf Training and Employment Guidance Letter (TEGL) No. 10-16 Performance Accountability Guidance for WIOA Title I, Title II, Title III and Title IV Core Programs https://wdr.doleta.gov/directives/corr_doc.cfm?DOCN=8226

Performance and Results Web Site WIOA Resource Page The Department of Labor (DOL), in coordination with the U.S. Departments of Education (ED) and Health and Human Services (HHS), has worked to prepare everyone for the implementation of WIOA. The WIOA resource page provides information and resources for States, local areas, non-profits and other grantees, and other stakeholders to assist with implementation of the Act. www.doleta.gov/wioa Performance and Results Web Site The Performance and Results Web Site will assist you in understanding how performance is measured, reported, and evaluated at ETA. www.doleta.gov/performance

Post Polling Question If a youth is determined eligible on September 1, receives an assessment and ISS on September 12, and receives their first WIOA youth service on September 22, what is their date of participation? A. September 12 B. September 22 C. September 1 D. They have not yet met the criteria to be a WIOA youth participant.

Ways to Engage and Connect: DOL and your colleagues need to hear from you! 1st things 1st: Become a GPS member and use the Member Directory Share your work Submit a Resource on WorkforceGPS. Send us your lessons learned and successful strategies to share with the field to youth.services@dol.gov Participate in Discussion Threads