1. WIOA Performance Accountability: A Focus on Youth Performance
September 17, 2018
WIOA Performance Accountability: A Focus on Youth Performance
October 25, 2017

3. Sara Hastings
Unit Chief /Division of Youth Services
DOL/ETA

4. Our Journey Together TA Series Two Day Kick-off Webinars
Series Kick-Off: Where we’ve been, where we are now and where we’re going! Tuesday, October 24, 2017 at 11:30 AM ET
WIOA Youth Eligibility Live Question and Answer Session Tuesday, October 24, 2017 at 2:30 PM ET
Coming Together at the Table--The Power of Youth Committees to Convene, Coordinate and Collectively Impact Youth Lives Wednesday, October 25, 2017 at 11:30 AM ET  
WIOA Youth Performance Accountability   Wednesday, October 25, 2017 at 2:30 PM ET

5. Evan Rosenberg Cesar Acevedo
US Department of Labor, Employment and Training Administration, Division of Youth Services
Cesar Acevedo
US Department of Labor, Employment and Training Administration, Office of Policy Development and Research

6. To provide state and local youth practitioners with what they need to know about performance accountability
Definitions of participant and exit
Primary indicators of performance for WIOA core programs and other workforce programs
Statistical Adjustment Model and Negotiations

7. Quiz Time Which of the following statements is true?
A. If an in-school youth becomes a participant on June 15, 2017, they are included in the credential indicator for Program Year 2016.
B. If an in-school becomes a participant on June 15, 2017, they are included in the measurable skill gains indicator for Program Year 2016.
C. A career readiness certificate counts as a positive outcome in the credential indicator.
D. A youth in postsecondary education in the 2nd quarter after exit, who also works a part-time job 6 hours a week during the second quarter after exit, is excluded from the median earnings indicator.

8. Guidance
TEGL Change 1, comprehensive, joint performance guidance published Dec 2016
TEGL 26-16, use of supplemental wage information guidance
TEGL 3-17, WIOA Annual Performance Report Submission
Data Validation guidance (Joint) to be published by end of 2017

9. Youth Participant
Definition of a Participant: reportable individual who receives services other than self-service or information-only services and has satisfied all applicable programmatic requirements including eligibility determination
For Youth this means:
Eligibility determination
Objective assessment
Individual service strategy
Received 1 of 14 youth program elements
Participation date isn’t until the last of these items occurs

10. Exit
Program exit occurs when the participant no longer receives services for 90 days and has no additional services scheduled
The date of program exit is determined after 90 days of no services have elapsed and no future services are planned. At that point the date of program exit is applied retroactively to the last date of service.
The following services do not delay an exit:
Follow-up services
Self-service
Information-only services
Common exit is encouraged but not required across DOL programs (no common exit with ED programs)

11. Types of Youth Follow-Up Services Post-Exit
There are five WIOA youth program elements that may be provided as follow-up services
If any of these 5 services are provided after exit, they do not trigger new eligibility determination/ new participation and may be coded as follow-up services
Supportive services
Mentoring
Financial literacy services
Services that provide labor market information
Postsecondary preparation activities

12. Primary Indicators of Performance
Six Primary Indicators of Performance:
Employment Rate 2nd Quarter after Exit
-(Education/Employment for youth)
Employment Rate 4th Quarter after Exit
Median Earnings in the 2nd Quarter after Exit
Credential Attainment Rate
Measurable Skill Gains
Effectiveness in Serving Employers

13. Employment/Education/Training Rate 2nd Quarter after Exit
Percentage of participants in education or training activities, or in unsubsidized employment during the second quarter after exit.
Successful outcomes include unsubsidized employment, secondary education, postsecondary education, and occupational skills training in 2nd quarter after exit
Adult indicator only recognizes unsubsidized employment as outcome
Unlike WIA, WIOA success in this indicator is only based on status in the 2nd quarter after exit regardless of status upon program entry. 
For example, if ISY is in high school at program entry and still in high school in the 2nd quarter after exit, they are a success in the indicator even though they were already in high school at program entry.

14. Employment/Education/Training Rate 4th Quarter after Exit
Percentage of participants who are in unsubsidized employment during the fourth quarter after exit from the program.
Successful outcomes include unsubsidized employment, secondary education, postsecondary education, and occupational skills training in 4th quarter after exit
Adult indicator only recognizes unsubsidized employment as outcome
Success based solely on status in 4th quarter regardless of status at participation or in 2nd quarter – not a retention measure

15. Median Earnings in the 2nd Quarter after Exit
The median earnings of participants who are in unsubsidized employment during the second quarter after exit from the program.
Only includes those in unsubsidized employment in 2nd quarter after exit
Median is the number that is in the middle of the series of numbers, so that there is the same quantity of numbers above the median as there are below the median

16. Credential Attainment Rate
Percentage of participants enrolled in education or training program (excluding on the job training (OJT) and customized training) who attain a recognized postsecondary credential or secondary school diploma within 1 year after program exit.
Only includes those who received training or education (excluding OJT or Customized Training)
Credential can be obtained during the program or within 1 year (365 days) following exit
If participant obtains secondary school diploma or equivalent, they must also be employed or in an education/training program leading to a postsecondary credential within 1 year after exit

17. Who is Included in Credential Attainment Indicator for WIOA Youth
What is considered education or training varies based on specific WIOA program
For WIOA Youth:
All ISY are included
Not all OSY are included; only those in the following are included:
Occupational skills training program element
Secondary education (at or above 9th grade level)
Postsecondary education
Adult Education (at or above 9th grade level)
YouthBuild
Job Corps

18. Credential Attainment Parameters
Only included in numerator and denominator of credential indicator 1 time regardless of number of credentials attained
Even if obtain secondary school diploma and another credential, only count 1 time in indicator

19. Types of Credentials Secondary School diploma or recognized equivalent
Associate’s degree
Bachelor’s degree
Occupational licensure
Occupational certificate, including Registered Apprenticeship and Career and Technical Education educational certificates
Occupational certification
Other recognized certificates of industry/occupational skills completion sufficient to qualify for entry-level or advancement in employment

20. Postsecondary Credential Definition Highlights
Awarded in recognition of an individual’s attainment of measurable technical or industry/occupational skills necessary to obtain employment or advance within an industry/occupation
Technical or industry/occupational skills based on standards developed or endorsed by employers or industry associations
Neither certificates awarded by workforce development boards nor work readiness certificates are included because neither document the measurable technical or industry/occupational skills
Must recognize technical or industry/occupational skills for specific industry/occupation rather than general skills related to safety, hygiene, etc., even if general skills certificates are broadly required to qualify for entry-level employment or advancement in employment

21. Secondary School Diploma Definition
A secondary diploma (or alternate diploma) is one that is recognized by a State and that is included for accountability purposes under the Elementary and Secondary Education Act of 1965 (ESEA), as amended by the Every Student Succeeds Act (ESSA).
A secondary school equivalency certification signifies that a student has completed the requirements for a high school education. The types of recognized equivalents, for those not covered under ESEA, are those recognized by a State.
Further examples of secondary school diplomas/equivalents can be found in TEGL

22. Examples – Credentials that Count
Examples of Credentials that Meet the Definition:
Certified Nursing Assistant (CNA) License
Example of Occupational Licensure
Automotive Service Excellence (ASE) Certification
Example of Occupational Certification
States must follow definition – not feasible to rule on each individual credential

23. Examples – Credentials that DO NOT count
Examples of common certificates that do not meet the credential definition:
Occupational Safety and Health Administration (OSHA) 10 Hour Course that provides awareness of job-related common safety and health hazards
Work/Career Readiness Certificates
Completion of an Assistive Technology training program (e.g., screen reading software)

24. Measurable Skill Gains Indicator
Percentage of program participants who, during a program year, are in an education or training program that leads to a recognized postsecondary credential or employment and who are achieving measurable skills gains, defined as documented academic, technical, occupational, or other forms of progress, towards such a credential or employment.
Only count in numerator and denominator 1 time in a reporting period (i.e., program year) regardless of how many skill gains are attained unless the individual has more than 1 period of participation in the program year
The measurable skill gains indicator measures progress in a program year; it is not exit-based

25. Measurable Skill Gains
Note: Participants (who are in education or training) are included in the indicator regardless of how long they have participated in the program year
Even if an individual is enrolled in June, s/he is still included in the indicator, if the individual meets the parameters for inclusion in the indicator
Programs should not delay enrollment or services to participants until a new program year even if programs believe there is insufficient time for the participant to make any type of measurable skill gain by the end of that program year.

26. Who is Included in Measurable Skill Gains for WIOA Youth
What is considered education or training varies based on specific WIOA program
For WIOA Youth:
All ISY are included
Not all OSY are included; only those in the following are included:
Occupational skills training program element
Secondary education (at or above 9th grade level)
Postsecondary education
Adult Education (at or above 9th grade level)
YouthBuild
Job Corps
Note: same Youth participants included as in credential indicator – this is different than in the Adult program

27. Five Types of Measurable Skill Gains
Achievement of at least one educational functioning level, if receiving instruction below postsecondary education level
The three ways to measure EFL gains are in TEGL 10-16
Attainment of secondary school diploma or equivalent
Secondary or postsecondary transcript for sufficient number of credit hours
Secondary: transcript or report card for 1 semester
Postsecondary: at least 12 hours per semester or, for part- time students, a total of at least 12 hours over 2 completed semesters during a 12 month period

28. Five Types of Measurable Skill Gains (continued)
Satisfactory progress report toward an established milestone from an employer or training provider
Passage of an exam required for an occupation or progress attaining technical/occupational skills as evidenced by trade-related benchmarks

29. Operational Parameters for Measurable Skill Gain for WIOA Youth
Measured by the achievement of any of the 5 types of measurable skill gains
No specific measurable skill gain types required for specific Youth participants
Type of skill gain should be based on youth’s Individual Service Strategy

30. Effectiveness in Serving Employers
System measure reported jointly across all core programs (i.e., youth program does not have a separate employer measure)
(States choose 2)
Retention with the same employer in the 2nd and 4th quarters after exit
Employer Penetration Rate
Repeat Business Customer Rate

31. WIOA Primary Indicators of Performance
Data Availability
WIOA Primary Indicators of Performance
PY 2016
PY 2017
PY 2018
PY 2019
PY2020
Oct. 2017
Oct. 2018
Oct. 2019
Oct. 2020
Oct. 2021
% Of Data Available on Report Delivery Date
Employment Rate 2nd Quarter After Exit 0%
100%
Employment Rate 4th Quarter After Exit
50%
Median Earnings 2nd Quarter After Exit
Credential Attainment Rate
Measurable Skill Gains
ESE - Retention With the Same Employer
ESE - Repeat Customer*
ESE - Market Penetration
*ESE - Repeat Customer indicator will build to a 3 year cohort. PY 2017 will be based on repeat customers in the last year. PY 2018 will be based on repeat customer in the last 2 years. PY 2019 will be the first year the indicator will be based on repeat customers in the last 3 years.

32. Overview of Negotiations Requirements
WIOA 116(b)(3)(A) requires the establishment of levels of performance for each performance indicator:
(iv) Negotiations shall take place in two year intervals, for the 1st two years (PY2016 and PY2017), and then 3rd and 4th years.
(v) Consideration of four factors when negotiating levels of performance for each indicator;
(viii) The statistical model will be used to make adjustments to the negotiated State levels of performance for actual economic conditions and participant characteristics.

33. Negotiations Requirements
The negotiated levels for each indicator are based upon:
Comparison with adjusted performance levels of other States;
Adjustments of levels using an objective statistical model;
Continuous improvement of each State; and
The requirements of the Government Performance and Results Act (GPRA).

34. Statistical Model Requirements
The Model must consider States’ actual economic conditions, including;
Differences in Unemployment Rate
Differences in job losses or gains in particular industries
The Model must also consider the characteristics of participants at the time of enrollment;
Indicators of poor work history
Low levels of literacy or English proficiency
Lack of work experience
Disability status
Lack of educational or occupational skills attainment
Homelessness
Dislocation from high-wage or high-benefit employment
Ex-offender status
Welfare dependency

35. Elements of the Model
The model is required to include two types of elements:
Participant Characteristics – variables are presented as percentages of the total exiters for the particular cohort.
Economic Conditions – variables are presented as percentages of total employment level by industrial sector and the State unemployment rate.

36. Elements of the Model (cont’d)
In addition to required elements, other relevant characteristics of participants served are captured, including:
Gender
Education Level
Age
Employment Status at Participation
Race/Ethnicity
For reference, details of the variables can be found here:
tical_Model_Methodology_Report pdf
Attachment-II_Acc.pdf

37. Implementing the Model: Baseline Indicators
Those indicators where there are insufficient data yet to support the statistical model have been designated as Baseline Indicators.
Baseline Indicators, by core program title, include:
For Title I:
Measurable Skill Gains,
Effectiveness in Serving Employers,
Median Earnings (Youth program only)
For Title III:
Effectiveness in Serving Employers

38. Determining Success or Failure
Is this my final answer?
The model must be applied at the end of the program year to adjust the negotiated State levels of performance for:
Actual participant characteristics, and
Actual economic conditions

39. Key Takeaways on Negotiation and the Statistical Adjustment Model for the Youth Indicators
Keep in mind that many of the indicators are still in their baseline period
Keep in mind that characteristics of hard to serve youth are factored into the statistical adjustment model
When negotiating levels keep in mind:
Median Earnings impact on youth who work part- time while in postsecondary or training
Measurable Skill Gains impact for those who enroll late in the program year

40. Workforce Innovation and Opportunity Act, Pub. L. 113-1128
Workforce Innovation and Opportunity Act Statute
WIOA Joint Rule for United and Combined State Plans, Performance Accountability, and the One-Stop System Joint Provisions, 81 FED. REG (Aug. 19, 2016)
This Joint WIOA Joint Final Rule provides guidance for State and local workforce development systems
Training and Employment Guidance Letter (TEGL) No
Performance Accountability Guidance for WIOA Title I, Title II, Title III and Title IV Core Programs

41. Performance and Results Web Site
WIOA Resource Page
The Department of Labor (DOL), in coordination with the U.S. Departments of Education (ED) and Health and Human Services (HHS), has worked to prepare everyone for the implementation of WIOA. The WIOA resource page provides information and resources for States, local areas, non-profits and other grantees, and other stakeholders to assist with implementation of the Act.
Performance and Results Web Site
The Performance and Results Web Site will assist you in understanding how performance is measured, reported, and evaluated at ETA.

42. Post Polling Question
If a youth is determined eligible on September 1, receives an assessment and ISS on September 12, and receives their first WIOA youth service on September 22, what is their date of participation?
A. September 12
B. September 22
C. September 1
D. They have not yet met the criteria to be a WIOA youth participant.

44. Ways to Engage and Connect: DOL and your colleagues need to hear from you!
1st things 1st: Become a GPS member and use the Member Directory
Share your work
Submit a Resource on WorkforceGPS.
Send us your lessons learned and successful strategies to share with the field to
Participate in Discussion Threads