WIA YOUTH PROGRAM COMMON PERFORMANCE MEASURES 1

WHERE’S THE GUIDANCE TEGL : Common Measures Policy for ETA Performance Accountability System and Related Performance Issues replaces the existing guidance with a single unified document. TEGL 17-05, Change 2: The intent of this guidance is to further clarify the application of the literacy/numeracy measure initially provided in Attachment C, Educational Functional Level Descriptors, in Training and Employment Guidance Letter (TEGL) 17-05, issued February 17, 2006, and to rescind TEGL , Change 1, issued August 13, This change TEGL only addresses specific reporting changes for the Workforce Investment Act (WIA) Youth literacy/numeracy measure. 2

Placement in Employment or Education Of those who are not in post-secondary education or employment (including the military) at the date of participation: # of youth participants who are in employment (including the military) or enrolled in post-secondary education and/or advanced training/occupational skills training in the first quarter after the exit quarter ___________________________________________ # of youth participants who exit during the quarter 3

Highlights of Youth Placement in Employment or Education Excludes youth in employment, the military or post-secondary education at participation Employment, military and education status at participation is based on information collected from the individual Employment and military status in the 1 st quarter after exit is based on wage records (supplemental data allowed) Education status in the 1 st quarter after exit is based on administrative records Youth in secondary school at exit are included, consistent with ETA’s vision to ensure youth successfully complete their secondary education 4

Defining Specific Terms Post-Secondary Education Post-Secondary Education – a program at an accredited degree-granting institution leading to an academic degree. Programs offered by degree-granting institutions that do not lead to an academic degree (such as certificate programs) do not count as a placement in post-secondary education but may count as a placement in “advanced training/occupational skills training” 5

Advanced Training/Occupational Skills Training Advanced Training/Occupational Skills Training o An organized program of study that provides specific vocational skills leading to proficiency in performing actual tasks and technical functions required by certain occupational fields at entry, intermediate or advanced levels. Such training should: 1)be outcome-oriented and focused on a long-term goal as specified in the ISS, 2)coincide with exit rather than short-term training 3)result in attainment of a certificate Much tighter definition! 6

7 Placement in Employment or Education Employed, in the military or in post-secondary education at participation? Has a qualifying outcome * in the 1 st quarter after exit? NO YES Excluded YES Numerator Denominator NO Qualifying Outcomes: employment, military, enrolled in post-secondary education, advanced training, or occupational skills training

Attainment of a Degree or Certificate Of those enrolled in education (at the date of participation or at any point during the program): # of youth participants who attain a diploma, GED, or certificate by the end of the third quarter after the exit quarter ________________________________________ # of youth participants who exit during the quarter 8

Highlights of Youth Attainment of Degree or Certificate Youth in secondary school at exit are included, which is consistent with ETA’s vision to ensure youth successfully complete their secondary education Diplomas, GEDs or certificates can be obtained during participation or at any point by the end of the 3 rd quarter after exit Work readiness certificates will not be accepted for this measure Consistent with ETA’s youth vision’s demand driven approach to ensure training consistent with employer needs 9

Defining Specific Terms Education Education – participation in secondary or post- secondary school, adult education programs, or any other organized program of study leading to a degree or certificate Diploma Diploma – any credential accepted by the State educational agency as equivalent to a HS diploma; also includes post-secondary degrees Certificate Certificate – awarded in recognition of an individual’s attainment of technical or occupational skills by specified institutions such as the State educational agency or institution of higher education o Does not include work readiness certificates or certificates awarded by local Boards 10

Certificates: Approved Awarding Institutions A state educational agency, or a state agency responsible for administering vocational and technical education within a state Institution of higher education (including community colleges) A professional, industry, or employer organization or a product manufacturer or developer A registered apprenticeship program A public regulatory agency (e.g., FAA aviation mechanic certification) A program approved by DVA (Veterans’ Affairs) to offer education and training to veterans and other eligible persons under the Montgomery GI Bill Office of Job Corps Institutions of higher education which are formally controlled, or formally sanctioned or chartered by the governing body of an Indian tribe's) 11

12 Attainment of a Degree or Certificate Enrolled in education at participation or any time during participation? Attained diploma, GED or certificate by the end of the 3 rd quarter after exit? YES NO Excluded YES Numerator Denominator NO

Literacy and Numeracy Gains (1 st year) Literacy and Numeracy Gains (1 st year) Of those out-of-school youth who are basic skills deficient: # of youth participants who increase one or more educational functioning levels _____________________________________________ # of youth participants who have completed a year in the program (i.e., one year from the date of first youth program service) plus the # of youth participants who exit before completing a year in the youth program 13

14 Literacy and Numeracy Gains - Subsequent Years Definition Of those out-of-school youth who are basic skills deficient: Number of participants who increase one or more educational functioning levels (EFL) Number of participants who have completed a year in the program (e.g., the anniversary date of their first youth service occurs during the reporting period)

15 Literacy/Numeracy Gains o Based on date of first youth service (not exit based) o Youth participants may be included for up to 3 years if they remain Basic Skills Deficient o Basic Skill Deficiency is determined at youth program eligibility. Basic skill deficiency is defined as meeting one of the 4: Initial Reading < 9.0, Math Level <9.0, Functionally Illiterate is yes, or Limited English is Yes o Participant is included in the measure even if they exit prior to end of the first year

Highlights of Literacy/Numeracy Gains Only common measure that is not exit-based Excludes in-school youth and out-of-school youth who are not basic skills deficient Includes individuals with learning disabilities A positive outcome means the youth must advance one or more Adult Basic Education (ABE) or English as a Second Language (ESL) functioning levels Gains can occur in literacy or numeracy (programs can pre- test at different levels in each category) Educational levels are consistent with Department of Education’s National Reporting System (NRS) 16

17 Literacy and Numeracy Definitions  Literacy = Read at an EFL of 7 or above on approved ABE and ESL assessment instruments.  Numeracy = Perform mathematical operations at an EFL of 7 or above on approved ABE / ESL assessment instrument  Skill Deficient = An EFL of 6 or below in Reading or Math on an approved assessment instrument.  Educational Functioning Level (EFL) are defined by US DOE and they reflect the various reading, speaking, and math skills necessary to function. An EFL roughly equates to two grade levels.  Approved Assessment Instruments :

About the Assessments All out-of-school youth must be assessed in basic reading, writing and math Pre-testing must occur within 60 days of the first youth program service; can use pre-test from up to six months prior to date of first youth service The same standardized assessment must be used for pre- and post-testing Youth should be post-tested by the end of one year of participation and compared to pre-test results obtained during initial assessment 18

About the Assessments (cont’d) If a youth continues to be basic skills deficient after the first 12 months of participation, they should continue to receive training in literacy and/or numeracy skills Youth should be post-tested and included in the measure at the completion of the 2 nd year only if they complete two full years in the program 19

20 Assessment and Service Requirements All Out-of-School Youth must be assessed for Literacy and Numeracy Pre-test must be given within 60 days of Date of First Youth Service o Recommend that the Pre-test be given as close to the 1 st youth service as possible (allows more time for remediation) If a previous assessment was conducted (i.e. by a partner program) within six months prior to Date of First Youth Service, then that test score can be used Assessment can be part of the Individual Service Strategy (ISS) development, which can be 1 st youth service Remediation Service (ABE or ESL) must be provided to all Skill Deficient Youth

21 Assessment and Service Requirements Continued Post-testing only has to be done in a functional area that an individual is skill deficient Skill deficiency, defined at eligibility, is equal to one of 4 factors: <9.0 in Reading or Math, Functionally Illiterate, or Limited English Speaking All participants must be post tested prior to their first youth service anniversary date at least annually to measure educational gain in deficient area(s). If youth is in program more than one year, the last post test from prior year becomes the baseline for measuring educational gains in the subsequent year Once youth have an EFL of 7 in functional area, no longer assess in that area

22 TEGL Change 2 BSD out-of-school youth: A youth who is not attending school (even if the youth has a H.S. diploma or its equivalent) or is attending post-secondary school and is basic skills deficient Literacy/Numeracy outcomes are measured on a yearly basis and are not determined until a full year has elapsed from the youth’s Date of First Youth Service. From that point on, the youth’s Literacy/Numeracy outcomes continue to appear in the quarterly and annual reports until the youth is excluded from or exits the measure The definition of Participation Year has been updated. The start date for year 2 participation is the 1 st youth service date + 1 year +1day, for year 3 start date it is the 1 st youth service date + 2 years + 1 day

23 TEGL Change 2 (Continued) Youth are included in the measure for their first participation year whether or not they complete the full participation year. For the second and third participation years, youth are excluded from the measure if they exit before completing the full participation year Youth are excluded from the measure after their third participation year, even if they are still BSD.

24 TEGL Change 2- Literacy Numeracy If an out-of-school, basic skills deficient youth does not have a post-test date or has a post-test date after the end of a given participation year, then the youth will be included in that year’s denominator only unless the youth has an exit-based exclusion during the participation year. If an out-of-school, basic skills deficient youth does not have pre-test data by the end of the first participation year, then the youth will be included in the denominator only unless he or she had an exit-based exclusion during the year. If the youth still does not have pre-test data by the end of the second or third participation years, the youth will continue to be included in the denominator only unless he or she exited (with or without an exclusion) during the given participation year.

25 Literacy and Numeracy EFL Crosswalk New EFL Old EFLABEESL Basic Skill Deficient 1 1 Beginning ESL Literacy Basic Skill Deficient 2 Low Beginning ESL 3 Beginning ABE Literacy High Beginning ESL 42 Beginning Basic Education Low Intermediate ESL 53 Low Intermediate Basic Education High Intermediate ESL 64 High Intermediate Basic EducationAdvanced ESL 75 Low Adult Secondary EducationExit ESL Not Basic Skill Deficient 86 High Adult Secondary Education

PARTICIPATION AND EXIT 26

Program Participation A participant is an individual who is determined eligible to participate in a program and receives a service funded by the program The participation date is the date the first service is received after the individual is determined eligible 27

Program Exit Exit occurs when a participant does not receive a service (funded by program or partner program) for 90 consecutive days There is no more concept of “hard exit” only what was a “soft exit” The exit date is the date of the last service received by the participant For measures using UI wage records, the exit quarter is the quarter containing the exit date 28

Clarifying Follow-Up Services Youth are required to receive at least 12 months of follow-up services, which are triggered at exit With a “soft exit” approach, do you lose the first 3 months of follow-up? NO Begin follow-up after the expected last service –- if only follow-up services are provided for next 90 days, record an “exit” o However, if youth need additional services, this 90 days provides a “window” to better ensure success (because no official exit was recorded) 29

Exiter A participant who hasn’t received a co- funded service for 90 consecutive days and no future services are scheduled o Two components 1.Hasn’t received a service 2.For 90 consecutive days with no future services scheduled Often referred to as a “soft exit approach” 30

Other Terms Related to Exiter Exit Date: Last date of service (program or partner- funded) o Once a participant completes all services and 90 days has passed, the exit date is applied retroactively to the actual or estimated end date of the last service assigned Exit Quarter: Calendar quarter containing the exit date Exit Cohort: Group of individuals who exit during the same calendar quarter 31

What Can Extend an Exit Date? “Gap in Service” Services provided by other programs 32

“Gap in Service” Final common measures policy uses “gap in service” as opposed to the former “planned gap” Three allowable circumstances, where the condition exists for at least 90 days 1.Delay before beginning of training 2.Health/medical condition of participant/family member 3.Temporary move from the area that prevents participation Gaps in service should be <180 days (from date of last service), although a subsequent gap could be initiated as necessary All gaps must be documented and are subject to audit 33

34 Services that commence participation Services that can extend the exit date Services that DO NOT commence participation or extend the exit date  Self-service or self-directed job search and/or workforce information services (does not pertain to WIA youth)  Staff-assisted job search, job referral, career counseling, skills assessment, testing, job development (working with employer and job seeker), workshops, job clubs  Comprehensive and specialized assessments, such as diagnostic testing and interviewing  Individual or Group counseling, career planning, development of individual employment plan  Case management  Short-term pre-vocational services  Training services (occupational skills, on-the-job training, workplace training, cooperative education, private sector training programs, skill upgrading and/or retraining, entrepreneurial training, job readiness training, adult education and literacy activities in combination with training, customized training  DOL funded One-Stop partner program services  All required and other appropriate partner program services  Trade readjustment allowances and other needs-related payments funded by TAA, WIA or NEG that are tied to continuous participation or other activities  Eligibility determination  Case management administrative activities involving regular contact to obtain information regarding employment status, educational progress, need for additional services, etc.  Income maintenance or support payments (e.g., UI, TANF)  Provide assistance not related to employment services (e.g., giving directions or allowing rest room access)  Post-employment follow-up services designed to ensure job retention, wage gains, and career progress SERVICES THAT TRIGGER PARTICIPATION AND CAN EXTEND EXIT

Further Clarification of DATES Participation and Exit Dates are always dates of service o Participation Date reflects first co-funded service o Exit Date reflects last co-funded service No more “Hard Exit” o Federal policy states that an exit cannot be officially recorded until that 90 days has elapsed o Not intended to take responsibility away from case managers; for WIA, case managers are encouraged to review potential soft exits 35

Placement in Employment and Education and Attainment of a Degree or Certificate measures are “exit-based” measures. Literacy and Numeracy Gains is calculated while in the program. 36

37 Youth Common Measure Summary Common Measures DenominatorNumeratorData Source Placed in Emp. or Ed. Those not in postsecondary ed or employed at participation Q1 post-exitWage records and supplemental data; Administrative records for education or training Attainment of Degree or Certificate Those enrolled in ed. At participation or during program By Q3 post-exitAdministrative records Literacy/ Numeracy All basic skill deficient out-of-school youth who completed a year in program or exited prior to completing a year Participants who increased one or more EFLs Assessment Instrument

Program Design Implication #1: Long-Term Participation Common Measure outcomes are unlikely to be met unless youth participate for extended periods o In-school younger youth must remain engaged until they graduate or receive a GED and go on to employment or post-secondary education o Out-of-school youth who are basic skills deficient will likely need time to improve one or more EFLs o Summer employment programs that enroll WIA youth for the summer only are unlikely to achieve positive outcomes 38

Long-Term Participation Engagement and Retention Strategies Keeping youth connected to an organization or program o Establishing a sense of self-worth through program participation (e.g., they get paid, receive positive feedback; their contributions matter, etc.) Providing incentives o Opportunities to demonstrate skills to family and peers; field trips and events; stipends, opportunities to serve and lead; supportive services; recognition Youth-centered programs o Youth-friendly intake procedures; interpersonal support in programs from personnel, parents, peers; honesty and authenticity in program (provide what’s promised); culturally competent staff and culturally relevant programs 39

Program Design Implication #2: Network of Services Since long-term participation is needed to achieve outcomes, more than ever, WIA youth must have access to a network of services, not a single program or provider o Strategic case management strategies will be needed to connect youth to multiple programs o Develop short-term attainable goals for youth and youth programs to support long-term positive outcomes 40

Network of Services Strategic Case Management Case management is key o Create and maintain positive relationships with youth; have youth see case manager role as connector to a network of services; use team approach to helping youth 41

Network of Services Short-Term Outcomes For Youth: o Setting intermediate goals (outcomes) helps them see progress, which contributes to engagement and retention Think SMART; Set goals WITH the youth; Make it VISUAL For WIA Youth Programs: o WIA outcomes aren’t appropriate to use as program outcomes for one-year contracts; develop intermediate measures to monitor program success Including participation rates, skill/goal attainment rates Balance outcomes with outputs 42

Program Design Implication #3: Assessment When done well, assessment sets the stage for all future services across programs; when done poorly, it can have a negative impact on individual youth success and youth program success o Youth friendly assessments Shorter versions when possible (subtests), pleasant testing environments, don’t send for a test right from the start o Tell youth why they are being tested and how test will be used – don’t retest too soon o Consider other assessments – style, self-awareness, learning style – remove the high-stakes test pressure 43

Which Reports? WIA Quarterly Reports - ETA Form 9090 WIA Annual Reports – ETA Form 9091 WIASRD - WIA Standardized Record Data 44