1. Youth CareerConnect (YCC) Participant Tracking System (PTS) Orientation and Training: Module #3—Quarterly Performance Reporting Presented by: Evan Rosenberg, U.S. Department of Labor (DOL) Alicia Leonard, Mathematica Policy Research March 3, 2015

2. 2 Training Agenda Please use the questions panel in the webinar to ask questions throughout the presentation TopicPresenter IntroductionEvan Rosenberg (DOL) Updates to program status and internship screensAlicia Leonard (Mathematica) Post-program outcomes trackingAlicia Data definitions and Q&AEvan Quarterly performance reports (QPR) generation and submission Alicia Report generation and submission Q&AAlicia Review of QPR calculation specificationsEvan QPR calculation specifications Q&AEvan

3. 3 Introduction

4. 4 Importance of Performance and Reporting System To provide program operators and frontline staff with information for decision making and continuous improvement To provide the U.S. Department of Labor (DOL) with information to ensure program accountability, assist in identifying technical assistance (TA) needs, and aid evaluation To demonstrate program success to stakeholders (for example, community, Congress)

5. 5 QPR Reporting Schedule Reports typically due 45 days following the end of a quarter, but still waiting on OMB approval of our report forms As soon as we receive OMB approval (hopefully some time in March), the first QPR, for the time period of Oct 1, 2014 – Dec 31, 2014 will be due 2 weeks after we receive OMB approval

6. 6 Iterative Rollout of PTS PTS is rolled out in a series of modules over the next several months First module included data entry and search functions Second module included participant service receipt information, outcomes tracking, and grantee-level data tracking Current (third) module: –Quarterly performance reporting –Post-program outcomes –Timing of module rollout Schedule for additional modules: –Export feature (spring 2015) –Random assignment system (mid-2015)

7. 7 Supporting Document Locations Visit the Employment and Training Administration (ETA) Grantee Youth CareerConnect Site to access:Employment and Training Administration (ETA) Grantee Youth CareerConnect Site –Youth CareerConnect PTS: Manual for Participant Enrollment, Service Receipt and Outcomes Tracking, and Grantee-Level Data Entry –PTS data elements and definitions –Recordings of PTS training sessions Click on the Manual link on the top right corner of the PTS banner

8. 8 Data Confidentiality Reminder The PTS will capture data that could be considered personally identifiable information, or PII Some data might not be PII on its own, but when combined with other information it becomes PII (for example, name and date of birth) Some tips for safeguarding PII: –Never email PII (including screenshots of the PTS)! Use the YCC ID (generated by the PTS) if emailing about a participant –Keep hard-copy materials with PII in locked storage –Do not allow unauthorized people to view your computer monitor Contact ycc_support@mathematica-mpr.com if you are unsure whether data are considered PIIycc_support@mathematica-mpr.com Mathematica adheres to DOL’s data security policies and procedures. Additional information is available in the training manual.

9. 9 Updated Program Status and Internship Screens

10. 10 Program Status Attending School update –ONLY affects participants who enter postsecondary education as part of the program model –If Attending School is set to a postsecondary school, a required field appears for the date of the participant’s entry into their first postsecondary school –This date should not be updated if the participant changes to another postsecondary school while in the program Program Participation Date update –Program Participation Date fields restricted to only allow dates after participant enrollment date

11. 11 Total Internship Hours A field has also been added to record total internship hours This field is required when Date Completed and Scored as Work Ready fields have been completed

12. 12 Post-Program Outcomes Information

13. 13 Post-Program Outcomes Post-Program Outcomes tab is available only after you have indicated an exit reason and date for the participant on the Program Status tab Contains four different tabs: –Unsubsidized Employment If you are unsure of the occupational code, click the link above the field Unsubsidized employment history listed at bottom of the screen

14. 14 Post-Program Outcomes (continued) –Postsecondary Education Postsecondary education history listed at bottom of the screen –Please note that this field is only used to enter postsecondary information that has occurred after the program end date

15. 15 Post-Program Outcomes (continued) –Occupational Skills Training If you are unsure of the occupational code, click the link above the field Occupational skills training history listed at bottom of the screen

16. 16 Post-Program Outcomes (continued) –Registered Apprenticeship If you are unsure of the occupational code, click the link above the field Registered apprenticeship history listed at bottom of the screen

17. 17 Data Elements and Definitions: Long-Term Indicators

18. 18 PTS Data Elements and Definitions Data element nameData definitionCode value Date Entered Unsubsidized Employment after Exit Record the date the participant entered unsubsidized employment immediately following program exit or in the first quarter after the quarter of program exit. Note: Each unsubsidized job the participant obtains that meets the DOL definition should be recorded in the PTS. These jobs can occur during program participation or upon completion during the three quarters after exit. YYYYMMDD Industry Code after Exit Record the two-digit NAICS code for the industry of the unsubsidized employment immediately following program exit or in the first quarter after the quarter of program exit. Record 00 if unknown. 00

19. 19 PTS Data Elements and Definitions (continued) Data element nameData definition Code value Occupational Code after Exit Record the eight-digit occupational code that best describes the individual's employment immediately following program exit or in the first quarter after the quarter of program exit using the O*Net Version 4.0 (or later versions) classification system. This information can be based on any job held after exit from the program. Leave blank if occupational code is not available. Note: If all 8 digits of the occupational skills code are not collected, record as many digits as are available. 00000000 Entered Training- Related Employment after Exit Record 1 if the employment in which the individual entered immediately following program exit or in the first quarter after the quarter of program exit is in the industry or occupation that was the industry/occupational focus of the YCC participant. Record 0 if the employment in which the individual entered is not in the industry/occupational focus of the YCC participant. Record NA if not known. Leave blank if the individual has not completed the program. 1 = Yes 0 = No NA = Unknown Blank = does not apply to individual Wage at Employment after Exit Record the hourly wage at employment immediately following program exit or in the first quarter after the quarter of program exit. If employment is based on a salary, divide the annual salary by 2,080 hours worked (40 hour work week for 52 weeks). $0.00

20. 20 PTS Data Elements and Definitions (continued) Data element nameData definition Code value Number of Hours Worked in First Full Week of Unsubsidized Employment after Exit Record the number of hours worked in the first full week of unsubsidized employment immediately following program exit or in the first quarter after the quarter of program exit. Record NA if not known. numeric value NA = Unknown Entered Nontraditional Employment after Exit Record 1 if the participant entered unsubsidized employment immediately following program exit or in the first quarter after the quarter of program exit in an industry in which individuals of the participant's gender or race are not commonly employed. Record 0 if the participant entered unsubsidized employment in an industry in which individuals of the participant's gender or race are commonly employed. Record NA if not known. 1 = Yes 0 = No NA = Unknown Name of Employer after Exit Record the name of the employer in which the participant enters unsubsidized employment immediately following program exit or in the first quarter after the quarter of program exit. Text field

21. 21 PTS Data Elements and Definitions (continued) Data element nameData definitionCode value Employer as Grant Partner after Exit Record 1 if the place of employment immediately following program exit or in the first quarter after the quarter of program exit was a formal grantee partner that participated in program work experiences and/or participant training. Record 0 if the place of employment was not a formal grantee partner. 1 = Yes 0 = No Date Entered Postsecondary Education Record the date the participant entered postsecondary education immediately following program exit or in the first quarter after the quarter of program exit. Postsecondary education is a program at an accredited degree-granting institution that leads to an academic degree (for example, A.A., A.S., B.A., B.S.). Programs offered by degree-granting institutions that do not lead to an academic degree (for example, certificate programs) do not count as a placement in postsecondary education. Note: This is only for postsecondary education that is entered following exit from the program and not as a part of program participation. YYYYMMDD

22. 22 PTS Data Elements and Definitions (continued) Data element nameData definitionCode value Type of Postsecondary Education Record 1 for a public 4-year (or above). Record 2 for a public 2-year. Record 3 for a public less-than-2-year. Record 4 for a private, nonprofit 4-year (or above). Record 5 for a private, nonprofit 2-year. Record 6 for a private, nonprofit less-than-2-year. Record 7 for a private, for-profit 4-year (or above). Record 8 for a private, for-profit 2-year. Record 9 for a private, for-profit less-than-2-year. 1 = public 4-year (or above) 2 = public 2-year 3 = public less-than-2-year 4 = private, nonprofit 4-year (or above) 5 = private, nonprofit 2-year 6 = private, nonprofit less- than-2-year 7 = private, for-profit, 4-year (or above) 8 = private, for-profit 2-year 9 = private, for profit less- than-2-year Postsecondary Education Remediation Record 1 if the participant required remediation as part of his or her postsecondary education courses. Record 0 if the participant did not require remediation. Record NA if unknown. 1 = Yes 0 = No NA = Unknown

23. 23 PTS Data Elements and Definitions (continued) Data element nameData definitionCode value Name of Postsecondary Education School Record the name of the postsecondary education school the participant entered.Text field Date Entered Occupational Skills Training Program – Post-program completion Record the date the participant entered occupational skills training immediately following program exit or in the first quarter after the quarter of program exit. Occupational skills training is an organized program of study that provides specific vocational skills that lead to proficiency in performing actual tasks and technical functions required by certain occupational fields at entry, intermediate, or advanced levels. Such training should: (1) be outcome-oriented and focused on a long-term goal as specified in the individual development plan, (2) be long-term in nature and commence upon program exit rather than being short-term training that is part of services received while enrolled in the program, and (3) result in attainment of a certificate. Note: This is only for occupational skills training that is entered following exit from the program and not as a part of program participation. YYYYMMDD

24. 24 PTS Data Elements and Definitions (continued) Data element nameData definitionCode value Occupational Skills Training Code Record the eight-digit O*Net 4.0 (or later versions) code that best describes the training occupation for which the participant entered occupational training. Record 00000000 or leave blank if occupational code is not available or unknown. 00000000 Name of Occupational Skills Training Provider Record the name of the provider of the occupational skills training.Text field Date Entered Registered Apprenticeship Record the date the participant entered a Registered Apprenticeship immediately following program exit or in the first quarter after the quarter of program exit. Registered Apprenticeship is a unique, flexible training system that combines job-related technical instruction with structured on-the-job learning experiences. Upon completion of a Registered Apprenticeship program, participants receive an industry-issued, nationally recognized, portable credential that certifies occupational proficiency. Registered Apprenticeship requires a written plan designed to move an apprentice from a low- or no-skill entry-level position to full occupational proficiency. Registered Apprenticeship programs must meet parameters established under the National Apprenticeship Act. YYYYMMDD

25. 25 PTS Data Elements and Definitions (continued) Data element nameData definition Code value Name of Apprenticeship Employer Record the name of the employer sponsoring the Registered Apprenticeship.Text field Occupational Code for Registered Apprenticeship Record the eight-digit O*Net 4.0 (or later versions) code that best describes the Registered Apprenticeship occupation for which the participant entered an apprenticeship. Record 00000000 or leave blank if occupational code is not available or unknown. 00000000

26. 26 Data Definitions File Location Most recent version of the PTS data elements and definitions can be found on the ETA Grantee Youth CareerConnect siteETA Grantee Youth CareerConnect site The data elements will be updated periodically with additional data elements as new modules of the PTS are released

27. 27 Questions?

28. 28 QPR Generation and Submission

29. 29 QPR Draft Generation Reports are accessed using the Reports link in the main menu Quarterly Performance Report [DRAFT] Used to review data before finalizing QPR Can be run as often as necessary Contains data that users have access to within the PTS (for example, grantee-level users see data from all schools) Report can be exported into another platform for review, manipulation, and printing. Word or Excel are recommended. –Dropdown menus 1.Report Quarter End Date: Both grantee- and school-level users must select the appropriate date. Please note that users can run the report for the new quarter as soon as it has started. 2.Grantee: The Grantee field will automatically fill for each user. 3.Attending School: a)School-level users will be able to run draft QPRs for all schools they have access to. Please note that users who have access to multiple schools will be able to run the report individually but will not be able to pair a subset of schools. b)Grantee-level users will be able to run the report for all schools or for individual schools.

30. 30 Final QPR Generation Final QPR –Select the report quarter end date, then enter data in Grantee Identifying Information and Report Certification/Additional Comments sections These fields will be pulled into the final QPR PDF All fields are required to generate the final QPR Users will be able to generate the final QPR for quarters one day after their end date –Select Generate Final QPR button and save PDF to your local drive for submission to DOL Please note that this file’s name will be automatically changed to the respective quarter end date and a shortened version of its grantee name

31. 31 QPR Submission QPRs are submitted to DOL using the Youth CareerConnect FileXchange Site (https://ycc-pts.com/FX)https://ycc-pts.com/FX –Log-in information for each grantee’s main PTS contact is the same as was used to upload PTS user files Contact ycc_support@mathematica-mpr.com if you need this information resentycc_support@mathematica-mpr.com Upload QPR files –Browse your computer for location of saved, final QPR Upload one file at a time Cannot remove files that have been uploaded, but can upload new version –Select File Type Date of QPR quarter end date –Name of staff submitting the file(s) –If successful, file will appear on right side of screen in appropriate section and “Upload successful” text will appear

32. 32 Narrative Submission Narrative reports are uploaded using the same process as QPRs. Narrative reports are submitted to DOL using the Youth CareerConnect FileXchange Site (https://ycc-pts.com/FX)https://ycc-pts.com/FX –Log-in information for each grantee’s main PTS contact is the same as was used to upload PTS user files Contact ycc_support@mathematica-mpr.com if you need this information resentycc_support@mathematica-mpr.com Upload narrative report files –Browse your computer for location of saved, final narrative report Upload one file at a time Cannot remove files that have been uploaded, but can upload new version –Select File Type Date of QPR quarter end date –Name of staff submitting the file(s) –If successful, file will appear on right side of screen in appropriate section and “Upload successful” text will appear

33. 33 QPR and Narrative Report Submission to DOL Click Notify DOL button Complete QPR submission email notification –Add email addresses for all recipients of the final documentation –Enter any necessary comments related to the upload/submission process Information related to the report itself should be included in the submitted narrative Click the Send Email Notification to DOL & Grantee button –Please be sure that your DOL federal project officer’s email is in the To field and that your grantee contact is in the CC field. Please note that multiple emails within the same field must be separated by a semicolon. –The email notification will also automatically be sent to ycc.ycc@dol.gov and ycc_support@mathematica-mpr.com, you do not need to enter these email addresses separately.ycc.ycc@dol.gov ycc_support@mathematica-mpr.com You will receive an indication on the main upload screen if the email was successfully sent.

34. 34 QPR Calculation Specifications

35. 35 Quarterly Performance Report QPR includes 8 sections QPR includes 5 columns: previous quarter, current quarter, program year-to-date, cumulative grant-to-date, and grantee target –Quarters based on calendar quarters (i.e., Jan – Mar) –Program year-to-date based on DOL program year of July 1 – June 30 –Cumulative grant-to-date = total since beginning of grant –Targets included are both grantee specific targets set in performance matrix and national targets where applicable Section A – basic grantee information

36. 36 QPR Section B Section B – grant summary information of key participant counts –New Participants Enrolled – unduplicated count of participants for enrollment rate; participants only go into this count one time in the quarter in which they are enrolled –Participants Receiving Program Services – count of participants each quarter and year who receive services; this is a duplicated count in the cumulative grant to date column and is used in the participants served count in performance matrix –Participants Completed Program – number of participants who have successfully completed the program; drives program completion rate –Participants Exited Program – number of participants who are no longer in the program; includes both those who have successfully completed and those who left the program prior to successful completion

37. 37 QPR Section C: Participant Summary Information Section C includes the basic demographic counts for the demographic information collected in the PTS Cumulative Grant-to-date displays totals for each of the participant characteristics –Gender –Ethnicity/Race –Other Demographic information such as disability, homeless, offender, etc. –Grade at time of enrollment

38. 38 QPR Section D: Program Services Section D includes service information on YCC Participants Nine types of services and one category for staff professional development –Line D1 is count for staff professional development –Lines D2 – 10 counts for YCC participant services Career/Academic Counseling Community Service Learning Mentoring Work Experience Leadership Development Supportive Service Internship Employer-Provided Service Industry-Specific YCC Course

39. 39 QPR Section E: Participant Outcomes Counts of various YCC participant outcomes Seven different outcomes tracked –Entered Unsubsidized Employment –Entered Training-Related Employment –Entered Long-Term Occupational Skills Training –Entered Post-Secondary Education –Earned Industry-Recognized Credential –Earned High-School Diploma –Entered Registered Apprenticeship Two sections (E.i. and E.ii.) –E.i. includes counts of outcomes for all participants –E.ii. Includes only counts of outcomes for YCC completers

40. 40 QPR Section F: Short-Term Performance Indicators Includes the 10 short-term indicators included in the SGA All short-term indicators are measures while participants are in the program Numerator, denominator and rate displayed for each indicator –Important context (i.e., is 0% 0/0 or 0/10) Targets included for each indicator (some targets set by grantee based on program model; others have national targets that are the same across grantees

41. 41 QPR Section F: Short-Term Performance Indicators Enrollment Rate – the number of participants enrolled in the program compared to the enrollment target –Calculation: new participants enrolled divided by enrollment target –Example: 80 participants enrolled and enrollment target of 100; 80/100 = 80% Attendance Rate – the rate of school attendance by participants in the program –Calculation: number of total school days (i.e., in quarter or in year) minus school days missed divided by total school days –Example: 70 school days in quarter; 4 days absent; 70-4/70 = 94%

42. 42 QPR Section F: Short-Term Performance Indicators Chronic Absence Rate - the percentage of participants who have missed 10 percent of school days for any reason-excused or unexcused -- as well as suspensions –Only while participants are in high school (grantees with post- secondary models don’t include participants in post-sec) –Calculation: number of active high school participants whose attendance is below 90% in report period divided by total active high school participants –Example: 10 HS participants with attendance below 90%; 200 active HS participants; 10/200 = 95% Mentoring Rate - percentage of participants who have matched one- on-one mentors and participate in formal mentoring –Calculation: number of participants in mentoring in report period divided by total active participants –Example: 50 participants in mentoring; 200 active participants: 50/200 = 25%

43. 43 QPR Section F: Short-Term Performance Indicators Yearly Program Retention Rate – percentage of participants who continue in program from one school year to the next –Calculation For Year 2: Number of participants enrolled in year 1 who participate in year 2 divided by number of participants enrolled in year 1 –Example: 100 participants enrolled in year 1; 90 of those participants continue into year 2; 90/100 = 90% Work Readiness Attainment Rate – the percentage of participants who are deemed work ready based on an employer evaluation conducted at the end of each internship –Calculation: Number of participants who complete an internship and score as work ready divided by number of participants who complete an internship –Example: 80 participants complete an internship; 75 of those score as work ready; 75/80 = 94%

44. 44 QPR Section F: Short-Term Performance Indicators Internship Placement Rate – the percentage of program participants who begin an internship –Calculation: the number of participants who enter an internship divided by the number of active participants –Example: 60 participants entered an internship in the report period; 200 active participants; 60/200 = 30% Internship Completion Rate - of those who begin an internship, the percentage who complete –Calculation: the number of participants who complete an internship divided by the number of participants who enter an internship –Example: 60 participants enter an internship; 50 complete an internship; 50/60 = 83%

45. 45 QPR Section F: Short-Term Performance Indicators Post-Secondary Credit Attainment Rate – the percentage of participants who complete a course that leads to the attainment of post-secondary education credit from courses taken during the program –Dual enrollment (at HS or IHE), AP, IB, certification based on articulation agreements –Calculation: the number of participants who complete a course that leads to post-sec credit divided by total active participants –Example: 190 participants complete course that leads to post-sec credit; 200 active participants; 190/200 = 95% Average Post-Secondary Credit Hours Attained – the average number of post-secondary credits attained per participant while in the program –Based on courses described in above measure –Calculation: total number of post-sec credit hours divided by number of participants with post-sec credit hours attained –Example: 500 post-sec credits; 190 with post-sec credit hours; 500/190 = 2.6 credits

46. 46 Section G: Long-Term Performance Indicators Includes the 5 long-term indicators included in the SGA All long-term indicators are measures of outcomes following program completion or exit Numerator, denominator and rate displayed for each indicator –Important context (i.e., is 0% 0/0 or 0/10) Targets included for each indicator (some targets set by grantee based on program model; others have national targets that are the same across grantees

47. 47 Section G: Long-Term Performance Indicators Final Program Retention Rate – the percentage of participants who complete the program, of those who enter the program –Measured at end of grant for 4 year models; measured earlier for models less than 4 years –Calculation: the number of participants who successfully complete the program divided by participants who reach their expected cohort completion date –Example: 80 participants successfully complete program; 100 reach cohort completion date; 80/100 = 80% High School Diploma Attainment Rate – the percentage of participants who attain a high-school diploma –Based on state definition of high school diploma/equivalent –Measured at end of grant for 4 year models; measured earlier for models less than 4 years –Calculation: number of participants who attain a HS diploma divided by number of participants who reach their expected HS graduation date –Example: 80 participants attain HS diploma; 100 participants reach expected HS graduation date; 80/100 = 80%

48. 48 Section G: Long-Term Performance Indicators Credential Attainment Rate – the percentage of participants who attain an industry- recognized credential –Measured at end of grant for 4 year models; measured earlier for models less than 4 years –Calculation: number of participants who attain industry-recognized credential divided by number of participants who reach expected cohort completion date and are in industry/occupation that can result in a credential during the program –Example: 75 participants attain credential; 100 participants reach expected cohort completion date and in industry that can result in credential; 75/100 = 75% Diploma and Credential Attainment Rate – the percentage of participants who attain a high-school diploma and credential in the specified H-1B industry or occupation –Measured at end of grant for 4 year models; measured earlier for models less than 4 years –Calculation: number of participants who attain a HS diploma AND industry- recognized credential divided by number of participants who reach expected cohort completion date and are in industry/occupation that can result in a credential during the program –Example: 70 participants attain HS diploma and credential; 100 participants reach expected cohort completion date and in industry that can result in credential; 70/100 = 70%

49. 49 Section G: Long-Term Performance Indicators Placement Rate – the percentage of participants who are placed in one of the following: unsubsidized employment, post- secondary education, occupational skills training, or Registered Apprenticeship –Measured at end of grant for 4 year models; measured earlier for models less than 4 years –Calculation: number of participants who are in one of the four placement outcomes by the end of the first quarter after exit from the program divided number of participants who exit the program –Example: 20 participants in unsubsidized employment, 30 in post-secondary education; 10 in occupational skills training, and 5 in a Registered Apprenticeship; 100 participants have exited the program; 65/100 = 65%

50. 50 Questions?

51. 51 Contact Information Questions related to the definitions of data elements recorded in the PTS, reporting requirements, and policy or programmatic issues should be directed to DOL: ycc.ycc@dol.gov Questions related to technical issues logging into or operating the PTS or FileXchange site should be directed to Mathematica’s YCC PTS technical support team: ycc_support@mathematica-mpr.com –Remember to never send PII via email (including in screenshots from the PTS)! Use the YCC ID to identify records.

52. 52 FAQs and Troubleshooting I forgot my password for the FileXchange site. –If you have forgotten your log-in information for the Youth CareerConnect FileXchange site, your grantee’s main PTS contact person should contact Mathematica (ycc_support@mathematica-mpr.com).ycc_support@mathematica-mpr.com How do I review my QPR before generating the final version? –You may run and review the Quarterly Performance Report [DRAFT] from the Reports Listing screen as many times as needed before generating the final QPR. When I export the Quarterly Performance Report [DRAFT] as a PDF, the formatting is difficult to read. –The reports are best exported into Word or Excel, where you can then adjust the print layout so that the report prints neatly. We recommend shifting the report into landscape view and specify its size as one page wide by four pages long. –If you have additional questions about outputting reports, contact ycc_support@mathematica-mpr.com with a description of the issue. ycc_support@mathematica-mpr.com I have questions about how the report is being calculated. –Please first review Appendix B of your PTS Training Manual for the QPR calculation specifications. If you have further questions, send an email to ycc.ycc@dol.gov.ycc.ycc@dol.gov

53. 53 Questions?