Key Considerations in Collecting Student Follow-up Data NACTEI May 15, 2012 Portland, OR Promoting Rigorous Career and Technical Education Programs of Study

2 Overview Background Data Collection Timeline Measurement Populations Threshold for involvement Comparison groups Accountability Data Elements In-school performance Post-program results Issues for Consideration

3 Background: Purpose To promote and improve State and local development and implementation of, and to assess the impact of student participation in, rigorous programs of study (RPOS) in CTE. Requirements: Identify an existing program of study (POS) in at least three local districts: urban, suburban, and rural Implement or enhance the 10 components in OVAE’s POS Design Framework within each site Conduct an annual self-assessment of RPOS implementation Submit data on a set of indicators for students in each site

4 Background: Performance Measures Secondary school completion Technical skill attainment Earned postsecondary credit during high school Enrollment in postsecondary education Enrollment in postsecondary in field/major related to POS Need for developmental course work in postsecondary education Postsecondary credential, certificate, or diploma attainment

5 Background: Evaluation Populations Data are collected on all grade 12 students in each participating site, with analysis focused on three comparison groups: RPOS concentrators Concentrators in other CTE programs All other students: students who have not met the concentrator threshold in any CTE program

6 Data Collection Timeline: Program Year June 2011: In-school data collection begins Technical skill attainment Earned postsecondary credit during high school High school graduation July – December 2011: Student transition January 2012: Postsecondary data collection begins Enrollment in postsecondary education Enrollment in postsecondary in field/major related to POS Need for developmental course work in postsecondary education Postsecondary credential, certificate, or diploma attainment

7 Issue: Measurement Populations ISSUE ‒States are asking local program staff to identify students concentrating in CTE programs SOLUTIONS ‒If consistent, statewide transcript data are available, identify CTE concentrators based on course taking. Absent statewide transcript data, identify CTE concentrators using state criteria WHY? ‒Allowing local staff to identify concentrators is creating significant cross-LEA variation that… ̴Leads to inconsistent identification of students across sites ̴Contaminates treatment and control groups ̴Limits comparisons across sites

8 Issue: Measurement Populations ISSUE ‒ States are developing different concentrator definitions for RPOS sites to boost populations SOLUTION ‒ Employ the same criteria for identifying RPOS concentrators as is used in other CTE programs WHY? ‒ Establishing unique definitions for RPOS students… ̴Creates a unique group of concentrators ̴ ̴Limits comparisons with concentrators in other programs ̴ ̴May cause confusion among local providers

9 Issue: Measurement Populations ISSUE ‒States are assigning RPOS concentrators with dual concentrations into nontraditional fields, based on their last area of concentration, or based on their CIP code (e.g., highest number) SOLUTION ‒ Assign dual concentrators to the RPOS cohort WHY? ‒ Assigning rigorous POS students to another program undermines statistical analysis that… ̴Contaminates treatment and control groups ̴May cause program effects to be assigned to another program

10 Data Elements: Secondary Demographic/Background Gender Race/ethnicity Special population status Programmatic Student ID number High school assessment results CTE concentrator status Technical skill attainment RPOS flag College credit earned Cluster(s)/pathway(s) Secondary completion

11 Data Elements: Postsecondary Programmatic Fall enrollment following high school graduation Need for developmental education ‒ ‒Tested need ‒ ‒Course enrollment Program or major field of study area Credential or degree attainment

12 Issue: Technical Skill Attainment ISSUES ‒Students take multiple assessments within a POS ‒External certification data not available SOLUTIONS ‒Create statewide assessments ‒Collect scores for all assessments administered ‒ Establish data sharing agreements with third party testing agencies ‒ Offer incentives to students for reporting scores on external certifications ‒Collect capstone course grades or certification exam results (if available) CONCERN ‒ Variation in local assessments may limit data comparability across sites

13 Issue: Earned Postsecondary Credit During High School ISSUE ‒ Strategies to track credits vary across sites SOLUTION ‒Track postsecondary credits earned using high school student transcript data CONCERNS ‒Criteria for award of postsecondary credit may vary by site (i.e., grades, exam scores, fees) ‒Students do not know they have earned credits ‒Earned credits may not be applied at the postsecondary level because institutions do not accept them

14 Issue: Enrollment in Postsecondary Education ISSUES ‒ Enrollment data are not available for all in-state public and private postsecondary institutions ‒ States have limited access to enrollment data for institutions located out-of-state SOLUTION ‒ Conduct an administrative record match with the National Student Clearinghouse (NSC) CONCERNS ‒ States must pay to access NSC data ‒ NSC data coverage is limited in some states for private proprietary schools ‒Data may be missing for apprenticeships ‒Match rates for state and NSC data vary

15 Issue: Enrollment in a Related Postsecondary Field/Major ISSUE ‒ Data on students’ secondary and postsecondary programs and majors varies by institution SOLUTIONS ‒ Use CIP codes to identify secondary and postsecondary programs and majors and match programs at the 2-digit code level ‒ Access data through the NSC CONCERNS ‒ Students do not declare a major upon enrollment in some states ‒ NSC data are only available after a student graduates and are often incomplete

16 Issue: Need for Developmental Course Work ISSUES ‒ Data on developmental course work may not be collected statewide ‒ Criteria for determining students’ need for developmental course work differ across programs and institutions SOLUTIONS ‒Collect data from partnering institutions ‒Consider expanding collection to all institutions ‒Establish statewide cutoffs for eligibility CONCERNS ‒States lack policies for collecting this data ‒Data are limited to students who attend a public postsecondary institution in-state ‒The NSC does not collect this data

17 Preparing for Reauthorization Create Technical Assistance Materials Strengthen validity and reliability of measures Clarify data collection processes Identifying best practices Develop Strategies for Assessing Programs of Study Strengthen state data collection capacity Identifying best practices Promote Common Population Definitions and Measures State Perkins Accountability Congress

18