1. The work reported herein was supported under the National Research Center for Career and Technical Education, PR/Award (No. VO51A070003) as administered by the Office of Vocational and Adult Education, U.S. Department of Education. However, the contents do not necessarily represent the positions or policies of the Office of Vocational and Adult Education or the U.S. Department of Education and you should not assume endorsement by the Federal Government. Pradeep Kotamraju, Ph.D. Deputy Director Principal Investigator Oscar Aliaga Ph.D. Program Director Emily Dickinson Graduate Assistant

2. Year 2 Year 3 Year 4 Technical Skills Crosswalk ValidationNCES Data AnalysisROI Developed Inventory Template Populated Inventory Template with Data Submitted by States Linked 3 rd Party Industry Certification to LMI Figure 1:The Work So Far Built a Common Postsecondary CTE Data Dictionary Analyzed Data To Test Common Postsecondary CTE Data Dictionary Jointly with the Center for Education and the Workforce (CEW), Georgetown U, Producing a Report on 2018 Employment Projections by Career Clusters Working on NCES Restricted Sample Survey Data to Analyze CTE Engagement, Achievement, and Transitions Community College Completions Using IPEDS Data; Published in Community College Journal of Research & Practice Developing an ROI Guidebook Neutral Intermediary Chaired NSWG on POS Measurement, Accountability, and Evaluation Produced a Brief on ROI for the USA TVET Consortium Made a Presentation on ROI at the ACTE Annual Conference An NRCCTE Report: Common Postsecondary CTE Data Dictionary

3. Technical Skills Crosswalk Validation NCES Data AnalysisROI Produce a Final Inventory Conduct Two Sub-Analyses (one on Soft Skills and the Other on Technical Skills and Connect to CTE Engagement, Achievement, and Transitions Using State-level Data Figure 2:The Work For Year 5 (and Beyond?) Develop and Validate Existing OVAE/Career Clusters Crosswalk State-Level Analysis on 2018 Employment Projections by Career Clusters/ Pathways Continue Work on NCES Restricted Sample Survey Data to Analyze CTE Engagement, Achievement, and Transitions Refining the ROI Guidebook Neutral Intermediary Produce Presentations, Reports, Monographs, Journal Article Submissions Preliminary Analysis of Using Existing Data Sets to Develop ROI for CTE As requested by Center Partners and by External Stakeholders

4. In Center Years 2 and 3 (August 1, 2008-July 31, 2010), data collection began and continued in which states were asked to submit information that corresponded with the elements of the template. Fourteen states submitted information that has been reformatted and entered into the template. Analysis of the information points to the following:  The top seven program areas (based on two-digit CIP codes) are IT, Health, Business, Manufacturing, Construction, Communications, and Agriculture.  Third-Party Certification/Technical Exams, State-Developed Program of Study Exams, and State-Developed End-of-Program Exams appear to be most commonly used in states. Outstanding issues to be resolved include:  Status of Postsecondary TSAs (no information has been provided as yet to the Center).  Program of Study level assessments – the extent to which the postsecondary partner has a role and input.  The shape, form and extent of the relationship of states with third-party TSA organizations. 4 Technical Skills Assessment

5. In Center Year 4(August 1, 2010-July 31, 2011), the NRCCTE was requested by the Office of Vocational and Adult Education (OVAE), U.S. Department of Education (USDE) to connect industry skill certifications, currently being offered within different state educational programs, to available labor market information (LMI). The analysis required obtaining data on industry skill certifications, educational program and CIP codes, labor market information, and a crosswalk connecting CIP codes and occupational (SOC) data. The methodology used: Converted the data states submitted to NRCCTE as part of the technical skills assessment (TSA) inventory template project into a database that could be used in the analysis. The template data was collected in a Microsoft Excel spreadsheet. The spreadsheet information was reformatted so that it could be imported into a Microsoft Access database. 5 Technical Skills Assessment

6. The crosswalk information, which was only available as a Microsoft Excel spreadsheet, was also reformatted for import into the same Microsoft Access database. The wage information was downloaded as a Microsoft Excel workbook from the U.S. Department of Labor and appropriate wage information was reformatted for import into the same Microsoft Access database. A database with all connected information was then assembled. The database contains 139 third-party, industry-based technical skill certifications, 111 secondary education programs, 56 CIP codes, and 110 SOC codes for the five states. 6 Technical Skills Assessment

7. 7 CIP Family Arts, Design, Entertainment, Sports and Media Construction and Extraction Healthcare Practioner and Technical Healthcare Support Installation, Maintenance, and Repair Office and Administrative Support Personal Care and Service Production Protective Services Sales and Related Transportation and Material Moving Total Industry Skill Certification AGRICULTURE, AGRICULTURE OPERATIONS, AND RELATED SCIENCES. 1 1 2 1 5 BUSINESS, MANAGEMENT, MARKETING, AND RELATED SUPPORT SERVICES. 4 2 2 8 COMMUNICATIONS TECHNOLOGIES/ TECHNICIANS AND SUPPORT SERVICES. 1 2 10 13 COMPUTER AND INFORMATION SCIENCES AND SUPPORT SERVICES. 22 CONSTRUCTION TRADES. 112 7 18 8 145 ENGINEERING TECHNOLOGIES AND ENGINEERING-RELATED FIELDS. 1 1 FAMILY AND CONSUMER SCIENCES/HUMAN SCIENCES. 3 3 HEALTH PROFESSIONS AND RELATED PROGRAMS. 16 2 9 HOMELAND SECURITY, LAW ENFORCEMENT, FIREFIGHTING AND RELATED PROTECTIVE SERVICES. 4 4 MECHANIC AND REPAIR TECHNOLOGIES/ TECHNICIANS. 5 31 24 1 61 PERSONAL AND CULINARY SERVICES. 5 2 7 PRECISION PRODUCTION. 22 TRANSPORTATION AND MATERIALS MOVING. 10 1 2 13 VISUAL AND PERFORMING ARTS. 4 4 All CIP Family 71271738335814311317 Tables 1: Relating CIP and Occupation Families: No. of 3 rd Party Industry Certifications

8. 8 CIP Family Arts, Design, Entertainment, Sports and Media Construction and Extraction Healthcare Practioner and Technical Healthcare Support Installation, Maintenance, and Repair Office and Administrative Support Personal Care and Service Production Protective Services Sales and Related Transportation and Material Moving Total Wage AGRICULTURE, AGRICULTURE OPERATIONS, AND RELATED SCIENCES. $19,630$17,190$16,010$15,680$17,128 BUSINESS, MANAGEMENT, MARKETING, AND RELATED SUPPORT SERVICES. $24,835$22,920$15,730$21,162 COMMUNICATIONS TECHNOLOGIES/ TECHNICIANS AND SUPPORT SERVICES. $20,210$18,195$18,860$19,088 COMPUTER AND INFORMATION SCIENCES AND SUPPORT SERVICES. $23,925 CONSTRUCTION TRADES. $25,453$29,249$19,390$29,448$25,885 ENGINEERING TECHNOLOGIES AND ENGINEERING-RELATED FIELDS. $24,620 FAMILY AND CONSUMER SCIENCES/HUMAN SCIENCES. $16,230 HEALTH PROFESSIONS AND RELATED PROGRAMS. $21,610$19,322$20,410$20,447 HOMELAND SECURITY, LAW ENFORCEMENT, FIREFIGHTING AND RELATED PROTECTIVE SERVICES. $23,990 MECHANIC AND REPAIR TECHNOLOGIES/ TECHNICIANS. $28,922$23,005$22,344$27,220$25,373 PERSONAL AND CULINARY SERVICES. $23,298$16,890$20,094 PRECISION PRODUCTION. $21,820 TRANSPORTATION AND MATERIALS MOVING. $26,104$29,500$18,855$24,820 VISUAL AND PERFORMING ARTS. $24,620 All CIP Family $22,270$26,826$21,610$18,256$26,127$22,563$16,120$21,517$23,990$15,705$25,174$21,833 Table 2: Relating CIP and Occupation Families: Average Entry-Wage for 3 rd Party Industry Certifications

9. The NRCCTE convened a workgroup consisting of researchers, data experts, and policymakers who met on January 12 th and 13 th, 2010, at the University of Louisville. The purpose of this meeting was to capture the evolving multi-state, multi- institutional collaborative efforts seeking greater consistency and clarity in Perkins secondary and postsecondary data collection and reporting, The goal of this meeting was to continue to move toward a common data crosswalk that links educational programs to occupations with career clusters and career pathways and create a foundation for more standardized accountability requirements in later iterations of the Perkins legislation. Concluding the meeting, participants recommended that the Center undertake the following tasks:  Document and validate the decision rules that connect occupations to educational programs, career clusters, and career pathways in several of the existing crosswalks.  Analyze and discuss the different units of analyses that are currently operational within several crosswalks, and outline for a plan for incorporating programs of study as a unit of analysis for crosswalks.  Develop a plan for disseminating the procedures, processes and products to data experts, researchers and policymakers. The project continues into Year 4 and is being jointly led by the University of Louisville and the NASDCTEc. Jointly with the Center for Education and the Workforce (CEW), Georgetown U, Producing a Report on 2018 Employment Projections by Career Clusters 9 Crosswalk Validation Project

10. Traditionally, only the total number of CTE credits enrolled and completed either across all areas or within one area are used to identify the extent of participating or concentrating in CTE. A three-credit threshold is used to divide CTE students into participants and concentrators The NRCCTE has begun exploring, examining, and analyzing credits taken within an occupational area, across all occupational areas, and the number of occupational areas in which high school graduates participate and concentrate The main source is the US Department of Education sample survey data, specifically the 2005 NCES High School Transcript Studies (HSTS) data and the Education Longitudinal Study (ELS) data 10 The NRCCTE CTE TYPOLOGY

11. 11 Conventional Way of Defining A CTE student Different denominations: concentrator, participant, investor … “Traditional” classification in large surveys: Academic track Vocational track Neither and both (dual)—as defined in the NCES sample survey NCES: Fulfillment of an occupational area if 3 or more credits are taken in that area (“occupational concentrator”) At the State Level: different levels, different goals

12. A Methodological Rationale for Measuring, Accounting, and Evaluating CTE

13. The purpose is to understand, the level, mix, and intensity of CTE course-taking within and across different occupational areas. The NRCCTE has expanded the participant-concentrator dichotomy by creating a taxonomy that more accurately represents CTE course-taking. Using the Classification of Secondary School Courses (CSSC) different parts of the high school graduates’ transcripts are analyzed Reformulating CTE Course Taking

14. 14 Following along the lines of NCES, the basic steps are as follows:  Identify CTE and Academic courses  Divided CTE into 13 occupational areas  Divided Academic into different subject areas, including Math and Science  Categorized Math & Science courses by three levels: Less than Basic, Basic, and Advanced  Identified Academic and CTE courses which are in sequence and further divided by No Sequence/Sequence  Determined the last grade year Math & Science course taken Reformulating CTE Course Taking

15. Dividing CTE course taking patterns of high school graduates (students) into three facets:  Identify how many credits high school graduates takes and completes within each occupational area  Determine the total number of CTE credits enrolled and completed across all occupational areas  Count the number of occupational areas in which each graduate enrolls and completes courses A crosstab has been built that relates the above three facets creating the following (shown on the next slide)

16. 16 Total CTE credits 0 - < 3 CTE Credits within an Occ. Area 0>0 -< 11>1 and <33>3 No of Occ. Areas Zero Occ. Area 0 Not Applicable Single Occ. Area 1 Multiple Occ. Area 2 Total CTE credits >=3 CTE Credits within an Occ. Area >0 -< 11>1 and <33>3 No of Occ. Areas Single Occ. AreaNot Applicable 4 Multiple Occ. Area 3 Reformulating CTE Course Taking No CTE Course Taking Concentrator Participant 0 is No CTE Course Taking; 1,2, and 3 are classified as Participants; 4 is classified as Concentrators

17. 17 Total CTE credits 0 - < 3 CTE Credits within an Occ. Area 0>0 -< 11>1 and <33>3 No of Occ. Areas Zero Occ. Area 8% Not Applicable Single Occ. Area 19% Multiple Occ. Area 30% Total CTE credits >=3 CTE Credits within an Occ. Area >0 -< 11>1 and <33>3 No of Occ. Areas Single Occ. AreaNot Applicable 17% Multiple Occ. Area 27% Reformulating CTE Course Taking No CTE Course Taking Concentrator Participant 0 is No CTE Course Taking; 1,2, and 3 are classified as Participants; 4 is classified as Concentrators

18. 18 CTE Credit Categorization% No CTE credits 8.0 More than 0 and less than 1 CTE credits, no occupational area fulfilled 7.5 1 CTE credit, no occupational area fulfilled 10.7 More than 1 and less than 3 CTE credits, no occupational area fulfilled 29.9 3 CTE credits, no occupational area fulfilled 7.6 More than 3 CTE credits, no occupational area fulfilled 19.6 3 CTE credits, 1 occupational area fulfilled 0.8 More than 3 credits, at least 1 occupational area fulfilled 16.0 Frequencies of CTE Credit Categorizations (n=2,698,609)

19. 19 0 1234567 Weighted n Female 9.08.111.430.87.618.40.913.8 1,378,033 Male 6.96.89.928.97.620.80.718.3 1,320,577 White 8.37.410.127.87.121.20.817.41,633,287 Black 8.05.78.933.57.817.91.316.9379,133 Hispanic 7.79.212.231.68.817.70.812.1421,325 Asian 10.39.817.034.97.212.00.38.4122,181 Other 7.59.821.070.216.546.81.326.7142,684 SES-Q1 (Low) 5.96.09.729.57.820.60.919.6640,172 SES-Q2 5.76.68.628.97.223.60.519.0687,844 SES-Q3 8.76.410.330.78.219.50.815.3703,344 SES-Q4 (High)11.711.014.130.47.114.51.010.2667,249 Group 0: No CTE credits Group 1: More than 0 and less than 1 CTE credits, no occupational area fulfilled Group 2: 1 CTE credit, no occupational area fulfilled Group 3: More than 1 and less than 3 CTE credits, no occupational area fulfilled Group 4: 3 CTE credits, no occupational area fulfilled Group 5: More than 3 CTE credits, no occupational area fulfilled Group 6: 3 CTE credits, 1 occupational area fulfilled Group 7: More than 3 credits, at least 1 occupational area fulfilled Frequencies of Gender, Race and Socioeconomic Status by CTE Credit Categorizations for Public High School Students

20. Students taking more than 3 CTE credits, with or without a focus on an occupational area, are 36% to 59% less likely to drop out of high school than those taking fewer CTE credits. Students who have taken 3 or more CTE credits, fulfilling the requirements of an occupational area or not, are at least 43% more likely to attend 2-year colleges than those taking fewer CTE credits. Based on high school graduate transcripts that describe CTE and academic course taking, and using conventional definitions of college, career, and college and career readiness, early estimates for each are 42%, 28%, and 18%, respectively. 20 Preliminary Results

21. 21

22. Kotamraju, P. (2011, June). A new taxonomy of high school career and technical education (CTE): Describing the engagement, achievement, and transitions of CTE students. Paper presented at the 15 h Annual Career Clusters Institute, Atlanta, GA. Kotamraju, P. (in press—expected publication date, Summer 2011). Vocational education and training quality and evaluation: Its place in the U.S. community college. In A. Barabasch & F. Rauner (Eds.), The art of integration: Work and education in America. Berlin, Germany: Springer Verlag Press. Kotamraju, P. (2011). Meeting the 2020 American Graduation Initiative (AGI) goal of increasing postsecondary graduation rates and completions: A macro perspective of community college student educational attainment. Community College Journal of Research and Practice, 35(1-2), 1-18. Kotamraju P., Richards, A., Wu, J., & Klein, S. (2010, April). A common postsecondary data dictionary for Perkins accountability. Louisville, KY, and Berkeley, CA: National Research Center for Career and Technical Education, University of Louisville, and MPR Associates, Inc. Retrieved from http://136.165.122.102/UserFiles/File/Tech_Reports/Postsecondary_Data_Dictionary_Rep ort_WEB.pdf http://136.165.122.102/UserFiles/File/Tech_Reports/Postsecondary_Data_Dictionary_Rep ort_WEB.pdf 22 Stakeholder Contacts: Publications

23. Kotamraju, P. (2011, May). CTE Measurement, Accountability and Evaluation: The NRCCTE’s Comprehensive Strategy for Technical Assistance. Paper To Be Presented at the National Association of Career and Technical Education Information annual Meeting, Philadelphia, Pennsylvania. Kotamraju, P. (2011, May). A New Taxonomy for Career and Technical Education (CTE): Measuring Engagement, Achievement, and Transitions of CTE Students. Paper To Be Presented at the National Association of Career and Technical Education Information annual Meeting, Philadelphia, Pennsylvania. Kotamraju, P. (2011, April). Getting Timing and Sequencing of Math and Science Course-Taking Right: A Possible Answer to the College and Career Readiness of High School Graduates Concentrating on Career and Technical Education (CTE). Paper presented at 2011 Council for the Study of Community Colleges Annual Conference. New Orleans. Aliaga, Oscar. (2011, April). Career and Technical Education Graduates and Enrollment in 2-Year Colleges. Paper presented at the annual meeting of the Council for the Study of Community Colleges. New Orleans, Louisiana. Kotamraju, P. (2011, April). The college and career readiness of U.S. high school graduates: From concept to measurement. Paper to be presented at the annual convention of the American Educational Research Association. New Orleans, LA. 23 Stakeholder Contacts: Paper Presentations

24. Kotamraju, P. (2010, December). A tool kit for measuring CTE effectiveness using return on investment and other related techniques: A basic introduction. Paper presented at the annual convention of the Association for Career and Technical Education, Las Vegas, NV. Kotamraju, P. (2010, October). Meeting the 2020 AGI completion goal: The role of the community college. Paper presented at the annual convention of the National Council of Workforce Education, Washington, DC. Kotamraju, P. (2010, June). Building a Technical Skills Inventory Database One State at a Time,” 14 th Annual Career Clusters Institute, Denver, Colorado. Kotamraju, P. (2010, May). A Cross-State Comparison of Postsecondary CTE Student Graduation Rates and Completions: Determining the Efficacy of Using IPEDS Data for Perkins Reporting,” National Association of Career and Technical Education Information Annual Meeting, St. Louis, Missouri. Kotamraju, P. (2010, May). Building a Useable Inventory Template for Collecting State and Local Information for Meeting the Technical Skills Accountability. National Association of Career and Technical Education Information Annual Meeting, St. Louis, Missouri. Kotamraju, P. (2010, April). A Cross-State Comparison of Postsecondary Student Graduation Rates: A Macro Perspective of Educational Attainment,” Annual Meeting, Council for the Study of Community Colleges, Seattle, Washington. 24 Stakeholder Contacts: Paper Presentations

25. Kotamraju, P. (2009, December). The Minnesota FastTRAC Project: Solving the Career Pathways Puzzle. 15 th Annual Data Quality Institute, Baltimore, Maryland. Kotamraju, P. (2009, December). Building a Technical Skill Inventory Database One State at a Time,” 15 th Annual Data Quality Institute, Baltimore. Kotamraju, P. (2009, May). A Repository Template of Technical Skills Assessment: Collecting, Accessing, Reporting and Using Information and Resources for Meeting the Perkins Technical Skill Attainment Indicator,” National Association of Career and Technical Education Information Albuquerque, New Mexico. Kotamraju, P. (2009, May). Building a Data Dictionary Template from State Administrative Record Data Systems,” Albuquerque, New Mexico. 25 Stakeholder Contacts: Paper Presentations

26. 26 Correspondence should be sent to: Pradeep Kotamraju, NRCCTE, College of Education and Human Development Room 351 University of Louisville, Louisville, KY 40292. Phone: 502-852-7714. Email: pradeep.kotamraju@nrccte.org. pradeep.kotamraju@nrccte.org