National Council for Workforce Education (NCWE) Darlene G. Miller, Ed.D., Executive Director Heldrich Center for Workforce Development, Rutgers University Michelle Van Noy, Ph.D., Researcher Jennifer Cleary, Ph.D., Researcher Burning Glass Technologies Kelly R. Bailey, Director of Partnerships and Alliances Aligning Workforce Programs with the Labor Market: Strategies and Tools for TAACCCT Applications

What this presentation can do for you Provide a framework for thinking about how to build TAACCCT-funded workforce programs that align with the labor market, as well as the needs of students and other stakeholders. Offer tools and information to aid in building strong programs. Provide tips on how to build an evaluation-friendly workforce program.

Overview of SGA and Labor Market Alignment

USDOL Priorities Challenge Colleges to Create New Program Models Key Goals: Align programs with the needs of the labor market and students Use sector-based & evidence-based approaches Evaluate program implementation and outcomes Data systems and administration goals Open sharing of curricula Coordinated data collection across programs

Evidence-Based Design Three levels of evidence:  Strong  Moderate  Evidence of Promise  Strong Theory Priority program approaches: Sector-based Career Pathways & Stackable Credentials Employers involved in ALL PARTS OF PROGRAM Use Real-time and other LMI (program selection, student advising) On-line learning Accelerated, contextualized remedial education Competency-based & Work-based learning (including entrepreneurship) Share outcomes and LMI with students Credit granting for non-credit work

Higher Education Labor Market Program-level Alignment Model Program Selection Recruitment & Enrollment Curriculum, Teaching, & Assessment Support Services Employer Needs Student Needs Data Collection/Validation Incorporate Relationship-building Outcomes Credential Attainment Employment outcomes Employer/ Student Satisfaction

Incorporating data collected into the program is a multifaceted, complex process. It is more than program selection! Employer & Student Needs Support & Follow-up Services Work- based Learning Competency Assessment / Self-Paced Learning Instruc- tional strategies Curriculum development Program selection & enrollment Contextual Factors Institutional Factors Program/ Institutional Structure Labor Market Dynamics Contextual Factors Institutional Factors Program/ Institutional Structure Labor Market Dynamics

The SGA requires alignment activities occur at many organizational levels System Institution Dept Program Class - Data systems to share student outcomes - Policies for open sharing curricula - Data systems to share student outcomes - Policies for open sharing curricula Policies/systems/facilities to allow: - Credit for non-credit work - Flexible schedule /enrollment - IT/On-line instruction - Accelerated & competency-based learning - Work-based/hands-on learning Policies/systems/facilities to allow: - Credit for non-credit work - Flexible schedule /enrollment - IT/On-line instruction - Accelerated & competency-based learning - Work-based/hands-on learning - Policies to allow for sharing of faculty for contextualized ed. Systems and skilled staff for: -Data collection and analysis -Incorporating data into program selection and all program activities -Measure interim goals -Build relationships with stakeholders Systems and skilled staff for: -Data collection and analysis -Incorporating data into program selection and all program activities -Measure interim goals -Build relationships with stakeholders - Effective pedagogy for contextualized and accelerated learning

Strategies for Successful Program Design

Alignment Activities: Systemic Change Grant Leads to System Change within the Institution (not a boutique program)  Embedded into the culture of the institution  Difficulty of navigating internal barriers  College level commitment  Administrative Engagement  Cross-functional Leadership Team  For Consortiums  Consortia Leadership Team  Consortium Director

Alignment Activities: Relationship Building Relationships to Improve Outcomes and Meet Grant Priorities  DOL Priority on Long Term Unemployed  Relationship with Workforce Investment Boards  Strong Employer Engagement and Commitment  Career pathways and stackable credentials Employer validation and data validation  Employer validation of supply/demand  Role of Industry Associations  Employer commitment to work-based learning

Alignment Activities: Relationship Building Relationships to Improve Outcomes and Meet Grant Priorities  For Consortium Grants – Peer Learning Networks  Learning best practices from each other  Sharing curriculum: Don’t reinvent the wheel  Online and face-to-face Platform and compatibility  Data collection and sharing Data systems in synch

Alignment Activities: Data Collection and Validation Alignment with Needs of Labor Market and Students  Real Time Jobs Analysis*  Allows understanding of job vacancy and skill needs of local labor market  Utilizing multiple “reliable” sources LMI Data, Local WIB data, state data, industry association data  Gap analysis of competing programs  What are other colleges in your area doing  What about local CBOs or apprenticeship programs  Understand available supply to fill job vacancies

Alignment Activities: Data Collection and Validation Alignment with Needs of Labor Market and Students  Convening stakeholder groups to collect information on job vacancy and skill demand  Multiple relationship building activities (WIBs, employers, etc)  Ensures multiple sources of evidence for demand and supply Validation by the right person (HR manager v CEO)  Allows for collection of skills information for curriculum development  Locally recognized industry-based credentials  Aligning enrollment targets with job vacancies and gaps  Employment data over time

Alignment Activities: Program Implementation Curriculum Development  Open sharing of curriculum  Curriculum is no long proprietary or owned by the faculty Open Sharing On-line using DOL platform Faculty support in budget for making the curriculum public and accessible  Acceleration strategies  Evidence-based design  Integration and contextualization for low-skilled adult learners

Alignment Activities: Program Implementation Curriculum Development  Credit vs Noncredit  Noncredit must be linked to industry-recognized credentials (NIMS, NCCER, etc)  Articulation between noncredit and credit  Acceleration and Credit for Prior Learning  BE CLEAR WITH THE FACULTY UPFRONT REGARDING PLA College policy Accreditation Transparency

“Real Time” Jobs Data

Evaluation Approaches and Considerations

Evaluation Consideration : Outcomes Measurement and Program Design Randomized Controlled Trials  Plan for recruitment and assignment? Regression Discontinuity  Assessment tests with cut-off scores? Common across colleges? Comparison Group with Controls  Drawn from similar programs at college either from prior or current years

Evaluation Consideration : Outcomes Measurement & Data Collection Develop a plan to collect key student data  Including Student Name, SSN, DOB Labor market outcomes data  Partnerships with state to link with UI wage data Other important data also need to be collected  Courses, grades, services received, assessments, scores, # attempts, activity in on-line courses, attendance Plan to store data & ensure consistent data entry

Evaluation Considerations: Program Implementation Goals Document implementation to inform outcomes analysis  Provide detail on how program was implemented including strengths and weaknesses Uncover opportunities to strengthen implementation  Evaluation can be an opportunity for on-going learning, reflection, and feedback

Evaluation Considerations: Program Implementation Components Multiple data collection efforts to provide a full understanding of implementation may include:  Focus groups  Surveys  Interviews Include the perspectives from a variety of stakeholders including:  College staff, students, employers, other partners

Evaluation Considerations: Program Implementation Issues How did you choose program design?  Employer involvement? Evidence? Does the delivery method fit student and employer needs? Are the assessments sufficient and well implemented?  Abilities, skills and interests  What ? Who? How implemented? How used? Is the program structure sufficient to ensure data collected on skills, etc. is incorporated in all program components?  Is there a communication or quality control system in place to ensure all parts of the program are coordinated around stakeholder priorities. Are your supportive services directly addressing employer, as well as, student needs? Career guidance for students, support of employer skill priorities Are your partners involved in all aspects of the program  What relationship building activities have you created to recruit, retain, and engage partners?

Final Thoughts

 Create a dashboard of metrics right up front along with a workplan for all of the partners  Make sure all of your partners can live with the plan and understand their role in accountability  Don’t bite off more than you can chew  Be realistic in your outcomes Just because you think a reviewer wants to see this, don’t put things into the grant that you cannot accomplish  Increased scrutiny by DOL  Congress wants to know that their money was well spent  Importance of a strong evaluation plan  Take audits seriously

NCWE Team of Technical Assistance Providers Darlene G. Miller, Ed.D., NCWE Executive Director Nan Poppe, Ed.D., Campus President Emeritus, Portland Community College Rebecca Nickoli, Ed.D., former Vice President of Corporate College (Workforce and Economic Development), Ivy Tech Community College Nick Kremer, Chair, CA Vocational Research and Accountability Committee and former Dean of Career and Technical Education, Cerritos College

For More Information… Darlene Miller, Michelle Van Noy, Jennifer Cleary, Kelly R. Bailey,