1. National Council for Workforce Education (NCWE) Darlene G. Miller, Ed.D., Executive Director Heldrich Center for Workforce Development, Rutgers University Michelle Van Noy, Ph.D., Researcher Jennifer Cleary, Ph.D., Researcher Burning Glass Technologies Kelly R. Bailey, Director of Partnerships and Alliances Aligning Workforce Programs with the Labor Market: Strategies and Tools for TAACCCT Applications

2. What this presentation can do for you Provide a framework for thinking about how to build TAACCCT-funded workforce programs that align with the labor market, as well as the needs of students and other stakeholders. Offer tools and information to aid in building strong programs. Provide tips on how to build an evaluation-friendly workforce program.

3. Overview of SGA and Labor Market Alignment

4. USDOL Priorities Challenge Colleges to Create New Program Models Key Goals: Align programs with the needs of the labor market and students Use sector-based & evidence-based approaches Evaluate program implementation and outcomes Data systems and administration goals Open sharing of curricula Coordinated data collection across programs

5. Evidence-Based Design Three levels of evidence:  Strong  Moderate  Evidence of Promise  Strong Theory Priority program approaches: Sector-based Career Pathways & Stackable Credentials Employers involved in ALL PARTS OF PROGRAM Use Real-time and other LMI (program selection, student advising) On-line learning Accelerated, contextualized remedial education Competency-based & Work-based learning (including entrepreneurship) Share outcomes and LMI with students Credit granting for non-credit work

6. Higher Education Labor Market Program-level Alignment Model Program Selection Recruitment & Enrollment Curriculum, Teaching, & Assessment Support Services Employer Needs Student Needs Data Collection/Validation Incorporate Relationship-building Outcomes Credential Attainment Employment outcomes Employer/ Student Satisfaction

7. Incorporating data collected into the program is a multifaceted, complex process. It is more than program selection! Employer & Student Needs Support & Follow-up Services Work- based Learning Competency Assessment / Self-Paced Learning Instruc- tional strategies Curriculum development Program selection & enrollment Contextual Factors Institutional Factors Program/ Institutional Structure Labor Market Dynamics Contextual Factors Institutional Factors Program/ Institutional Structure Labor Market Dynamics

8. The SGA requires alignment activities occur at many organizational levels System Institution Dept Program Class - Data systems to share student outcomes - Policies for open sharing curricula - Data systems to share student outcomes - Policies for open sharing curricula Policies/systems/facilities to allow: - Credit for non-credit work - Flexible schedule /enrollment - IT/On-line instruction - Accelerated & competency-based learning - Work-based/hands-on learning Policies/systems/facilities to allow: - Credit for non-credit work - Flexible schedule /enrollment - IT/On-line instruction - Accelerated & competency-based learning - Work-based/hands-on learning - Policies to allow for sharing of faculty for contextualized ed. Systems and skilled staff for: -Data collection and analysis -Incorporating data into program selection and all program activities -Measure interim goals -Build relationships with stakeholders Systems and skilled staff for: -Data collection and analysis -Incorporating data into program selection and all program activities -Measure interim goals -Build relationships with stakeholders - Effective pedagogy for contextualized and accelerated learning

9. Strategies for Successful Program Design

10. Alignment Activities: Systemic Change Grant Leads to System Change within the Institution (not a boutique program)  Embedded into the culture of the institution  Difficulty of navigating internal barriers  College level commitment  Administrative Engagement  Cross-functional Leadership Team  For Consortiums  Consortia Leadership Team  Consortium Director

11. Alignment Activities: Relationship Building Relationships to Improve Outcomes and Meet Grant Priorities  DOL Priority on Long Term Unemployed  Relationship with Workforce Investment Boards  Strong Employer Engagement and Commitment  Career pathways and stackable credentials Employer validation and data validation  Employer validation of supply/demand  Role of Industry Associations  Employer commitment to work-based learning

12. Alignment Activities: Relationship Building Relationships to Improve Outcomes and Meet Grant Priorities  For Consortium Grants – Peer Learning Networks  Learning best practices from each other  Sharing curriculum: Don’t reinvent the wheel  Online and face-to-face Platform and compatibility  Data collection and sharing Data systems in synch

13. Alignment Activities: Data Collection and Validation Alignment with Needs of Labor Market and Students  Real Time Jobs Analysis*  Allows understanding of job vacancy and skill needs of local labor market  Utilizing multiple “reliable” sources LMI Data, Local WIB data, state data, industry association data  Gap analysis of competing programs  What are other colleges in your area doing  What about local CBOs or apprenticeship programs  Understand available supply to fill job vacancies

14. Alignment Activities: Data Collection and Validation Alignment with Needs of Labor Market and Students  Convening stakeholder groups to collect information on job vacancy and skill demand  Multiple relationship building activities (WIBs, employers, etc)  Ensures multiple sources of evidence for demand and supply Validation by the right person (HR manager v CEO)  Allows for collection of skills information for curriculum development  Locally recognized industry-based credentials  Aligning enrollment targets with job vacancies and gaps  Employment data over time

15. Alignment Activities: Program Implementation Curriculum Development  Open sharing of curriculum  Curriculum is no long proprietary or owned by the faculty Open Sharing On-line using DOL platform Faculty support in budget for making the curriculum public and accessible  Acceleration strategies  Evidence-based design  Integration and contextualization for low-skilled adult learners

16. Alignment Activities: Program Implementation Curriculum Development  Credit vs Noncredit  Noncredit must be linked to industry-recognized credentials (NIMS, NCCER, etc)  Articulation between noncredit and credit  Acceleration and Credit for Prior Learning  BE CLEAR WITH THE FACULTY UPFRONT REGARDING PLA College policy Accreditation Transparency

17. “Real Time” Jobs Data

18. Evaluation Approaches and Considerations

19. Evaluation Consideration : Outcomes Measurement and Program Design Randomized Controlled Trials  Plan for recruitment and assignment? Regression Discontinuity  Assessment tests with cut-off scores? Common across colleges? Comparison Group with Controls  Drawn from similar programs at college either from prior or current years

20. Evaluation Consideration : Outcomes Measurement & Data Collection Develop a plan to collect key student data  Including Student Name, SSN, DOB Labor market outcomes data  Partnerships with state to link with UI wage data Other important data also need to be collected  Courses, grades, services received, assessments, scores, # attempts, activity in on-line courses, attendance Plan to store data & ensure consistent data entry

21. Evaluation Considerations: Program Implementation Goals Document implementation to inform outcomes analysis  Provide detail on how program was implemented including strengths and weaknesses Uncover opportunities to strengthen implementation  Evaluation can be an opportunity for on-going learning, reflection, and feedback

22. Evaluation Considerations: Program Implementation Components Multiple data collection efforts to provide a full understanding of implementation may include:  Focus groups  Surveys  Interviews Include the perspectives from a variety of stakeholders including:  College staff, students, employers, other partners

23. Evaluation Considerations: Program Implementation Issues How did you choose program design?  Employer involvement? Evidence? Does the delivery method fit student and employer needs? Are the assessments sufficient and well implemented?  Abilities, skills and interests  What ? Who? How implemented? How used? Is the program structure sufficient to ensure data collected on skills, etc. is incorporated in all program components?  Is there a communication or quality control system in place to ensure all parts of the program are coordinated around stakeholder priorities. Are your supportive services directly addressing employer, as well as, student needs? Career guidance for students, support of employer skill priorities Are your partners involved in all aspects of the program  What relationship building activities have you created to recruit, retain, and engage partners?

24. Final Thoughts

25.  Create a dashboard of metrics right up front along with a workplan for all of the partners  Make sure all of your partners can live with the plan and understand their role in accountability  Don’t bite off more than you can chew  Be realistic in your outcomes Just because you think a reviewer wants to see this, don’t put things into the grant that you cannot accomplish  Increased scrutiny by DOL  Congress wants to know that their money was well spent  Importance of a strong evaluation plan  Take audits seriously

26. NCWE Team of Technical Assistance Providers Darlene G. Miller, Ed.D., NCWE Executive Director Nan Poppe, Ed.D., Campus President Emeritus, Portland Community College Rebecca Nickoli, Ed.D., former Vice President of Corporate College (Workforce and Economic Development), Ivy Tech Community College Nick Kremer, Chair, CA Vocational Research and Accountability Committee and former Dean of Career and Technical Education, Cerritos College

27. For More Information… Darlene Miller, executivedirector@ncwe.org 603-714-1918executivedirector@ncwe.org Michelle Van Noy, mvannoy@rutgers.edu 732-932-4100 x6213mvannoy@rutgers.edu Jennifer Cleary, jcleary@ejb.rutgers.edu 732-932-4100 x6215jcleary@ejb.rutgers.edu Kelly R. Bailey, kbailey@burning-glass.com 732-800-2484kbailey@burning-glass.com