Cluster, Multi-Site, and Multi-Level Evaluation Systems and Frameworks Ashweeta Patnaik Anne Markiewicz

Ashweeta Patnaik

Integrated Data Systems for Multi-site Evaluations Benefits, Challenges & Lessons Learned

INTRODUCTION WIF grant from DOL Regional, multi-partner, strategically comprehensive effort Accelerate certification, employment, and career advancement Growing Regional Opportunity for the Workforce funded by the U.S. Department of Labor under the Workforce Innovation Fund Grant Program (WIF), which supports innovative approaches to the design and delivery of employment and training services that generate long-term, cost effective improvements in the performance of the public workforce system in terms of outcomes for job seekers and employers. an ambitious regional, multi-partner, strategically comprehensive effort that builds upon successful and innovative practices to accelerate certification, employment, and career advancement in demand occupations for an array of economically marginal target groups.

INTRODUCTION Multi-site The service area covers five Workforce Investment Boards (WIBs) that span the entire Texas - Mexico border area from the City of Brownsville in the south to El Paso in the north.

INTRODUCTION Differentiated service delivery Cohort B B1 B2 Cohort C Cohort A A1 A2 A3 Target Population The target population is partitioned by academic proficiency into 3 cohorts A, B and C . These 3 cohorts are further partitioned into multiple sub-cohorts. Service delivery varies by these sub-cohorts.

INTRODUCTION Multiple program offerings Commercial Driver’s License (CDL) First Responder Construction Engineering technology Medical Receptionist Maintenance & Repair Medical Office Clerk HVAC Technician Clinical Medical Assistant Emergency Medical technician (EMT) Multiple program offerings in a number of demand occupations

INTRODUCTION Integrated data system - ASPP real time, common information technology platform Purpose facilitate operational alignment supplement management and evaluation capacity Administrative System for Program Participation A real time, common information technology platform designed specifically for GROW Purposes facilitate operational alignment across partners facilitate real time client information exchanges between service delivery partners supplement management and evaluation capacity. serve as the unique database for program performance management and evaluation purposes

EVALUATION & THE DATA SYSTEM Lowest wage job 9 Nine Nine dollars $9 9 an hour 9 per hour 9 hourly 9.0 20 a day 400 weekly Highest education level 10 10 Yrs 11 11TH GRADE 7 8 9 Highest Grade Completed 12 Grade , No Diploma No GED Up Until the 8th Grade Mixed-Method evaluation Utilized ASPP for evaluation Providing input to the data system design Monitoring data collection and entry. Conducting regular data quality checks. Mixed-method evaluation implementation/process, outcomes, net impacts, and cost-effectiveness analyses.

CHALLENGES Building local technical capacities Ensuring accurate and timely data entry Embedding new data entry practices and data discipline. Building local technical capacities Ensuring accurate and timely data entry Embedding new data entry practices and data discipline in entrenched bureaucracies.

benefits Enhanced data quality Defined data element structures and codes Consistent data collection & entry Frequent data quality checks When working with multiple sites, evaluators often have to rely on existing data systems at individual sites that frequently differ in scope and structure. Evaluators often spend considerable time and effort in reviewing data dictionaries for individual data systems, identifying common data elements, aligning data structures, recoding variables and ensuring data quality. A more effective and efficient option is for the program design to include the construction, implementation, and use of a data system designed specifically for the program. evaluators have the clean data they need for program evaluation at the end of program implementation.

benefits Real-time data collection and entry Ongoing assessment of program fidelity Timely formative feedback Midcourse program corrections data is collected and entered in real time so it can be quickly accessed by both program administrators and evaluators.

benefits Builds local evaluation capacities Program performance management track client flow assess need for support services monitor service delivery by partners assess alignment of service delivery monitor client and program outcomes. sustain after grant funding ends Program components are often sustained after grant funding ends - the integrated data system can also be sustained and adapted to the long term goals and needs of the local sites.

Ashweeta.Patnaik@RayMarshallCenter.Org

Anne Markiewicz