1. Cluster, Multi-Site, and Multi-Level Evaluation Systems and Frameworks
Ashweeta Patnaik Anne Markiewicz

2. Ashweeta Patnaik

3. Integrated Data Systems for Multi-site Evaluations
Benefits, Challenges & Lessons Learned

4. INTRODUCTION WIF grant from DOL
Regional, multi-partner, strategically comprehensive effort
Accelerate certification, employment, and career advancement
Growing Regional Opportunity for the Workforce
funded by the U.S. Department of Labor under the Workforce Innovation Fund Grant Program (WIF),
which supports innovative approaches to the
design and delivery of employment and training services
that generate long-term, cost effective improvements in the performance of the public workforce system
in terms of outcomes for job seekers and employers.
an ambitious regional, multi-partner, strategically comprehensive effort that
builds upon successful and innovative practices
to accelerate certification, employment, and career advancement
in demand occupations
for an array of economically marginal target groups.

5. INTRODUCTION Multi-site
The service area covers five Workforce Investment Boards (WIBs) that
span the entire Texas - Mexico border area from the City of Brownsville in the south to El Paso in the north.

6. INTRODUCTION Differentiated service delivery Cohort B B1 B2 Cohort C
Cohort A A1 A2 A3
Target Population
The target population is partitioned by academic proficiency into 3 cohorts A, B and C .
These 3 cohorts are further partitioned into multiple sub-cohorts.
Service delivery varies by these sub-cohorts.

7. INTRODUCTION Multiple program offerings
Commercial Driver’s License (CDL)
First Responder
Construction Engineering technology
Medical Receptionist
Maintenance & Repair
Medical Office Clerk
HVAC Technician
Clinical Medical Assistant
Emergency Medical technician (EMT)
Multiple program offerings in a number of demand occupations

8. INTRODUCTION Integrated data system - ASPP
real time, common information technology platform
Purpose
facilitate operational alignment
supplement management and evaluation capacity
Administrative System for Program Participation
A real time, common information technology platform designed specifically for GROW
Purposes
facilitate operational alignment across partners
facilitate real time client information exchanges between service delivery partners
supplement management and evaluation capacity.
serve as the unique database for program performance management and evaluation purposes

9. EVALUATION & THE DATA SYSTEM
Lowest wage job 9
Nine
Nine dollars $9
9 an hour
9 per hour
9 hourly
9.0
20 a day
400 weekly
Highest education level 10
10 Yrs 11
11TH GRADE 7 8 9
Highest Grade Completed 12 Grade , No Diploma No GED
Up Until the 8th Grade
Mixed-Method evaluation
Utilized ASPP for evaluation
Providing input to the data system design
Monitoring data collection and entry.
Conducting regular data quality checks.
Mixed-method evaluation
implementation/process, outcomes, net impacts, and cost-effectiveness analyses.

10. CHALLENGES Building local technical capacities
Ensuring accurate and timely data entry
Embedding new data entry practices and data discipline.
Building local technical capacities
Ensuring accurate and timely data entry
Embedding new data entry practices and data discipline in entrenched bureaucracies.

11. benefits Enhanced data quality
Defined data element structures and codes
Consistent data collection & entry
Frequent data quality checks
When working with multiple sites, evaluators often have to rely on existing data systems at individual sites that frequently differ in scope and structure. Evaluators often spend considerable time and effort in reviewing data dictionaries for individual data systems, identifying common data elements, aligning data structures, recoding variables and ensuring data quality. A more effective and efficient option is for the program design to include the construction, implementation, and use of a data system designed specifically for the program.
evaluators have the clean data they need for program evaluation at the end of program implementation.

12. benefits Real-time data collection and entry
Ongoing assessment of program fidelity
Timely formative feedback
Midcourse program corrections
data is collected and entered in real time so it can be quickly accessed by both program administrators and evaluators.

13. benefits Builds local evaluation capacities
Program performance management
track client flow
assess need for support services
monitor service delivery by partners
assess alignment of service delivery
monitor client and program outcomes.
sustain after grant funding ends
Program components are often sustained after grant funding ends - the integrated data system can also be sustained and adapted to the long term goals and needs of the local sites.

15. Anne Markiewicz