1. Evaluating the Impact of MEP Programs and Services
Presented by:
John Wight
Program Manager
Georgia Department of Education
Migrant Education Program
&
Omar Lopez-Nunez
State Research & Development Coordinator
Migrant Education program
Title I Conference
June , 2014
11/22/2018

2. Evaluating the Impact of MEP Programs and Services
Overview:
Evaluation Definition & Background Information
Statewide Service Delivery Plan Evaluation of Services
Current Impact of MEP Program and Services as Measured by Evaluations
11/22/2018

3. Program Purpose
The purpose of the MEP in Georgia (and the United States) is to ensure that migrant children fully benefit from the same free public education provided to all children and that the unmet education-related needs resulting from their migrant lifestyle are met.
7/26/2011

4. Legal Reference for Program Evaluation
Statute:
Part C of Title I of the Elementary and Secondary Education Act (ESEA) of 1965 – Sections 1304 (b) and 1306 (a)
Code of Federal Regulations: 34 CFR
Guidance:
Non-Regulatory Guidance, October 2012, p
11/22/2018

5. Legal Reference for Program Evaluation
Under Title I, Part C, Education of Migratory Children of the Elementary and Secondary Education Act, State educational agencies (SEA) must deliver and evaluate Migrant Education Program (MEP)-funded services to migratory children based on a State plan that reflects the results of a current statewide comprehensive needs assessment.
11/22/2018

6. Georgia MEP Program Evaluation
Section 1304(c)(5) of the Elementary and Secondary Education Act (ESEA) requires states to conduct a program evaluation for the Migrant Education Program. The purpose of conducting an evaluation of the Georgia Migrant Education Program is to examine program effectiveness and results of implemented program activities.
11/22/2018

7. Georgia MEP Program Evaluation
The Georgia MEP will conduct a statewide summary evaluation of all program projects and services based on the guidance and suggestions delineated in the U.S. Department of Education’s MEP Program Evaluation Toolkit at the end of every academic year.
In order to prepare for, conduct, and report on a statewide evaluation plan of the Georgia Title I, Part C Migrant Education Program, the Georgia MEP will:
11/22/2018

8. Georgia MEP Program Evaluation
Conduct a review of current and existing data, data sources and related reports generated by the state, regional, and or local migrant projects;
Disaggregate statewide assessment data and compare the progress of migrant students with non-migrant students; disaggregate statewide assessment data and compare migrant PFS students with non-PFS and non-migrant students;
Review and compare the performance of migrant PFS students and migrant non-PFS students within the national Government Performance and Results Act (GPRA) measures;
Conduct on-site visits at the local project levels for the purpose of conducting implementation plan observations and gathering additional facts and information relative to project plans and implementation fidelity; and
Analyze the information gathered and create a written evaluation incorporating implications and recommendations for overall program improvement, marking the close of the Continuous Improvement Cycle (CIC) for the Georgia MEP.
11/22/2018

9. MEP Evaluation Cycle: An Ongoing Process
Georgia Continuous Improvement Cycle (GCIC):
11/22/2018

10. Georgia MEP Continuous Improvement Cycle (GCIC)
The strategies for delivery of services have been determined by setting a three-step project planning process that every LEA must follow in order to ensure fidelity in the Continuous Improvement Cycle (CIC) of the Georgia MEP. The three steps in the project planning process for LEAs involve:
Submission of a CNA profile that captures the needs of the migrant population at the LEA level;
Submission of implementation plan(s) in order to establish academic support services to be provided with projected measurable outcome(s);
Observations of implementation plans and services to monitor fidelity of implementation (GaDOE MEP and LEA staff); completion of surveys by SSPs to monitor fidelity of implementation; and
Completion of implementation plan evaluation(s) for each implementation plan submitted at the end of project cycle in order to validate actual measurable outcome(s) as projected on original implementation plan(s) submitted.
11/22/2018

11. Georgia MEP Continuous Improvement Cycle (GCIC) - Continued
11/22/2018

12. Program Evaluation Defined
Evaluation is the systematic application of scientific methods to assess the design, implementation, improvement or outcomes of a program (Rossi & Freeman, 1993; Short, Hennessy, & Campbell, 1996).
11/22/2018

13. Evaluation Consideration
Effective evaluation is not an "event" that occurs at the end of a project, but is an ongoing process which helps decision makers better understand the project; how it is impacting participants, partner agencies and the community; and how it is being influenced/impacted by both internal and external factors.
W.K. Kellogg Foundation Evaluation Handbook, p. 3
11/22/2018

14. Why do we conduct evaluations?
Establish model programs and best practices by providing feedback about what worked and what failed.
Tool of good management and quality improvement - gain insight into effective strategies on how to improve performance.
Measures impact the program is making.
11/22/2018

15. Service Delivery Plan (SDP)
2013 Statewide
Service Delivery Plan (SDP)
Continuous Improvement Cycle (CIC) for the Georgia MEP
Service Delivery
Evaluation
2013 Statewide CNA SDP Report

16. Service Delivery Plan (SDP) – Overview
As required under Section 1306 of the reauthorized ESEA, the Georgia Migrant Education Program (Georgia MEP) has developed a statewide Service Delivery Plan (SDP) to be implemented during the school years.
This state plan is a current and comprehensive plan for how the services provided by the Georgia MEP and local educational agencies (LEAs) are to be delivered in order to meet the needs of the migrant children and youth throughout the state.
11/22/2018

17. Service Delivery Plan (SDP) – Purpose (Continued)
Design Georgia programs to help migrant children and youth overcome educational disruption, cultural and language barriers, social isolation, various health-related problems, and other factors that inhibit their ability to do well in school, and to prepare them to make a successful transition to postsecondary education or employment; and
Ensure that migrant children and youth benefit from the state of Georgia and local systemic reforms.
11/22/2018

18. MEP Service Delivery Plan Goals
Goal 1: Migrant students …will improve their writing proficiency …….
Goal 2: Migrant students … will improve their math proficiency ….
Goal 3: The Georgia Migrant Education Program will improve school readiness …
Goal 4: The MEP …will continue to support … OSY/DO at the district level….to foster English language acquisition, health and other relevant supplemental services…
Goal 5: Migrant students…will continue to meet and/or exceed their proficiency in reading ….
Goal 6: Georgia MEP SSP staff …will improve their professional competencies when working with migrant participants for short periods of time …
11/22/2018

19. Performance Targets ESEA Flexibility Waiver
Migrant is not a subgroup
College and Career Ready Performance Index (CCRPI)
State performance targets in elementary, middle and high school
11/22/2018

20. Migrant Students Performance Targets and Results
The SDP will set the state performance targets for migrant students on par with the performance targets set for all students in CRCT Reading, English/Language Arts and Math (elementary and middle school level), EOCT in 9th Grade Literature, American Literature, Mathematics I and Mathematics II (high school level) and graduation rate performance targets in Georgia.
So, as the all-student population makes progress (as established in the performance targets), so will migrant students statewide.
The State MEP and LEA MEPs will analyze the progress of migrant students as compared with non-migrant, and migrant Priority for Service (PFS) students as compared with migrant non-PFS and non-migrant students on local and state formative and summative assessments.
11/22/2018

21. Sample CRCT Assessment Migrant vs. Non-Migrant
3rd Grade
2011
Migrant Meets and Exceeds
Non-Migrant Meets and Exceeds
GAP
CRCT Reading
90.85%
94.43%
3.57%
CRCT ELA
82.62%
89.13%
6.51%
CRCT Math
75.68%
81.21%
5.54%
4th Grade
80.19%
87.92%
7.72%
76.85%
88.03%
11.18%
73.44%
81.10%
7.66%
11/22/2018

22. Migrant Students Performance Targets and Results
State-level academic performance data for migrant students is used by the Georgia MEP to ensure migrant student academic progress follows that of the all students’ performance targets in the state.
These data are also used by the Georgia MEP to develop program policy and to target specific migrant education projects and interventions that will increase the academic achievement and success of migrant children and youth statewide.
11/22/2018

23. Current Impact of MEP Program and Services as Measured by Evaluations from 2012-2013
School Readiness and Preschool
K-12 Enrolled Children
Out of School Youth
11/22/2018

24. Measuring Fidelity
Completed by SSPs and Tutors After Observation
Planning and Delivery
Student Engagement
11/22/2018

25. Measuring Fidelity 2013-2014 Application of Professional Development:
Provided by SSPs and Tutors After Observation
Application of Professional Development:
PDNow!, RESA, school level
Conference or workshop
MEP staff
Identifying the Instructional Strategy
Guided reading, direct instruction, modeling
Manipulatives, visual aids, hands on activities
Teacher guided questioning
11/22/2018

26. Reporting Evaluation Results
The level and scope of content depends on to whom the report is intended (OME, State, LEAs staff, Parent Groups – PAC and CNA)
Be sure internal staff have a chance to carefully review and discuss the report to ensure results and activities are followed.
Translate recommendations to action plans, including who is going to do what about the program and by when.
Include professional development to address MEP staff instructional duties and responsibilities
Create a specific plan to observe MEP funded services in an ongoing manner; making changes to plans based on observations (fidelity of implementation).
11/22/2018

27. Reporting Evaluation Results
Summarize the results in an executive summary; summary should include a an interpretation of migrant student performance on local and State formative and summative assessments (a list of test scores or grades is not helpful without this interpretation);
Be sure to record and store the evaluation plans and results in a location where it can referenced when a similar program evaluation is needed in the future.
11/22/2018

28. Pitfalls to Avoid in Evaluations
There is no "perfect" evaluation design. Don't worry about the plan being perfect. It's far more important to do something, than to wait until every last detail has been tested.
Don't report just the successes. You'll learn a great deal about the program by understanding its failures too!
Don't throw away evaluation results once a report has been generated. Results don't take up much room, and they can provide precious information later when trying to understand changes in the program.
11/22/2018

29. Questions?
11/22/2018

30. Georgia MEP on Twitter @GeorgiaMEP
8/2/2013

31. THANK YOU! Contact Information
John Wight Program Manager Georgia Department of Education Migrant Education Program Omar Lopez-Nunez State Research & Development Coordinator Georgia Department of Education Migrant Education Program
11/22/2018