Dr. John D. Barge, State School Superintendent “Making Education Work for All Georgians” Building an Effective Evaluation of Your Flexible Learning Program (FLP) Presenter: Sue Hawks Foster, FLP Education Program Specialist Georgia Department of Education (404)

Dr. John D. Barge, State School Superintendent “Making Education Work for All Georgians”  Federal Regulation Requires Evaluation of the Title I Programs  FLP Evaluation Requirements-  Federal, State, and District  Building an Effective Program Evaluation  LEA Plan for District FLP Evaluation  Performance Based Observation of FLP  Most Common Findings FY13  Questions and Answers  References Agenda – FLP Evaluation 2

Dr. John D. Barge, State School Superintendent “Making Education Work for All Georgians” FEDERAL REQUIREMENTS FOR EVALUATING TITLE I PROGRAMS 3

Dr. John D. Barge, State School Superintendent “Making Education Work for All Georgians” Federal regulations requires that a school operating a Title I program annually evaluate the implementation of, and results achieved by, the Title I program ESEA Title I program compliance requires written program evaluation procedure and program evaluation summary Federal Regulation for Title I Programs 4

Dr. John D. Barge, State School Superintendent “Making Education Work for All Georgians” The Title I program evaluation is an annual review of Title I activities to determine if they are contributing to the desired outcomes: Improved student achievement Effective family engagement High quality professional development Federal Requirements for Evaluation 5

Dr. John D. Barge, State School Superintendent “Making Education Work for All Georgians” Evaluation Components: Measures efficacy and impact of the Title I FLP program Documents impact of Title I FLP services on student learning Informs school and district planning (cycle of continuous improvement) Is an advocacy tool Is conducted at end of program year and used for the next years planning Federal Requirements for Evaluation 6

Dr. John D. Barge, State School Superintendent “Making Education Work for All Georgians” Description of: Data used Constituents consulted Process used to arrive at evaluation findings – Previous data – Surveys How findings are utilized for planning and improvement Federal Requirements for Evaluation 7

Dr. John D. Barge, State School Superintendent “Making Education Work for All Georgians” Summary of: Program strengths Program weaknesses Subsequent program changes Reallocation of funds Federal Requirements for Evaluation 8

Dr. John D. Barge, State School Superintendent “Making Education Work for All Georgians” What are the differences and how do they relate to each other? Program evaluation – Evaluates Title I program-FLP specifically – Completed after the fact (end of year) – Helps to plan Title I program for next year including FLP Needs Assessment & Program Evaluation 9

Dr. John D. Barge, State School Superintendent “Making Education Work for All Georgians” Needs assessment – Looks at all school data, including Title I Schoolwide data in all subject areas Targeted Assistance data in all subject areas FLP data – Completed before beginning of school year – Assesses priority needs in district/schools and determines where Title I FLP fits in, as well as, other resources Needs Assessment & Program Evaluation 10

Dr. John D. Barge, State School Superintendent “Making Education Work for All Georgians” FLP EVALUATION REQUIREMENTS 11

Dr. John D. Barge, State School Superintendent “Making Education Work for All Georgians” FLP Evaluation Requirements State Evaluation – Participant Feedback Evaluation Question: What is the overall experience of participants with program? Data Source: Student and Parent Surveys – Service Delivery Evaluation Question: Is the LEAs and programs in compliance with laws and regulations? Data Source: Annual monitoring data, Program documentation, Federal reporting, Public reporting, Technical Assistance, etc. 12

Dr. John D. Barge, State School Superintendent “Making Education Work for All Georgians” State Evaluation cont’d State Evaluation – Effectiveness Evaluation Question: Are programs contributing to increased student academic achievement and performance on state education standards? Data Source: Student performance on state tests, Pre-post assessment measures of state standards and academic skills targeting by programs, Performance Flag data, and student growth in schools offering FLP Evaluation results would be shared with stakeholders and the public and used to inform ongoing program improvement 13

Dr. John D. Barge, State School Superintendent “Making Education Work for All Georgians” State Evaluation cont’d LEA Evaluation – LEAs will be required to evaluate outcomes of FLP interventions implemented at schools within their district that have been identified – LEAs will submit their evaluation plan as part of their overall FLP plan for approval by GaDOE Title I Specialists 14

Dr. John D. Barge, State School Superintendent “Making Education Work for All Georgians” State Evaluation cont’d LEA Evaluation The LEA must submit the following components of an evaluation plan with their FLP plan in the CLIP for each Priority, Focus, or Alert school within their district required to implement FLP: – Effectiveness target or overall quantifiable goal of the program – Assessment instrument(s) that will be used to measure this target/goal – Plan for collecting and maintaining data – Data analysis plan 15

Dr. John D. Barge, State School Superintendent “Making Education Work for All Georgians” State Evaluation cont’d What do the LEA evaluation requirements mean? – Focused, purposeful intervention – Tailor evaluation needs of program to match program goals and student needs – Expand on broadband state assessments to quantify benefits of FLP participation – Collect meaningful information for decision making and continuous program improvement 16

Dr. John D. Barge, State School Superintendent “Making Education Work for All Georgians” BUILDING AN EFFECTIVE PROGRAM EVALUATION PLAN 17

Dr. John D. Barge, State School Superintendent “Making Education Work for All Georgians” Building Effective Program Evaluation Determining Purpose of Evaluation – Why should we evaluate our program? – What is our program trying to do? – What information do our funders expect? – What questions do we want to answer? 18

Dr. John D. Barge, State School Superintendent “Making Education Work for All Georgians” Building Effective Program Evaluation Performance Measures – Measures of effort – Measures of effect Strong performance measures should: – Have strong ties to program goals, inputs, and outputs – Be compatible with age and stage of your program – Consider if the data you need are available/accessible – Yield useful information to the program 19

Dr. John D. Barge, State School Superintendent “Making Education Work for All Georgians” Building Effective Program Evaluation Assessing Capacity for Evaluation – Who should be involved in the process? – What resources must be in place? – Who will conduct our evaluation? 20

Dr. John D. Barge, State School Superintendent “Making Education Work for All Georgians” Building Effective Program Evaluation Selecting Evaluation Design – What type of evaluation should we conduct? Formative to understand the changes that have occurred Summative demonstrate they have achieved the standard required – What type of data should we collect? Qualitative Quantitative 21

Dr. John D. Barge, State School Superintendent “Making Education Work for All Georgians” Building Effective Program Evaluation Selecting Evaluation Design (cont.) – What is our time frame for collecting data? – Causal Experimental Quasi-experimental – Non-experimental Descriptive Pre-experimental 22

Dr. John D. Barge, State School Superintendent “Making Education Work for All Georgians” Choosing evaluation instruments – Construct – Audience – Format – Reliability – Validity Building Effective Program Evaluation 23

Dr. John D. Barge, State School Superintendent “Making Education Work for All Georgians” Analyzing data – How will data be analyzed and shared? – Quantify central evaluation question (and supporting questions) Communicating results – How? – Where? – Who? Building Effective Program Evaluation 24

Dr. John D. Barge, State School Superintendent “Making Education Work for All Georgians” LEA PLAN FOR DISTRICT FLEXIBLE LEARNING PROGRAM (FLP) EVALUATION 25

Dr. John D. Barge, State School Superintendent “Making Education Work for All Georgians” List the effectiveness target(s) or overall quantifiable goal(s) of your programs What are the measureable outcomes that your intervention is designed to improve? Must include at least one academic achievement indicator LEA Plan for District FLP Evaluation 26

Dr. John D. Barge, State School Superintendent “Making Education Work for All Georgians” List the assessment instrument(s) that will be used to measure each listed effectiveness target/goal Include a plan for administering assessment instruments and how you will collect and maintain data LEA Plan for District FLP Evaluation 27

Dr. John D. Barge, State School Superintendent “Making Education Work for All Georgians” When will you administer pre- and post- assessments? How will you maintain student assessment information to ensure confidentiality? How will you determine program effectiveness based on program goals and measurement instruments listed above ? LEA Plan for District FLP Evaluation 28

Dr. John D. Barge, State School Superintendent “Making Education Work for All Georgians” FLP Program Evaluation – Includes Academic Data and Narrative FLP Program Summary Chart Parent, Staff and Student Summary Results LEA Plan for District FLP Evaluation 29

Dr. John D. Barge, State School Superintendent “Making Education Work for All Georgians” Are graduation rates improving? Are CRCT scores increasing? Are EOCT pass rates increasing? Is our CCRPI increasing? True Test for FLP 30

Dr. John D. Barge, State School Superintendent “Making Education Work for All Georgians” PERFORMANCE BASED OBSERVATION (PBO) 31

Dr. John D. Barge, State School Superintendent “Making Education Work for All Georgians” Performance Based Observation FY13 PBO in Review – 12 Districts were selected for PBO as a pilot – 16 schools were observed FY14 Review – 26 Districts have been selected for review – 26 School will be observed 32

Dr. John D. Barge, State School Superintendent “Making Education Work for All Georgians” Performance Based Observation Components of Performance Based Observation(PBO ) – Review of Priority, Focus, and Alert School List – Select number of districts to be reviewed based on the ESEA Flexible Waiver – Selection of schools in district to be monitored 33

Dr. John D. Barge, State School Superintendent “Making Education Work for All Georgians” Performance Based Observation Components of PBO cont’d – Notifications Sent to LEAs Call to Title I Director Follow up Letter to LEA – Superintendent, Title I Director and Principal of selected schools with brief outline of expectations 34

Dr. John D. Barge, State School Superintendent “Making Education Work for All Georgians” Performance Based Observation Components of PBO cont’d Agenda Developed by Title I Office Follow up call to LEA by PBO Team Lead before visit – Addresses and list of staff reviewed – Selection of Teachers/Tutors for Interview 35

Dr. John D. Barge, State School Superintendent “Making Education Work for All Georgians” Performance Based Observation Actual Visit to LEA – Interview with LEA Staff Superintendent Title I Director School Principal School FLP Coordinator School FLP Instructors/Tutors (at least 2) Other vital staff as needed 36

Dr. John D. Barge, State School Superintendent “Making Education Work for All Georgians” Performance Based Observation Actual Visit to LEA – Classroom Observation -20 minutes per classroom – Talk with students and staff in classroom – Observe student and FLP staff interactions – Depart Building- Inform building staff we are leaving the building 37

Dr. John D. Barge, State School Superintendent “Making Education Work for All Georgians” Performance Based Observation Follow-up Letter – Review of the PBO findings for the district – Non-Compliance If a LEA is found out of compliance the district will be notified immediately by Program Manger If corrective action is required it will be provided in writing and followed up with the FLP Specialist and/or the Program Director for FLP 38

Dr. John D. Barge, State School Superintendent “Making Education Work for All Georgians” FY13 FLP MOST COMMON FINDINGS 39

Dr. John D. Barge, State School Superintendent “Making Education Work for All Georgians” FLP’s Most Common Findings Models used for FLP –After school programs were used by Middle and High Schools –Elementary and Middle schools extended the day or rotated the scheduled day –High schools used Saturday sessions –Summer Sessions were held when extra funds were available to extend the learning year 40

Dr. John D. Barge, State School Superintendent “Making Education Work for All Georgians” FLP’s Most Common Findings cont’d Technology Usage Increased Increase Professional Learning Increased Student Instruction Time More contact with parents Reported Increase in Test Scores 41

Dr. John D. Barge, State School Superintendent “Making Education Work for All Georgians” FLP’s Most Common Findings cont’d Frustration of operating the program by some districts because it was so new Large amounts of carryover because of FLP funds were not used last year Special Education Departments not connecting to FLP in various districts Lack of staff in some districts to serve students 42

Dr. John D. Barge, State School Superintendent “Making Education Work for All Georgians” FLP’s Most Common Findings in Plans Lack of understanding of FLP purpose Did not provide performance goals for plans Determining the needs assessment was not fully understood to determine subject area Many did not identify all subgroups within the schools Schools did not understand providing data for more than one year in order to provide a trend 43

Dr. John D. Barge, State School Superintendent “Making Education Work for All Georgians” FLP’s Most Common Findings in Plans Special Education Departments were not really aware of what was going on with Students with Disabilities (SWD) in FLP. Student ratio needs to be smaller Individual Learning Plans needed to be developed and followed up Documentation of Multiple Selection Criteria was lacking in some districts Lack of understanding Professional Learning 44

Dr. John D. Barge, State School Superintendent “Making Education Work for All Georgians” QUESTIONS AND ANSWERS 45

Dr. John D. Barge, State School Superintendent “Making Education Work for All Georgians” References Afterschool Evaluation 101: How to Evaluate an Expanded Learning Program, Erin Harris, Harvard Family Research Project, Doing What Works Clearinghouse, USED- Preschool for California's Children Harvard Family Research Project, Georgia Department of Education, FLP Report Programs/Pages/default.aspx 46

Dr. John D. Barge, State School Superintendent “Making Education Work for All Georgians” Contact Information Sue Hawks Foster FLP Education Specialist