Learning about learning The GEC approach to M&E UKFIET Conference Joseph Holden & Jason Calvert 15 th September 2015 © PEAS

Structure 2 | UKFIET 2015 Conference Challenges with the approach -In school vs. out-of-school approaches -Enumerators and testing -PbR + Accountability vs. flexibility -Cost and capacity 1 2 The GEC approach to M&E -Background -Focus on outcomes and PbR -Quantitative benchmark for achievement -Mixed methods for outputs -The evaluator model

The GEC, 37 projects working in 18 countries 3 | UKFIET 2015 Conference

The GEC has three major outcomes… 4 | UKFIET 2015 Conference ALL ATTENDANCE Girls that stay and attend school throughout life cycle of the project LEARNING Girls supported by GEC with improved learning outcomes LEVERAGE Additional funds secured & sustainability mechanisms established PbR Indicators: Attendance Net enrolment PbR Indicators: Literacy Numeracy Non-PbR Indicators: Match funding Influence on policy and community perceptions

A rigorous approach, means control groups… 5 | UKFIET 2015 Conference Time Learning Outcomes Control Group Intervention Group Counterfactual Intervention Group GEC Intervention Commences Control Group Trajectory Intervention Group Trajectory Outcomes are defined as ‘additional’ learning

With a consistent approach to target setting… 6 | UKFIET 2015 Conference y 75 th percentile μ Frequency Score y μ 75 th percentile Frequency Score Distribution of test scores for girls in one grade above in baseline Expected distribution of girls in the intervention group at midline The target is the expected improvement on the learning test over and above control group y μ = Mean z Frequency Score σ σ Y = 0.2 SD

Evidence supports the focus on learning... 7 | UKFIET 2015 Conference EGRA words per minute (wpm) for Grade 5 girls. Selected GEC projects + estimate from United States, compared to estimate of speed required for comprehension

8 | UKFIET 2015 Conference But also a mixed methods approach… IMPACTLONG TERM STUDY OUTCOMES OUTCOME SPREADSHEET & MID / ENDLINE OUTPUTLOGFRAME INPUTS MILESTONES & VfM TABLES Improved Employment, Reduced Child Marriage etc. Learning Improved teaching skills Teacher training Improved access to materials Distribution of textbooks Attendance Access to appropriate finance Bursaries Safe mode of transport Van / Bus Economy Efficiency Effectiveness

9 | UKFIET 2015 Conference The independent, External Evaluator: –Helps to design and finalise tools –Collects data on learning indicator and information on attendance –Conducts HHS –Conducts qualitative research including output indicators if requested –Produces data and draft report The project: –Contracts the evaluation –Helps design and finalise tools –Feeds in theory of change to evaluation –Conducts own qualitative research including informing logframe indicators, as well as own attendance spot checks –Quality controls the outputs from the evaluator And projects must be independently evaluated

Part two: Challenges 10 | UKFIET 2015 Conference 1 2 Challenges with the approach -In school vs. out-of-school approaches -Enumerators and testing -PbR + Accountability vs. flexibility -Cost and capacity The GEC approach to M&E -Background -Focus on outcomes and PbR -Quantitative benchmark for achievement -Mixed methods for outputs -The evaluator model

11 | UKFIET 2015 Conference GEC projects employ a longitudinal study, tracking a cohort of girls across 3 years, representative of their beneficiary population. Finding the same girls is difficult, if OOS… Projects have done so using: –The cohort via a household survey –The cohort via schools There are large statistical benefits to tracking a cohort – reduces standard errors + allows for other factors to be used in regressions BASELINEMIDLINEENDLINE

12 | UKFIET 2015 Conference Tests are not perfect, particularly at higher levels.. Challenges have arisen over the course of the baseline. These include: Complexity of enumeration and data collection: –More training of enumerators at baseline would have been beneficial. –EGRA – the use and reporting of the timed dimensions of the test. –EGMA – the need to weight sections equally. –ASER / UWEZO – scoring tests clearly given a binary, level-based approach to assessments. The findings of ceiling or floor effects –Adaptations have been needed and re-testing where “ceiling” or “floor” effects have been found.

PbR challenges: accountability vs flexibility… 13 | UKFIET 2015 Conference

14 | UKFIET 2015 Conference To ensure complete and good quality data, and that enumerators and data processors (e.g. data entry clerks and translators) are collecting and processing data in the correct way, fieldwork monitoring and data quality assurance should always be divided into three key components: 1.Pre-fieldwork processes 2.Fieldwork 3.Post-fieldwork processes Investing in all three processes may save time and resources in the future. MANY GEC PROJECT EVALUATORS HAVE LACKED CAPACITY AT THESE STAGES. Cost and capacity constraints for fieldwork…

15 | UKFIET 2015 Conference Respondent Fatigues Attendance data -Accuracy -Spot checks Organisational Capacity -Enumeration and -data issues PbR related Risks - Stat significance Ethical Concerns - Control groups Fragile States -Security -Access Balance between evaluation and implementation So, a host of challenges, this is not easy….

16 | UKFIET 2015 Conference 80,000 The amount of girls in the full GEC cohort (approx.) Outcomes model VfM methodology used across other DFID projects Quantitative and qualitative data contributing towards detailed understanding on the context and education barriers in developing countries But also some achievements!