Designing Influential Evaluations Session 3 Evaluation Design Uganda Evaluation Week - Pre-Conference Workshop 19 th and 20 th May 2014

Training Objectives By the end of the session, participants will: Understand the use of programme logic models and theory of change Understand how to develop evaluation questions Lewis Caroll, author of ‘Alice in Wonderland’ 2

LOGFRAME AND THEORY OF CHANGE Logic models 3

Simple results chain Physical & financial resources Processes which turn Inputs into Outputs Operational changes (new skills or abilities, the availability of new products and services). Behavioural or institutional/ changes in response to the outputs Progress towards thematic priorities Results External perspective Internal perspective Management control diminishes External factors become more important 4

A logframe is an ‘abstract’ of a project InputsCosts The resources required How much the project will cost What the project will do & deliver Outputs Activities Goal(s) Purpose Objectives What the project seeks to achieve Factors that may affect progress & success Pre- conditions Assumptions How progress & success will be measured What will be How will it be measured? IndicatorsVerification 5

The missing middle Activities Outputs Purpose Goal 6

Response to outputs – the ‘miracle’! Goal Purpose Outputs ActivitiesInputsCosts ObjectivesIndicatorsVerificationAssumptions Response to project services Behaviour or system change! Critical conditions for success 7

Discussion Turn to your neighbour or work in a small group Discuss for 5 minutes: ◦ What are the common weaknesses in development and use of logframes? Summarise 2-3 points Brief plenary discussion on common themes 8

Analysis of DFID logframes Drew & Albone, 2008; Agulhas, 2007 Problems: Only 18% of performance indicators measured outcomes 64% measured ‘inputs’ and ‘processes’ Only 8% disaggregated data by gender Indicators not specific and time bound Measurement and the evidence base not well planned Assumptions/risks not monitorable Clear targets support better performance ◦ Only 24% of projects with a partially clear Purpose scored 1 or 2 at OPR ◦ 73% of projects with unclear Purpose scored <3 Statement Fully clear Partially clear Unclear Goal79%14%6% Purpose54%35%10% Output64%29%6% 9

10

What is ‘Theory of change’? A theory of change is a set of statements about how change happens, and/or an attempt to describe underlying logic and assumptions 11

Key elements of a theory of change But no agreed format or convention 12

Funds to Min of Ed. School needs assessment Curriculum development Buildings Attendance Teachers Decision to fund Select teachers Mobilise PTA for building Teachers attend training Build/ rehabilitate schools Teachers return to schools Teachers apply new skills Improved facilities supplemented by materials & curriculum Parents respond to improvements and permit boys & girls to enrol Boys & girls respond to improved quality Increase in teaching quality Theory of change for a primary education project Enrolment improves Attendance improves Retention improves Boys & girls literacy rates increase Assume teachers can travel to training Risk of teachers moving schools Assume training effective Assume Min of Ed funds books & materials Assume children can be released from other tasks Assume building is of satisfactory quality Assume selection on merit Risk of children leaving for employment or family duties Assume no barriers to children enrolling Other contextual factors such as roads & transport Start End 13

Discussion exercise Working in small groups, consider how a Theory of Change approach might affect the way you plan your evaluation? Prepare your ideas for presentation and discussion in plenary. 14

EVALUATION QUESTIONS Theory of change & evaluation criteria 15

Formulating questions – from simple logframe for education Funding & staffing School building, teacher training & curriculum development Improved access to & quality of schooling Increased net enrolment Increased literacy What was the distribution of funding? How many staff were appointed? 2.How many schools received building projects? How many teachers were trained? How was the curriculum developed? 3.How much did access change? What aspects of quality were improved? 4.Did a change in net enrolment take place? 5.Did literacy rates improve? 16

Funds to Min of Ed. School needs assessment Curriculum development Buildings Attendance Teachers Decision to fund Select teachers Mobilise PTA for building Teachers attend training Build/ rehabilitate schools Teachers return to schools Teachers apply new skills Improved facilities supplemented by materials & curriculum Parents respond to improvements and permit boys & girls to enrol Boys & girls respond to improved quality Increase in teaching quality Theory of change for a primary education project Enrolment improves Attendance improves Retention improves Boys & girls literacy rates increase Assume teachers can travel to training Risk of teachers moving schools Assume training effective Assume Min of Ed funds books & materials Assume children can be released from other tasks Assume building is of satisfactory quality Assume selection on merit Risk of children leaving for employment or family duties Assume no barriers to children enrolling Other contextual factors such as roads & transport Start End 17

If … then … - the intervention logic If the school needs assessment is effective Then the schools with the greatest need and potential will be chosen If teachers are selected on merit Then those with potential will be put forward for training If teachers are able to travel Then selected teachers will attend training If teachers gain new skills And do not leave for jobs in other locations And facilities are improved by the building programme And improved facilities are complemented by teaching materials and the new curriculum Then teachers will apply new skills If teachers will apply new skills Then quality of teaching will improve If parents respond to improvements And children can be released from other tasks And there are no socio-cultural barriers to boys and girls enrolling Then enrolment of boys and girls will increase If boys and girls respond to improved quality Then attendance will increase 18

Hypotheses about education improvement 19

Formulating questions Funding & staffing School building, teacher training & curriculum development Improved access to & quality of schooling Increased net enrolment Increased literacy How were schools selected for the project; what was known about their enrolment and literacy status in advance? 2.Who was selected for teacher training; how effective was it; how many teachers were retained afterwards? 3.How much did access change and for who; what aspects of quality were improved and against what standard? 4.Did a change in net enrolment take place? Was it the result of the outputs delivered under this project or did other factors such as improved roads and transport, or demographic changes affect the outcome? 5.Did literacy rates improve? For both boys and girls? How widespread or consistent have these changes been. Do they reflect specific locations or a general trend? 20

OECD-DAC Evaluation criteria 21

Relate to evaluation criteria Funding & staffing School building, teacher training & curriculum development Improved access to & quality of schooling Increased net enrolment Increased literacy Economy Efficiency Effectiveness Impact 22

Developing questions 23

Discussion exercise Working in small groups, consider at what stage evaluation questions should be developed and whose responsibility that should be? Prepare your ideas for presentation and discussion in plenary. 24

Summary Analyse the intervention logic using a logic model such as the logframe or (better) theory of change Develop evaluation questions from the details of the intervention logic These provide the basis for considering an approach and methods An evaluation framework can help link questions to approach and methods

END