end of project evaluation Typical Design: Post-test only of project participants X P Project participants 6 Based on: Jim Rughs presentation, Stockholm, Semptember 2011
baseline end of project evaluation Comparison group post project evaluation Longitudinal Quasi-experimental P 1 X P 2 X P 3 P 4 C 1 C 2 C 3 C 4 Project participants midterm 7 Source: Jim Rughs presentation, Stockholm, Semptember 2011
baseline end of project evaluation Pre+post of project; no comparison P 1 X P 2 Project participants 8 Source: Jim Rughs presentation, Stockholm, Semptember 2011
end of project evaluation Comparison group Post-test only of project and comparison X P C Project participants 9 Source: Jim Rughs presentation, Stockholm, Semptember 2011
baseline end of project evaluation Comparison group Pre+post of project; post-only comparison P 1 X P 2 C Project participants 10 Source: Jim Rughs presentation, Stockholm, Semptember 2011
baseline end of project evaluation Comparison group Quasi-experimental (pre+post, with comparison) P 1 X P 2 C 1 C 2 Project participants 11 Source: Jim Rughs presentation, Stockholm, Semptember 2011
No need to go always for longitudinal design with control group. But there are many stronger designs than one-group, post- test only…
Steps for a better future 1 Clear and explicit definition of programme/operations goals that are attributable to it. Preferrably in terms of well-being of people. Be aware of contribution x attribution difference.
Higher work intensity sustained Parents work intensity increased Companies want to share kindergartens costs with parents Prosperity of Companies Kindergartens used Companies employ parents Parents want to work more Parents want to send children in Kindergartens built Kindergarten is affordable (costs, transport) for parents Increased employment rate Inspired by Jim Rugh. Area of control / attribution Area of decreasing influence / contribution
Steps for a better future 2 Go for a better design than one group post-test only. Evaluation design should be known before the operation starts, incl. draft of evaluation question. This requires monitoring system to be adjusted to evaluation needs and to be able to collect baseline data. From current practices it is obvious that large parts of evaluation budgets are spent on re-doing insufficient or not well planned monitoring – this takes money and time.
Steps for a better future 3 Evaluation should be regarded as one of the main instrument to allow necessary responsible flexibility in OP execution / planning. Monitoring and a system of indicator-based targets answers the question are we doing the things right? Evaluation should periodically check if we are doing the right things – do the originally expressed assumptions still hold? Are the programme goals still relevant? Are the targets/indicators free of perverse incentives? What change is caused by us?
Steps for a better future 4 Actively build the evaluative culture – at the level of external consultants, at the level of programme internal evaluation specialists and at the level of programme management as well.