Developing an Evaluation Strategy – experience in DFID Nick York Director – Country, Corporate and Global Evaluations, World Bank IEG Former Chief Professional Officer and Head of Evaluation, DFID Presentation to IEG Evaluation Week 19 March 2013

Policy context  First UK government department to have a formal evaluation policy (2009)  Coalition government (May 2010) announced their commitment to rigorous, independent evaluation of a rapidly expanding aid programme  Decided to honour the international commitments on aid (0.7% of GNI) but…  Strong focus on value for money and impact  “Decisions on the basis of evidence not guesswork”  Decision to set up Independent Commission for Aid Impact

Evaluation strategy - part of a wider strategy on value for money, evidence and results  Programme design and appraisal – significant changes in DFID to move to new rigorous business case process including ex ante appraisal of evidence for new programs  Evidence in decision making – major changes to DFID’s research strategy and use of evidence in policy and programme design  Strengthening performance monitoring and data systems in DFID – and also in developing countries  Developing skills and a culture where use of evidence and evaluation is the norm

Key strategic themes in evaluation policy and approach in DFID  Independence – do you tackle this through structures and reporting, or professionalism and integrity of staff  Relevant and useful – recognise the endemic weakness in many evaluation systems on follow up and feedback into decision making  High quality – including rigour on methods and use of multidisciplinary, cutting edge techniques  Partnership – we depend on data systems and engagement from other development partners  Capacity and systems – in DFID and in developing countries

Independent Commission on Aid Impact  Visibly independent and well resourced –reports to Parliament, through international development select committee, not to DFID Ministers. –Senior commissioners with strong leadership skills –Staffing from National Audit Office secondees and private sector –£5m per annum programme  Covers all Overseas Development Aid, including in DFID but also other government departments.  Forward work programme was based on public consultation but also kept flexibility to respond to topical issues from Parliament and media  Forward work programme was based on public consultation but also kept flexibility to respond to topical issues from Parliament and media  Strong follow up arrangements through reevaluation after 1 year

Embedding evaluation in DFID  External scrutiny is only part of the picture – need to build evaluation into the programme cycle and make it a routine part of how we do business  Ambitious expansion in decentralised evaluations is planned – number and quality  Evaluations commissioned by operational teams with professional analytical input and support from centre  Part of this is a much greater emphasis on rigorous methods and impact evaluation with clear counterfactual  Requires a change in culture and an expansion in evaluation skills within operational/policy teams

Leadership, Incentives and Support  Leadership is vital - Ministers and Board clear that high quality evaluation (including at operational level) is a priority within DFID’s approach to value for money  Shift away from central evaluation department to encouraging staff to take ownership of agenda  New evaluation cadre of specialists with peer recognition from accreditation, training and guidance  Use professional groups such as economists, statisticians and health, education, governance, social development specialists etc.  Financial incentives – come from career opportunities, making additional resources available for new studies, and pump priming and offering funds for training and support

Strategic choices  Types of evaluation – process, summative, impact evaluation, post project evaluation  Delivery modes – from within DFID or through international systems and partners  Role of centre versus operational and embedded analysts, how much to manage/deliver internally or contract out  Timing - Design into programmes from the start – to allow impact to be measured rigorously.  Focus on external scrutiny and upward/outward accountability; or providing evidence for decision making and internal lesson learning

Building a culture?  Clear policy framework, leadership and incentives  Take seriously the need for better data and indicators, including baselines that take account of how we will want to assess impact later in the programme  Major shift in timing and approach from mainly ex post approach to prospective, formative and ex post. Build evaluations into programmes from the start and consider and plan for M&E before programs are designed and resources are released.  Build evaluation skills in DFID and in partner organisations, including systems for independent evaluation in country

Lessons and challenges (1)  Shift to operational ownership of evaluations has major benefits on relevance, design, scope to measure impact and potential for lesson learning by decision makers  More decentralised approach places big demands on skills and requires thought on quality standards and continuous professional development  Creates risks around strategic focus and requires central mechanisms (led by policy teams) for lesson learning across the whole organisation (as opposed to within programs)

Lessons and challenges (2)  Major expansion in impact evaluations (using a range of methods) and use of systematic reviews. This creates more ‘global knowledge’ output from evaluation.  Decentralisation of M&E expertise allows much closer working with client countries, including for evaluation capacity development and joint, country-led evaluation  Independence of evaluation and accountability - now mainly driven by ICAI (super independent) and within DFID the focus is on quality, credibility and integrity of evaluations – with more potential for lesson learning