1. Is evaluation of public policy in Ireland dying? And if it is should we care? Dr. Richard Boyle Head of Research Institute of Public Administration Dublin Presentation to Irish Evaluation Network 20 November 2009

2. Overview The good times? 1980s and 1990s Where are we now? Problems but green shoots? What should happen?

3. The good times Well developed evaluation system based around EU requirements for evaluation of structural funds Public service modernisation – Strategic Management Initiative and Expenditure Review Initiative (later Value for Money and Policy Review initiative)

4. Where are we now? EU pressure no longer a key driver National Development Plan evaluation: –NDP evaluation unit focused on capital expenditure appraisal –Last evaluation report was 2005 Value for Money and Policy Review –Patchy and slow –Resourcing reviews a big challenge –Little evidence of impact The problem is more of demand than of supply

5. Does the Oireachtas engage with evaluation reports? 2002-2004 round of Value for Money and Policy Reviews: –43 reports produced –Survey evidence from Select Committees about 37 of these reports –Select Committees has received only 2 of these reports, and one noted by the Committee but not discussed (Source McMahon, 2007) Now getting reports to Committees but engagement still an issue

6. What do VFMPR reports recommend? 13 reports produced by Department of Agriculture and Food (one of the leaders) reviewed: –99 conclusions and recommendations –No direct recommendations for programme termination –14 supported the status quo –Remainder largely focus on minor programme alterations or changes in management processes ‘It is quite obvious that these reviews look at areas of efficiency and effectiveness that must be addressed within the programmes themselves rather than a more fundamental process of evaluation that points to the need for a continuation of the scheme or programme or the need to radically redistribute resources between various programmes’ (Source: Smyth, 2007)

7. An Bord Snip Nua ‘…the Group noted a general deficiency of information regarding the public service impacts associated with particular items of expenditure. The management focus across departments generally still seems to be on securing and retaining the maximum volume of expenditure for particular areas, and on accounting for departmental activities in financial terms; details on outputs and actual performance seem secondary.’

8. An Bord Snip Nua ‘The Value for Money and Policy Reviews, in which various expenditure programmes are critically reviewed by the responsible department, have up to now had limited success in redirecting scarce resources away from lower-priority, lower performing areas to areas where they can be put to more productive use (although recent reforms are designed to address these shortcomings)’

9. Green shoots? Philanthropies requirements for evaluation acting as an alternative external driver to the EU? Changes to VFMPR: more focused and targeting areas of significant spending

10. Kinds of evaluation information coming from different anchors Operational issuesImpact issuesContinued relevance issues Programme manager ?  Corporate group in organisation ? ?? Corporate government group ? ? Legislative audit office ?? Legislative bodies  Outside government  Source: Mayne, Divorski and Lemaire, 1999

11. Issues and what should happen now? Evaluation model/approach – mix between developing internal culture and external scrutiny? Political demand – a lost cause? Periodic scrutiny of the evaluation system Structured publication of evaluation results and links with expenditure decisions