2. Regional Policy Meta-Evaluation Some Reflections Evaluation Conference, Warsaw 12/13 November 2012 Veronica Gaffey Head of Evaluation DG for Regional and Urban Policy European Commission 2 Regional Policy

3. 3 Presentation 1.Meta-Evaluation – just how ambitious do we want to be? 2.A more modest approach focused on the accumulation of knowledge 3.Appropriate level – EU/MS or both? 4.The role of meta-analysis in ex ante evaluation 5.Elements in the new regulation to support accumulation of knowledge, moving towards meta-analysis Regional Policy

4. 4 1. Meta Evaluation What is it?  Aim to improve decision making through systematic reviews on the effects of intervention Examples:  Cochrane Collaboration – Health – since 1992  Campbell Collaboration – education, crime and justice, social welfare – since 1999 Is our evidence base sufficiently strong? Issues of internal & external validity Regional Policy

5. 5 Our evidence base Of 266 completed evaluations examined by Expert Network in 2011, only 34% concerned effects; 66% implementation issues Pilot review process for DG REGIO finds few evaluations of Cohesion Policy of really good quality Difficult to judge quality of evaluations without terms of reference, budget and knowledge of context Regional Policy

6. 6 2.Accumulation of Knowledge Some experience here:  Meta-analysis of 200 evaluations across Europe prepared for Polish Presidency Conference on Evidence based Cohesion Policy  Forthcoming Regional Focus on What we have learned from counterfactual evaluations of innovation and enterprise support (Daniel Mouqué)  Internal Analysis on Transport and Environmental Infrastructure – likely to be transformed into guidance Insights rather than proofs; questions rather than prescriptions Regional Policy

7. 7 3. Appropriate Level for Accumulating Knowledge Experience shows it can be done at EU level, if  Terms of Reference and budget are public  Context and method are explicit  "Why it works?" is addressed as well as "Does it work?" Language issue can be overcome DG REGIO starting to build a library of MS evaluations Also at Member State level, though  Difficult to generate critical mass Regional Policy

8. 8 4. Meta-Analysis in Ex Ante Evaluation Ex ante evaluation should be theory and evidence based Testing the logic of intervention is at its heart – evidence from literature and other evaluations should be used Unpick the triangle in the next slide…. Regional Policy

9. 9

10. 10 5. Elements in the New Regulation Clearer articulation of objectives with associated result indicators and baselines – success can be judged Requirement for an evaluation plan Requirement for an impact evaluation of each priority during the programming period Summary report in 2021 – a meta- evaluation? Regional Policy

11. 11 Possible Next Steps… We continue to encourage more impact evaluations – using different and combined methods Competition/Prize for good Terms of Reference? Structured exchanges of experience – based on themes Building communities of practice – involving public administrations & academic experts Regional Policy

12. Thank you for your attention 12 Regional Policy