Regional Policy How are evaluations used in the EU? How to make them more usable? Stockholm, 8 October 2015 Kai Stryczynski, DG Regional and Urban Policy Unit Evaluation and European Semester Cohesion Policy 1

Regional Policy Evaluations in a system of shared management - responsibilities Ex ante – Member States (MS) During the programming period – MS (and Commission) Ex post – Commission with MS 2

3 Focus of the Regional and Cohesion Fund evaluations, programming period

Regional Policy Lessons from : Evaluation Great diversity in practice and quality among 800 evaluations of Regional and Cohesion Fund supported interventions carried out between and 721 evaluations of the ESF up to December 2013 Limited but growing numbers of evaluations dealing with effects and impact Cohesion Policy 4

Regional Policy Types of Use of Evaluation Instrumental (implementing recommendations) Conceptual (helping policy makers to understand) Enlightenment (accumulating knowledge) Process use (organizational changes) Persuasive or Symbolic (accountability or legitimization) Saunders

Regional Policy Maximising the Impact of Evaluation Research uncertain – no one way to maximise impact Use is diverse, unpredictable and can be political But, we can expect maximum influence when it is relevant, timely and of high quality 6

Regional Policy We are becoming better – but still have a quality problem Insufficient attempt to examine logic of interventions. Deficiencies in the design and use of methods. Terms of Reference often poorly specified, Sometimes bias in the Terms of Reference, with indications given in the Terms of Reference of the conclusions to be drawn. A lack of public discussion of findings (Gaffey, Riché, 2014) 7

Regional Policy Objectives in OPs – what do you want to change? Each priority axis shall articulate what it wants to change. What motivates the policy?  Specific Objective Observe whether the change is taking place. How to know if policy succeeds?  Result Indicator In other words: the subject of future evaluations

Regional Policy 9..because too often it was not clear "Lack of clarity in policy documents …on the precise objectives which programmes …were intended to achieve and how their achievement was to be verified" Ex post evaluation , synthesis report, p157 One of the triggers of result orientation Cohesion Policy

Regional Policy 10 Link to policy monitoring Result indicators capture the development of the problem to be fixed – in a region,… …and should be monitored Interventions are set to contribute to solution – need for evaluation to estimate contribution Final step for evaluations – look at contribution and overall picture Note: Values for beneficiaries are also needed! Regional Policy Cohesion Policy

Regional Policy Impact evaluations What is required by Regulation? -Evaluate contribution of OP to objectives of each Priority Axis What does this mean? -Each Specific Objective should be covered -To differentiate across specific objectives the rigour – be demanding for new - important objectives -Theory-based and / or counterfactual evaluations, CBA,… Cohesion Policy 11

Regional Policy Evaluation plans Legal requirement – decided by monitoring committee Recommendation on content by Commission What we hope for from advanced regions: Innovations Counterfactual evaluations Build evaluability into programmes

Regional Policy Evaluation plans (2) think about Usability Built into the design of the evaluation Requires reflection on  Reasons and purposes of evaluation  Potential users  Evaluation objectives  The nature of data and evidence  Audience for the output  Timing of release of evaluation output  Agency undertaking the evaluation Saunders

Regional Policy 14 Steps towards meta-evaluations (?) Legal requirement on COM to summarise evaluations from MS How to do it? Quality assurance? Use for future thematic work – ex post evaluation Cohesion Policy

15 Further EC Support REGIO/EMPL have put in place a Helpdesk: To review evaluation plans To provide targeted methodological support for selected evaluations/Member States who request support To organise trainings for managing authorities Cohesion Policy

Further EU support (2) CRIE: Center for Research on Impact Evaluation -Focus on counterfactual impact evaluation -Capacity building of MS: training and practical advice -Regional Workshops on CIE Initiative -Accompanying pilot projects in different MS - 16

Ex ante evaluations – some question marks Are ex ante evaluations still equally justified given the emphasis of impact evaluations during the programming period? Did ex ante evaluations deliver value for money? Was result orientation understood / explained well enough? Were evaluators critical enough? Were findings used by programmers? Governance (responsibility) optimal ? 17

ex post evaluations Cohesion Policy 18 Ambitious excercise -series of thematic evaluations and synthesis until spring 2016 Aim of ex post evaluations -analyse the extent to which resources were used, effectiveness, efficiency and impacts of funding Expected use -for accountability purposes to the Court of Auditors, European Parliament, etc. -for learning purposes for '14-'20 and beyond.

Open data – transparency Open Data pilot project Data for research ta-for-research/ 19

20 Evaluation Conference 2016 EU Cohesion Policy at work: a results-oriented policy Zagreb, 9-10 June 2016 Cohesion Policy

21 Competition for best evaluations and evaluation proposals Completed evaluations by any individual or company of the effectiveness of EU Cohesion Policy interventions supported by the European Regional Development Fund, the Cohesion Fund or the European Social Fund. Use at least one of 3 methods:  Ex post cost benefit analysis  Theory based evaluation  Counterfactual evaluation Deadline for the entries: 31 December 2015 For regular updates on the competition and the 2016 conference, visit the page: Or contact: Proposal for an impact evaluation of Cohesion Policy Evaluation not started Preferably using data in a new way – data on the DG REGIO or EMPL websites or data from a national or regional authority Open to all individuals and participation of young researchers and PhD students is strongly encouraged