1. COP MEETING May 19, 2020 Bruxelles * General Director, Ministry of Economy and Finance, Italy. Special Adviser at the EU Commission METHODOLOGICAL ISSUES ON THE WAY TO A RESULT-ORIENTED COHESION POLICY by Fabrizio Barca

2. 2 THE MOUNTING PRESSURE FOR COHESION POLICY REFORM AND THE ISSUE OF “TECHNICAL FEASIBILITY”  The pressure for a “sharp turn” in the governance of cohesion policy is mounting :  The option of a high-level political pact on reforming cohesion policy is stronger.  The issue then becomes not “whether” but “how” reform: What are the “technical feasibility” and the effectiveness of the specific proposals of reform?  the EU 2020 strategy is calling for a credible “delivery channel”,  social pressure is growing (with or without the crisis),  the “Emotional Sunday “ (May 2nd 2010) effect is denting into all actors and is calling for stronger guarantees and conditionalities.

3. 3 THE CORE OF THE REFORM: STRENGHTENED CONDITIONALITIES AND A NEW METRIC OF RESULTS  the establishment of targets and outcome,  the institutional innovations needed in order to pursue those targets.  Conditionalities must be focused on:  the construction and the use of a system of outcome indicators and targets,  the assessment and evaluation of impacts.  The new metric of results must be focused on :

4. 4 “ORIENTING POLICY TO RESULTS”: A CLARIFICATION  “Orienting policy to results” includes four different functions: This is what cohesion policy already does   Learning about what policy actually is  evaluation of the implementation process is needed, since “policy” is not adequately identified by its normative description, but it is shaped through implementation  Ensuring that policy-makers comply with “results” which are (almost) fully within their means, i.e. procedural, financial and (partly) output targets  automatic incentives and sanctions can be used (through complete or incomplete contracts) in order to improve policy-makers’ effort This is the core of the new metric   Promoting policy-makers’ focus on final targets in terms of people’s well-being, i.e. “results” which (unlike procedural or financial outputs) are only partly within policy- makers’ means,  outcome indicators and targets linked to contexts and interventions must be used,  automatic incentives and sanctions must be employed with great care, since we have very limited understanding of the causal link between policy and outcomes and the failure to measure some dimensions can produce a serious bias  Learning about what (which policy) works  Impact assessment and evaluation is needed (not indicators!) to estimate causality between interventions and outcomes

5. 5 TODAY’S PROBLEMS AND HOW TO MOVE FORWARD  In the current design of cohesion policy and in the debate on simplification these four VERY DIFFERENT functions are confused  Once they are properly distinguished: 1. The focus on final objectives can be strongly empowered by:  using the extraordinary know-how on indicators and targets accumulated all over the world and in other EU policy fields (  Open Method of Coordination)  reducing policy-makers’ incentive to “cheat” on targets 2. A major investment can be made by the Commission and Member States on impact assessment and on prospective counterfactual impact evaluation by:  cashing in on very relevant methodological advances made in the last twenty years  using it as a strong disciplinary device

6. 6 1. ON OUTCOME INDICATORS AND TARGETS (I)  Following well-established protocols, a system of outcome indicators will be built which responds to the following requisites :  direct relevance for people’s well-being,  statistical validation and clarity of interpretation,  responsiveness to policy interventions,  distinction between objective and subjective types.  The system will consist of three layers: A.A few (15-20) core indicators for the whole EU, agreed at EU wide level; B. National and regional indicators, selected by MS and Regions for each of their priorities; C.Project or community-based indicators, selected by the private-public partnership running the projects.  Targets will be established for categories B and C and will be included in the NSRF and OPs.  Information on indicators and targets will be presented and discussed :  in Annual Strategic Reports, for categories A and B,  at revitalized Monitoring Committees, for all categories, while modern systems of communication will be employed to ensure public debate.

7. 7 1. ON OUTCOME INDICATORS AND TARGHETS (II)  Progress (or lack of progress) towards targets will be the main focus of Annual Reports to be debated in a structured way in the Council and in the European Parliament.  Failure to more towards targets will not lead to any sanction, but failure to provide convincing explanations for lack of progress will lead to ad-hoc evaluation analysis and to recommendations by the Commission; and failure to respond to recommendations will lead to sanctions.  Room will be allowed for motivated revision of indicators and targets in response to unforeseen events and learning.  The Commission will promote the system of indicators and targets by:  creating a very accessible data base,  providing technical assistance to MS and Regions,  launching methodological research on the subject,  testing new methodologies in its Innovative Territorial Actions. In order to achieve this task a significant investment in junior and senior human resources will be made by the Commission in this area, leading to similar decisions by MS and Regions..  Member States and Regions will be encouraged and assisted to introduce, whenever suitable, and at their discretion, financial incentives linked to targets, coupled with the adoption of public Action Plans to achieve them.

8. 8 2. LEARNING ABOUT WHAT WORKS: THE USE OF PROSPECTIVE COUNTERFACTUAL IMPACT EVALUATION (I)  All methodologies of impact evaluation addressing the question “what works?” will be promoted, since they reduce our ignorance on the casual link between policy and results.  A special effort will be made to promote counterfactual impact evaluation (CIE) - where impact is estimated by comparing outcomes for the beneficiaries of the intervention with outcomes for a similar population of non-beneficiaries - designed while interventions are being designed (prospective CIE). Why?  A large body of practical experience has accumulated all over the world in different policy fields and it is ready to be exploited.  Prospective CIE can provide ex ante a strong disciplinary effect on the transparent identifications of:  objectives (“what does > mean?”),  mechanisms for selecting beneficiaries (since a “similar” population of non- beneficiaries must be identified,) and on  the timely collection of necessary data.  Prospective CIE can provide local actors with some hard facts around which to mobilize and to express support or dissent.

9. 9  A place-based strategy also poses specific challenges to CIE (heterogeneity of interventions due to context-dependency, and multi-component nature of interventions) that call for a gradual learning process.  The European Commission should certainly not make CIE compulsory, but it should rather:  promote CIE and provide strong technical assistance,  create a “clearing house” for making all CIE studies accessible,  conduct CIE pilots. 2. THE USE OF PROSPECTIVE COUNTERFACTUAL IMPACT EVALUATION (II)  it allows the space for a cooperation between evaluators, policy-makers and beneficiaries,  its experimental and participatory nature makes it easier to deal with the ethical issues linked to randomization (which is particularly suitable to identify a “similar” population), like ensuring truly informed consent  A place-based strategy is particularly suitable to implement and develop CIE, because:

10. 10 MEMBER STATES AND REGIONS WILL ACCEPT AND EVEN SUPPORT THESE CHANGES IF…  Changes are truly inspired by the awareness of our ignorance on the causality links between action and results and are appropriately debated in the policy community, so as to dismiss fears about a sanctioning or procedure- oriented use of the innovations.  Changes are part of a more comprehensive reform – as advocated by the Report – whereby Commission’s discretionality:  Member States and Regions themselves – their administrators and politicians - become aware that there is no alternative :  is accompanied by a radical investment of DG Regio and DG Employment on human resources,  is kept at bay by new political checks-and-balances, via an informed and structured public debate in the Council and in the European Parliament,  is accompanied by much stronger experimentalism at place level.  cohesion policy will not last – unless downsized to a “de-minims” level and fragmented in sectoral Funds – if no decisive move is made towards making its results measurable, known and debated.