1 Ex post evaluation of cohesion policy interventions financed by the Cohesion Fund Work Package C – cost-benefit analysis of selected environment projects Adam Abdulwahab Evaluation network, Brussels, 20th October 2011

# Background Study period: January 2010 – September 2011 Consortium: 2 09 / 2011 Ex post evaluation for Cohesion Fund , WP C

# The 3 overall questions the present study set out to address 1.What were the impacts of the examined projects? 2.How can ex post cost-benefit analyses contribute to the practice of ex ante cost-benefit analyses? 3.What are the potentials and limits of carrying out an ex post cost-benefit analysis to identify and/or analyse the impact of projects? Is it an appropriate tool for impact analysis? 3 09 / 2011 Ex post evaluation for Cohesion Fund , WP C Recommendations

# Methodology 4 09 / 2011 Ex post evaluation for Cohesion Fund , WP C Initial list of 40 major environmental Cohesion Fund/ISPA projects that were completed (or nearly completed). 20 projects were selected to ensure a good mix of sectors (waste management, wastewater and water management) and to ensure wide geographical coverage. The short listed 20 projects were briefly examined (brief project descriptions based on available material and initial contacts). 10 projects were selected for in-depth case studies using CBA ex post. The ten case studies were based on extensive data collection among stakeholders. The figure presents a brief overview of data collection approaches applied -> -> -> -> ->

# Methodology used in the case studies The basis of the study was 10 Cost Benefit Analyses ex post, where the following issues were also addressed:  Utilisation,  Validity of assumptions made during the ex ante analyses,  Risk mitigation measures, including consequences of mitigated and (both foreseen and unforeseen) non-mitigated risks,  Costs and benefits that cannot be expressed in monetary forms,  Meeting environmental requirements,  The contribution of accompanying actions which are outside the project but intended to enhance the project success (qualitative approach only),  Unintended effects, and  The calculation of the margin of error of the ex post cost benefit analysis / 2011 Ex post evaluation for Cohesion Fund , WP C

# Overview of the 10 case studies Ex post evaluation for Cohesion Fund , WP C Waste management Water management Waste water management 01 Bulgaria 03 Pilsen 17 Barcelona 06 Crete 09 Zaragosa 27 Hungary 22 Dublin 13 Madrid 29 Poland 50 Portugal

# 7 09 / 2011 Ex post evaluation for Cohesion Fund , WP C Interpretation of the study results – based on the 10 cases  Positive net impacts for some but not all projects  2 projects (solid waste PT, water supply ES) showed net positive impacts  4 projects (water CZ, water management ES, water PL, wastewater IE) showed positive impacts (better services to population, impacts on natural environment) but at high costs resulting in negative ENPV and B/C ratio < 1  4 projects produced a stream of benefits that did not outweigh costs - overcapacity (solid waste ES), demand forecasts (solid waste EL), expensive technical solutions (solid waste BG, wastewater HU)  But caution: relying on ENPV and B/C ratio can be misleading!  ENPV >0 and B/C ratio >1: monetised benefits exceed monetised costs over time  ENPV <0 and B/C ratio <1: monetised cost exceed monetised benefits over time  Benefits are more difficult to monetise (i.e. quantify and attach economic value expressed in money)

# 8 09 / 2011 Ex post evaluation for Cohesion Fund , WP C Interpretation of the study results – based on the 10 cases  Legal compliance:  Does not ensure positive impacts in economic welfare terms – in most cases this was achieved at costs that exceeded the benefits that could be quantified  Achieving positive ENPV may be impossible on project level; especially where projects were undertaken to meet regulatory requirements  Two cases of non-compliance driven objectives (water supply ES, water management ES): objectives had environmental focus relevant to the needs of the region  Generally, the projects generated positive side effects in terms of environmental awareness and institutional learning  Distributional effects  many projects benefit low-income regions, indirectly paid by higher income regions  CBA as a tool does not capture such distributional effects.

# 01 Bulgaria Waste management Investment 57 million / population 9 09 / 2011 Ex post evaluation for Cohesion Fund , WP C  Project objectives: to achieve solid waste management in full compliance with EU and Bulgarian Regulations and as a result avoid potential environmental damages from landfills  Comparison of ex ante and ex post CBA results: Analysis:  Risk assessment: main challenge is affordability because of low income level  Financial sustainability: only operational costs can be financed by the waste tax, reinvestment costs are not included e.g. for sealing of the cells  Direct benefits: Reduction of illegal waste disposal  Other outcomes: A first step towards modern waste handling system  The project could have taken waste management further by introducing much more recycling measures in the project Legal compliance as driverYes Technical solutionOk Capacity issuesOk Budget reasonabilityOk Options analysisN/A

# 06 Greece Waste management Investment 28 million / population / 2011 Ex post evaluation for Cohesion Fund , WP C  Project objectives: to collect and treat waste produced by the Municipalities of Chania, Akrotiri, Souda, Keramies, Eleftherios Venizelos, Therisos, Kydonia, Platania and Mousouri  Comparison of ex ante and ex post CBA results: Analysis:  Risk assessment: financial sustainability is an issue. Operations need to focus on improving recycling rate and market uptake of compost.  Financial sustainability: operational costs are much higher than budgeted, and revenues lower than budgeted.  Direct benefits: reduction of illegal waste disposal of tons/year, recycling of tons/year Legal compliance as driverYes Technical solutionOk Capacity issuesOver capacity Budget reasonabilityOk Options analysisYes  Unrealised benefits: use of compost, reduced use of pesticides, refined sorting of water, 100% waste management coverage  Hindrances for benefits: lack of awareness among farmers, lack of education of households to sort waste, lack, lack of reinforcement of illegal waste disposal  Unintended effects: waste management know-how, increased environmental awareness

# 13 Madrid/Spain Waste & energy recovery Investment 74 million / 3 million population / 2011 Ex post evaluation for Cohesion Fund , WP C  Project objectives: to seal off of the 'old' landfill, the construction of an Energy Recovery Plan and landscape restoration on top of the old landfill  Comparison of ex ante and ex post CBA results: Analysis:  Risk assessment: No major risk elements were identified  Financial sustainability: The energy generation component generate a surplus, while the other components require operational funding from Municipality  Direct benefits: Closing off illegal landfill, stop to negative environmental impacts, energy generation  Unrealised benefits: unutilized synergies with other facilities of the entire Valdomingez facility  Hindrance to benefits: Contractual bindings of current contracts with different operators  Other outcomes: the project was important part of the combined Madrid Waste Management facility Legal compliance as driverYes Technical solutionOk Capacity issuesOver capacity Budget reasonabilityOk Options analysisPart of bidding process

# 50 Lipor/Portugal Waste management Investment 80 million / population / 2011 Ex post evaluation for Cohesion Fund , WP C  Project objectives: to complement and to extend some of the existing activities LIPOR and its city councils shareholders and to remedy some environmental liabilities, aiming at the consolidation of the integrated management system in the region.  Comparison of ex ante and ex post CBA results: Analysis:  Risk assessment: the composting facility is the main reason for poor economic performance of the overall investment  Financial sustainability is ensured by green or 'guaranteed' administrative price and close control of costs  Direct benefits: comprehensive and efficient management of urban solid waste  Hindrances for benefits: implementation delays, acceptance of compost quality by farmers, awareness of waste sorting and recycling among the population Legal compliance as driverYes Technical solutionOk Capacity issuesOk Budget reasonabilityOk Options analysisPartly

# / 2011 Ex post evaluation for Cohesion Fund , WP C 03 Czeck Rep / Pilsen: combined water Investment 30 million EUR / population Project objectives: was to expand and rehabilitate the water and wastewater system in order to comply with the EU environmental acquis Comparison of ex ante and ex post CBA results: Analysis: Legal compliance as driverYes Technical solutionOk Capacity issuesOk Budget reasonabilityOk Options analysisuncertain FNPV M€FRR % ENPV M€ ERR %B/C ratio Ex ante % %0.74 Ex post % %0.66 Risk assessment: No major risks have been identified, risks were assessed and mitigated Financial sustainability: Yes, the affordability of the total tariffs showed no risk for payment problems among the households Direct benefits: Improved quality of drinking water, reduction of pollution in the river due to less spill over and connection of 1000 household to sewage system Other outcomes: the environment in the river is improving

# 09 Spain /Zaragoza: Water supply Investment 121 million EUR / population / 2011 Ex post evaluation for Cohesion Fund , WP C  Project objectives: to implement a water supply system capable of guaranteeing constant distribution of good quality water in the target area.  Comparison of ex ante and ex post CBA results: Analysis: Risk assessment: undertaken and the design is based on the results Financial sustainability: The project is sustainable thanks to full cost recovery tariffs Direct benefits: improved water quality Other outcomes: reduction in water treatment costs – reduced water leakages – awareness raising activities Legal compliance as driverNot only Technical solutionOk Capacity issuesOk Budget reasonabilityOk Options analysisYes

# 17 Spain /Barcelona: Waste water Investment 69 million EUR / population / 2011 Ex post evaluation for Cohesion Fund , WP C  Project objectives: to regulate flooding in certain parts of the city building by the retention tanks and belonging connectors.  Comparison of ex ante and ex post CBA results: Analysis: Risk assessment: it was concluded that the project is unlikely to ever generate an economic surplus. Important variable were investment costs, value of beach days, avoided damage costs and the discount rate. Financial sustainability: The project is not sustainable. No income is generated. However, the economic assessment shows a sustainable investment. Direct benefits: Reduced risk of flooding, improved know- how of water management, access to ground water for irrigation and street cleaning, improved ecosystem in marine water. Other outcomes: a lot of knowledge and experience has been gathered. The project has many visits from other cities and countries. High awareness raising among population. Legal compliance as driverNo Technical solutionOk Capacity issuesOk Budget reasonabilityOk Options analysisYes

# / 2011 Ex post evaluation for Cohesion Fund , WP C 22 Dublin Ireland: Waste water Investment 500 million EUR / population Project objectives: to comply with the environmental legislation, and through this compliance improve the quality of life of the Dubliners Comparison of ex ante and ex post CBA results: Analysis: Legal compliance as driverYes Technical solutionNot optimal Capacity issuesUnder- capacity Budget reasonabilityNo Options analysisYes Risk assessment: The project is very unlikely to generate an economic surplus Financial sustainability: need for further identification of investment funds and a revision of user fee structure Direct benefits: an increase in housing permissions and recreational benefits, in particular with cleaner Dublin Bay beaches Hindrances to realising benefits: the design capacity of the new infrastructure is too low and it does not comply with the requirements of the newly designated sensitive waters, hence, further investments are needed to fully reap the benefits

# / 2011 Ex post evaluation for Cohesion Fund , WP C 27 Szeged/Hungary: Waster water Investment 95 million EUR / population Project objectives: to comply with the Waste Water Directive, including the expansion of the existing waste-water treatment plant, sewer expansion and household connections Comparison of ex ante and ex post CBA: Analysis: Legal compliance as driverYes Technical solutionOk Capacity issuesOk Budget reasonabilityNo Options analysisN/A Risk assessment: financial risk was investment costs overrun, economic feasibility of sewage network expansion is uncertain, the plant expansion is not economically feasible. Financial sustainability: Yes, but fixed assets are not in the books. Tariffs are sufficient to cover operating and reinvestment costs. Direct benefits: Extension of the sewage system to new users Other outcomes: increased awareness among population. The project was excellently run.

# / 2011 Ex post evaluation for Cohesion Fund , WP C 29 Szczecin/Poland: Waste water Investment 281 million EUR / population Project objectives: a complete renovation of an outdated waste water treatment Comparison of ex ante and ex post CBA results: Analysis: Legal compliance as driverYes Technical solutionOK Capacity issuesOK Budget reasonabilityOK Options analysisYes Risk assessment: The project is not financially sustainable and likely to lead to an economic return in the future. Financial sustainability: the project appears to be financially sustainable as user fees have been allowed to increase in line with the increase in the quality of service provision. Direct benefits: improved quality of waste water and water supply services, and improved environmental conditions of the Oder river. Other outcomes: Szczecin is no longer a HELCOM hotspot.

# 2. What have we learned that can improve ex ante CBA? Mar 2011 Ex post evaluation for Cohesion Fund , WP C "it is a matter of making the financial analysis look as bad as possible in order to increase the funding need, and to make the economic analysis to look as positive as possible in order to justify the public funding" Quote from workshop with MS The main shortcomings identified:  not integrated in the decision process (merely 'a ticking the box' exercise)  not looking at individual components  missing the bigger pictures (e.g. synergies, risk un-realised benefits)

# 2. What have we learned that can improve ex ante CBA? / 2011 Ex post eval cohesion fund_DG Regio Recommendations:  CBA thinking should apply to the selection of alternatives, prior to the final design of the project.  Do the CBAs much earlier in the process. A solid CBA should precede the final technical design of the project.  CBA to be related to the Master Plan context.  Cost efficiency analysis could be considered for 'need to have' projects.  Clearer distinction between the financial and the economic CBA. CBA training to enhance the CBA culture from the start of project generation Access to common resources, e.g. best practice examples, catalogue of how to calculate externalities, standard values, etc.

# 3. Is ex post CBA a good methodology to identify and assess the impacts of the projects?  CBA as a methodology is not common practice for ex post evaluation  However this study has applied CBA post-decision Lessons learnt: 1)Timing matters: ex post analysis is too early right after project completion 2)starting from individual components is the easiest way to identify the benefit elements 3)concentrate on valorisation of the main benefit elements 4)wider benefits are often the important outcomes but difficult to quantify / 2011 Ex post evaluation for Cohesion Fund , WP C Considerations  break-even analysis might be more convincing to illustrate economic surplus/deficit  develop a price and benefit catalogue (Excel tool) to support CBA in practice  if the ex ante CBA is done according to the guidelines, the ex post CBA is merely 'a reality check' updating key variables  CBA to be combined with other qualitative impact assessment methodologies to improve the capturing of wider benefits

# Conclusions  CBA is an efficient tool both for ex ante and ex post analysis  In practice, however, CBA is not used to its potential since practitioners find it too academic, too complicated….  What to do about that? –encourage CBA thinking early in the project process –simplify the CBA exercise to focus on the most critical aspects –CBA training (development of eLearning tools for practitioners) –access to 'easy to use' CBA resources (best practice, on-line QA, cost/benefit catalogue, benefit transfer studies, etc)  Ex post CBA's? –yes the CBA methodology works –Institutional set-up? (MS responsibility or DG Regio?) / 2011 Ex post evaluation for Cohesion Fund , WP C

# 23 3 Feb 2011 Ex post eval cohesion fund_DG Regio Thank you very much for your attention!