Presentation on theme: "Evaluation within the context of the intervention of the Structural Funds in Portugal IFDR 1 October 2008."— Presentation transcript:
Evaluation within the context of the intervention of the Structural Funds in Portugal IFDR 1 October 2008
General contents of the presentation A. Fundamental issues for the evaluation of public interventions B. Evolution of evaluation processes over the CSF C. Evaluation in the NSRF
Evaluation Fundamental issues for the evaluation of public interventions Evaluation Fundamental issues for the evaluation of public interventions
Relevance – Evaluation of the closeness of the strategy to reality Efficiency – Evaluation of the way in which resources were transformed into output and results Efficacy – Evaluation of the way in which the resources contributed towards realising the objectives Utility – Evaluation of the benefits for recipients Sustainability – Evaluation of the extent and durability of the alterations made Aspects to take into consideration in the evaluation: Fundamental issues for the evaluation of public interventions
Relevance Evaluation of the closeness of the strategy to reality Fundamental issues for the evaluation of public interventions
Efficiency Evaluation of resources and its outputs Fundamental issues for the evaluation of public interventions
Efficacy Evaluation of the way in which resources were applied Fundamental issues for the evaluation of public interventions
Utility Evaluation of the benefits for recipients Fundamental issues for the evaluation of public interventions
Sustainability Evaluation of the extent and durability of the alterations made Fundamental issues for the evaluation of public interventions
Criteria of Quality Quality should be ensured: – in the Evaluation Process – in the Evaluation Report Fundamental issues for the evaluation of public interventions
Principle of Proportionality Principles of Evaluation Financial and administrative resources of evaluations Proportional to the total amount of the expenses allocated to the OP Principle of Partnership Cooperation of the major agents throughout the planning process and implementation of evaluations
Principles of Evaluation Principle of transparency Disclosure of the main results of the evaluations, as well as their use as a resource for the qualification of public debate ACCOUNTABILITY Evaluation exercises by entities (internal or external to the Public Administration) functionally independent of the Management Authorities and of the entities with responsibilities in the NSRF and of the OP Principle of independence
A.Fundamental issues for the evaluation of the public interventions B. Development of Evaluation processes over the CSF C. Evaluation in the NSRF General contents of the presentation
Evaluation throughout the CSF 1. Evaluation in the CSF I ( ) 2. Evaluation in the CSF II ( ) 3. Evaluation in the CSF III ( )
CSF I Evaluation CONCEPTUAL MODEL Institucional and Functional independence between Management & Evaluation Different Structures methodologies Ex-ante, ongoing e ex-post Evaluations In collaboration: Commission - MemberState Monitoring and evaluation functionally & institucionally independent from Management Observatory of CSF I Political Decision Makers Independent Experts Ex-post Evaluation “IFDR” Coordination Terms of Reference
CSF II Evaluation Main changes: Increasing importance of the ex-ante appreciation and ex-post evaluation Evaluations were carried out by each Operacional Intervention. The Commission encouraged the ex- post evaluation of the CSF II
CSF II Evaluation Main conclusions: High performance Around 7.7% of the GFCF in the period was directly induced by the CSF II Around jobs were created and maintained up to 1999 directly induced by the investments made in the CSF II Ex-post evaluation
CSF II Evaluation Changes into the Information System Ex-post evaluation creation of a single Information System Incorporating all of the Funds. Improve the efficacy of the programmes Improve monitoring functions Separation between the activities of project control and monitoring implementation of a system of management by objectives ControlMonitoring ≠ Main Recommendations
CSF III Evaluation Evaluation Phases ( ) Ex-ante Evaluation from the Member State responsibility carried out by independent evaluators Mid Term Evaluation – December 2003 carried out by independent evaluators - responsibility of the MA allocation of the Performance Reserve Mid Term Evaluation Update – December 2005 Ex-post Evaluation – 3 Years after programming period carried out by independent evaluators - responsibility of the EC Diagnosis of the departure situation and analysis of the expected results. Analysis of the first results of the interventions, taking into account the Ex- Ante Evaluation.
CSF III Evaluation Ex-ante evaluation = basis for the preparation of the interventions * being the responsibility of the Member State Checks strategy and objectives coherence execution and monitoring rules planned 3 different moments of evaluation Mid Term ReviewEx-anteEx-Post results of the evaluations Important input
CSF III Evaluation Enables: the knowledge and analysis of the first results the pertinence and the implementation of the objectives the follow-up of financial allocations the functioning of the monitoring and the execution of activities. * being the responsibility of the Management Authorities in collaboration with the Commission Mid Term Review Each OPCSF
Ex-post Evaluation CSF III Evaluation Independent evaluators 3 years after end programming period Commission in collaboration with Member State Intended to report on achievements and effects, the use of resources, the efficacy and efficiency of the OP, The analysis of the causes of inefficacy, the impacts
Included results of the evaluation of macro-economic impacts through the analysis of different reports cross-referencing them with the results of the evaluation studies Performed by independent evaluators – until Dec 2003 Mid Term Review CSF III Evaluation Recommended Changes in OPMid Term Review redefinition of strategies and priorities redistribution of financial allocations Funds OP Priorities
Performed by independent evaluations Mid Term Review CSF III Evaluation Changes suggested Intermediate reprogramming of 2004 Update of evaluation in 2005 until Dec 2003 Preparation of the subsequent interventions indicating priorities for the following programming period
Mid Term Review CSF III Evaluation quality of the evaluation exercise results of the Mid Term Review 4% of the total allocations planned for each Member State To reward the more efficient OP attribution of the efficiency reserve Reprogramming of 2004
Attribution of the Efficiency Reserve CSF III Evaluation Stage 1 Selection of the OP which meet the criteria Stage 2 Determination of the amount of the efficiency reserve to be awarded
Efficiency Reserve CSF III Evaluation Criteria for all OP - Considering the fulfilment by the OP of good management practices Management criteria Management System Quality Quality of the system of control Quality of the selection criteria Quality of the evaluation system Financial criteria Global financial execution Financial execution in LVT Leverage effect Efficacy criteria Measures the effects of the Funds through output of result indicators
Attribution of the Efficiency Reserve CSF III Evaluation Common Management Criteria Very efficient: 5 of 6 criteria Efficient: 4 of 6 criteria Not Efficient: 3 criteria Financial Criteria Very efficient: at least 80% of output & result Indicators Efficient: at least 60% of output & result Indicators Not efficient: < 60% Very efficient: reaches 90% of targets Efficient: 75 – 90% of targets Not Efficient: < 75% Efficacy Criteria
Results of The Mid-Term Review CSF III Evaluation Main Conclusions – 2003 Relevance of the European Strategy For Employment Reduction of unqualified employment Reduction of early school leavers Integration of education investment in companies Modernization of Public Administration Focus on Telecommunications and Energy (with direct impact of the Portuguese economy)
Results of The Mid-Term Review CSF III Evaluation Main Conclusions Slow but progressive increase in the qualification of labour supply Reduced expression of entrepreneurship in Portugal High level of performance of OP in terms of approval, without perfect correspondence in terms of financial execution
Mid-Term Review CSF III Evaluation Main Conclusions for 2007 – 2013 Integration of policies of innovation and productivity Development of territorial competitiveness Active employment policies bringing them closer to the regional and local dimensions of structural unemployment Develop articulation of institutions and organizations as to avoid losses in the efficiency Improve global coordination
Analysis of the evaluation exercises In the whole period 2000 – 2006 133 evaluation studies 39 = evaluations of an obligatory nature One significant investment – but only 3% of the global financial allocations of technical assistance of the CSF III 44% of the financial resources were intended for evaluations of an obligatory nature
Evaluation studies of an obligatory nature Analysis of the evaluation exercises 20 Mid-Term Review N.º of studies Total Cost l (1000 €) o 15Consultant Companies o 15Consultant Companies o 1Public Administration o 1 Institutions o 3 Research Universit. Centers o Follow up of Mid Term-Review
Analysis of the evaluation exercises Evaluation studies of an obligatory nature Strong points Overall quality: credibility and utility Significant progress on analysis of implementation processes; global rationality of the interventions; quantification of outputs; Institutional involvement and use of the evaluation as a support tool for decision making Weak points Rigid calendars Exaggerated scope Weaknesses in quantification or estimate of impacts
Analysis of the evaluation exercises Evaluation studies of an obligatory nature: Recommendations Good practices of Generalization of exercises of this nature in management practice Flexibility and subsidiarity Disclosure
A.Fundamental issues for the evaluation of the public interventions B. The evolution of the evaluation processes throughout the CSF –Evaluation in the CSF I ( ) and CSF II( ) –Evaluation in the CSF III ( ) C. Evaluation in the NSRF General contents of the presentation
Evaluation in the NSRF Evaluation in the NSRF
Strategic purpose: Operational Programmes of three large Thematic Agendas knowledge science, technology innovation promotion of high and sustained levels of economic and socio- cultural development and of territorial qualification EDUCATION and QUALIFICATION Through: Equal opportunities. Structural Funds and Cohesion Fund Increased efficiency and quality of public institutions. Promoted by Short presentation of the NSRF
National strategic goals and priorities NSRF OP Renewed Lisbon Agenda National Public Policies EU Strategic Guidelines Growth and Employment OUTPUTSOUTPUTS Short presentation of the NSRF
Thematic Operational Agendas NSRF Operational Programmes Human Potential Factors of Competitivity Short presentation of the NSRF Enhancement of the Territory
Articulation of instruments of national public intervention ENDS PNACE PNPOT National Strategy For Energy National Employment Plan National Programme for Inclusion National Plan for Equality Cultural Development Plan National Strategic Plan for Tourism Technological Plan NSRF Territorial Development Plans Autonomous Regions Plans Nat. Strat. Rural Develop. Plan / FEADER National Nature and Biodiversity Conservation Strategy National Strategy for the Oceans PEAASAR Cities Policy Nat. Forestry Plan National Water Plan: Hydrographic Basins and Efficient Use of Water National Coastal Zone Integrated Strategy National Waste Plan PNAC Short presentation of the NSRF
Characteristics Before the start of the programming period During the programming period After the programming period Ex ante Ex post On going TIMING OBJECTIVE To improve the quality, efficacy, efficiency and coherence of the intervention of the funds and of the strategy and execution of the OP. Evaluation
Strategic Operational NATURE RESPONSI BILITY EC Ex-post MS Ex-anteOn-going Analysis of the evolution of an OP or group of OP in relation to the community and national priorities To support the monitoring of an OP Characteristics Environmental Strategic Evaluation of the effects of the Interventions on the Environment
Institutional Framework On-going Evaluation Responsibility Strategic Monitoring Financial and Operational Monitoring Management, Evaluation, Internal Control
To ensure the efficient use of the Evaluations To ensure that the evaluations are integrated and considered as a management tool during the implementation of the OP Evaluation Plan Covers the whole programming period Nature of the evaluations Description of the coordination and articulation mechanisms An indicative list of Evaluation exercises Foreseeable calendar
Evaluation Plan Includes Strategic evaluations Operational evaluations Level : NSRF OP/Fund Parts: I - Coordination II – Evaluation activities and reports
From the Mid Term Review to the on-going Evaluation Regulation no allows for a NEW CONCEPT of Evaluation: More flexible In line with the needs of the policy decision-making process and with the more efficient management of resources
More Flexible By Topic By OP By Priority By Actions Major projects For all OP For one OP At NSRF level From the Mid Term Review to the on-going Evaluation
Two closely related concepts, but with different objectives and functions Monitoring analysis, monitoring and verification of results Evaluation analysis and interpretation of the information obtained through the monitoring, and other sources in order to find and explain the effects of the interventions. From the Mid Term Review to the on-going Evaluation
Monitoring Evaluation Provides quality information and respective analyses Helps the decision-making process From the Mid Term Review to the on-going Evaluation
Regular monitoring Alerts for the need for Evaluation Regular follow up of the Evaluation Ensures implementation of reccommmendations From the Mid Term Review to the on-going Evaluation
Evaluation in the NSRF Practical applications Evaluation in the NSRF Practical applications
Reflections of the Evaluation of the CSF III on the NSRF CSF IIINSRF Insufficient concentration of financing options on the key areas corresponding to the major development problems of the country Reduction in the no. of operational interventions More refined criteria in the prioritization of projects Insufficient strategic alignment of operational instruments and of projects supported On-going strategic monitoring mechanisms Consolidation of rationality centers Articulation between strategic objectives and financing models
Reflections of the Evaluation of the CSF III on the NSRF CSF IIINSRF Difficulty in making the innovative potential of agents emerge Dissemination of good practices, monitoring systems, Evaluation and benchmarking, inter-institutional coordination and innovative approaches Insufficient attention to bolstering the institutional capacity of the Public Administration Notion of Strategic State: Strategic planning Inter-sectorial coordination Monitoring and Evaluation Insufficient focus on the quality of the effects, efficiency and sustainability of co-financed operations Quality of the expense: Relevance of the investments and economic and financial sustainability Results to be achieved and the effects this will bring
SWOT Analysis of the Ex-ante Evaluation of the NSRF What is it? It defines a strategy through the relationships existing between the strong and weak points with the most important trends noted in the global external context, whether due to the economy, legal impositions, etc.
SWOT Analysis of the Ex-ante Evaluation of the NSRF The term SWOT comes from the initials of the words: S trengths W eaknesses O pportunities T hreats What is it?
SWOT Analysis of the Ex-ante Evaluation of the NSRF Weaknesses Strengths External context Threats Opportunities Strategy Internal context
SWOT Analysis of the Ex-ante Evaluation of the NSRF What is it? Opportunities Threats Limiting exogenous factors (external analysis) Strong Points Weak Points Existing applications On-going dynamics (internal analysis) For a good strategist there are no threats, only opportunities
SWOT Analysis of the Ex-ante Evaluation of the NSRF How? Strong Points Weak Points Opportunities Threats Build a table with the four elements: with the strong and weak points on one side and the opportunities and threats on the other Identify the key elements which help to establish priorities and take strategic decisions
SWOT Analysis of the Ex-ante Evaluation of the NSRF OpportunitiesThreats Broadening of the process of integration of Iberian economies Completion of the High Speed Lisbon – Madrid project Financial difficulties in realising a fundamental project for the interna- tional connectivity of the Portuguese economy – high speed trains Extending the market ado to companies which until now have been more focussed on the domestic market (potentially interesting for Portuguese SME in industry and services), within the space of proximity which is the Spanish economy Growing affirmation of Spain as an European force, affecting the balanced development of Luso-Spanish trade and the maintenance of national decision- making centres in strategic sectors Analysis of the opportunities and threats which confront the development process, as well as of the strengths and weaknesses of the Portuguese situation.
StrengthsWeaknesses Environment and natural heritage Generalised coverage of the population in terms of water supply Insufficient levels of service in basic areas with emphasis on the drainage and treatment of waste waters Great diversity of natural heritage with high conservational value; 22% of the national territory is calssified with the status of nature protection and conservation Insufficient protection and enhancement of natural heritage, frequently associated to the lack of knowledge for the management of protected species and habitats SWOT Analysis of the Ex-ante Evaluation of the NSRF
SMART Criteria in the preparation of the Indicator File The definition of the objectives may follow SMART criteria S pecific M easurable A ttainable R ealistic T imely What is it?
S pecific They should clearly define what is intended What? Why? How? SMART Criteria in the preparation of the Indicator File What is it?
M easurable If it cannot be measured, it cannot be managed!! The objectives should be quantifiable SMART Criteria in the preparation of the Indicator File What is it?
A ttainable The objectives should be achievable SMART Criteria in the preparation of the Indicator File What is it?
R ealistic Objectives should be defined taking in to account the effort required to achieve them SMART Criteria in the preparation of the Indicator File What is it?
T imely Deadline - clarifies when it is intended for SMART Criteria in the preparation of the Indicator File What is it?
Indicator File SMARTSMART SMART Criteria in the preparation of the Indicator File Identification Planning and Execution Characterization
Appreciation of the Quality of the Evaluation Mid Term Evaluation CSF III ExcellentGoodSufficient Unacceptable 1. Information needs are satisfied 2.Pertinence of the scope of the Evaluation exercise 3. Appropriate character of the Methodology 4.Reliability of the Data 5.Validity of the Analysis 6.Credibility of the Results 7.Impartiality of the Conclusions 8.Utility of the Recommendations 9.Clarity of the Report Qualitative Appreciation Criteria
Thank you for your attention! Thank you for your attention!