Presentation is loading. Please wait.

Presentation is loading. Please wait.

Evaluation within the context of the intervention of the Structural Funds in Portugal IFDR 1 October 2008.

Similar presentations


Presentation on theme: "Evaluation within the context of the intervention of the Structural Funds in Portugal IFDR 1 October 2008."— Presentation transcript:

1 Evaluation within the context of the intervention of the Structural Funds in Portugal IFDR 1 October 2008

2 General contents of the presentation  A. Fundamental issues for the evaluation of public interventions  B. Evolution of evaluation processes over the CSF  C. Evaluation in the NSRF

3 Evaluation Fundamental issues for the evaluation of public interventions Evaluation Fundamental issues for the evaluation of public interventions

4  Relevance – Evaluation of the closeness of the strategy to reality  Efficiency – Evaluation of the way in which resources were transformed into output and results  Efficacy – Evaluation of the way in which the resources contributed towards realising the objectives  Utility – Evaluation of the benefits for recipients  Sustainability – Evaluation of the extent and durability of the alterations made Aspects to take into consideration in the evaluation: Fundamental issues for the evaluation of public interventions

5 Relevance Evaluation of the closeness of the strategy to reality Fundamental issues for the evaluation of public interventions

6 Efficiency Evaluation of resources and its outputs Fundamental issues for the evaluation of public interventions

7 Efficacy Evaluation of the way in which resources were applied Fundamental issues for the evaluation of public interventions

8 Utility Evaluation of the benefits for recipients Fundamental issues for the evaluation of public interventions

9 Sustainability Evaluation of the extent and durability of the alterations made Fundamental issues for the evaluation of public interventions

10 Criteria of Quality Quality should be ensured: – in the Evaluation Process – in the Evaluation Report Fundamental issues for the evaluation of public interventions

11  – Proportionality  – Partnership  – Independence  – Transparency Principles of Evaluation

12 Principle of Proportionality Principles of Evaluation Financial and administrative resources of evaluations Proportional to the total amount of the expenses allocated to the OP Principle of Partnership Cooperation of the major agents throughout the planning process and implementation of evaluations

13 Principles of Evaluation Principle of transparency Disclosure of the main results of the evaluations, as well as their use as a resource for the qualification of public debate ACCOUNTABILITY Evaluation exercises by entities (internal or external to the Public Administration) functionally independent of the Management Authorities and of the entities with responsibilities in the NSRF and of the OP Principle of independence

14 A.Fundamental issues for the evaluation of the public interventions B. Development of Evaluation processes over the CSF C. Evaluation in the NSRF General contents of the presentation

15 Evaluation throughout the CSF 1. Evaluation in the CSF I ( ) 2. Evaluation in the CSF II ( ) 3. Evaluation in the CSF III ( )

16 CSF I Evaluation CONCEPTUAL MODEL Institucional and Functional independence between Management & Evaluation Different Structures methodologies  Ex-ante, ongoing e ex-post Evaluations  In collaboration: Commission - MemberState  Monitoring and evaluation functionally & institucionally independent from Management Observatory of CSF I Political Decision Makers  Independent Experts  Ex-post Evaluation  “IFDR” Coordination Terms of Reference

17 CSF II Evaluation Main changes:  Increasing importance of the ex-ante appreciation and ex-post evaluation  Evaluations were carried out by each Operacional Intervention.  The Commission encouraged the ex- post evaluation of the CSF II

18 CSF II Evaluation Main conclusions:  High performance  Around 7.7% of the GFCF in the period was directly induced by the CSF II  Around jobs were created and maintained up to 1999 directly induced by the investments made in the CSF II Ex-post evaluation

19 CSF II Evaluation  Changes into the Information System Ex-post evaluation creation of a single Information System Incorporating all of the Funds.  Improve the efficacy of the programmes  Improve monitoring functions  Separation between the activities of project control and monitoring implementation of a system of management by objectives ControlMonitoring ≠ Main Recommendations

20 CSF III Evaluation Evaluation Phases ( ) Ex-ante Evaluation  from the Member State responsibility  carried out by independent evaluators Mid Term Evaluation – December 2003  carried out by independent evaluators - responsibility of the MA  allocation of the Performance Reserve Mid Term Evaluation Update – December 2005 Ex-post Evaluation – 3 Years after programming period  carried out by independent evaluators - responsibility of the EC Diagnosis of the departure situation and analysis of the expected results. Analysis of the first results of the interventions, taking into account the Ex- Ante Evaluation.

21 CSF III Evaluation Ex-ante evaluation = basis for the preparation of the interventions * being the responsibility of the Member State Checks  strategy and objectives  coherence  execution and monitoring rules planned  3 different moments of evaluation Mid Term ReviewEx-anteEx-Post results of the evaluations Important input

22 CSF III Evaluation Enables:  the knowledge and analysis of the first results  the pertinence and the implementation of the objectives  the follow-up of financial allocations  the functioning of the monitoring and the execution of activities. * being the responsibility of the Management Authorities in collaboration with the Commission Mid Term Review Each OPCSF

23 Ex-post Evaluation CSF III Evaluation  Independent evaluators  3 years after end programming period  Commission in collaboration with Member State Intended to report on achievements and effects, the use of resources, the efficacy and efficiency of the OP, The analysis of the causes of inefficacy, the impacts

24 Included  results of the evaluation of macro-economic impacts through the analysis of different reports  cross-referencing them with the results of the evaluation studies  Performed by independent evaluators – until Dec 2003 Mid Term Review CSF III Evaluation Recommended Changes in OPMid Term Review  redefinition of strategies and priorities  redistribution of financial allocations  Funds  OP  Priorities

25 Performed by independent evaluations Mid Term Review CSF III Evaluation Changes suggested Intermediate reprogramming of 2004 Update of evaluation in 2005 until Dec 2003 Preparation of the subsequent interventions indicating priorities for the following programming period

26 Mid Term Review CSF III Evaluation quality of the evaluation exercise results of the Mid Term Review 4% of the total allocations planned for each Member State To reward the more efficient OP attribution of the efficiency reserve Reprogramming of 2004

27 Attribution of the Efficiency Reserve CSF III Evaluation Stage 1  Selection of the OP which meet the criteria Stage 2  Determination of the amount of the efficiency reserve to be awarded

28 Efficiency Reserve CSF III Evaluation Criteria for all OP - Considering the fulfilment by the OP of good management practices Management criteria  Management System Quality  Quality of the system of control  Quality of the selection criteria  Quality of the evaluation system Financial criteria  Global financial execution  Financial execution in LVT  Leverage effect Efficacy criteria  Measures the effects of the Funds through output of result indicators

29 Attribution of the Efficiency Reserve CSF III Evaluation Common Management Criteria  Very efficient: 5 of 6 criteria  Efficient: 4 of 6 criteria  Not Efficient: 3 criteria Financial Criteria  Very efficient: at least 80% of output & result Indicators  Efficient: at least 60% of output & result Indicators  Not efficient: < 60%  Very efficient: reaches 90% of targets  Efficient: 75 – 90% of targets  Not Efficient: < 75% Efficacy Criteria

30 Results of The Mid-Term Review CSF III Evaluation Main Conclusions – 2003  Relevance of the European Strategy For Employment  Reduction of unqualified employment  Reduction of early school leavers  Integration of education investment in companies  Modernization of Public Administration  Focus on Telecommunications and Energy (with direct impact of the Portuguese economy)

31 Results of The Mid-Term Review CSF III Evaluation Main Conclusions  Slow but progressive increase in the qualification of labour supply  Reduced expression of entrepreneurship in Portugal  High level of performance of OP in terms of approval, without perfect correspondence in terms of financial execution

32 Mid-Term Review CSF III Evaluation Main Conclusions for 2007 – 2013 Integration of policies of innovation and productivity Development of territorial competitiveness Active employment policies bringing them closer to the regional and local dimensions of structural unemployment Develop articulation of institutions and organizations as to avoid losses in the efficiency Improve global coordination

33 Analysis of the evaluation exercises In the whole period 2000 – 2006  133 evaluation studies  39 = evaluations of an obligatory nature  One significant investment – but only 3% of the global financial allocations of technical assistance of the CSF III  44% of the financial resources were intended for evaluations of an obligatory nature

34 Evaluation studies of an obligatory nature Analysis of the evaluation exercises 20 Mid-Term Review N.º of studies Total Cost l (1000 €) o 15Consultant Companies o 15Consultant Companies o 1Public Administration o 1 Institutions o 3 Research Universit. Centers o Follow up of Mid Term-Review

35 Analysis of the evaluation exercises Evaluation studies of an obligatory nature Strong points  Overall quality: credibility and utility  Significant progress on analysis of implementation processes; global rationality of the interventions; quantification of outputs;  Institutional involvement and use of the evaluation as a support tool for decision making Weak points  Rigid calendars  Exaggerated scope  Weaknesses in quantification or estimate of impacts

36 Analysis of the evaluation exercises Evaluation studies of an obligatory nature: Recommendations Good practices of  Generalization of exercises of this nature in management practice  Flexibility and subsidiarity  Disclosure

37 A.Fundamental issues for the evaluation of the public interventions B. The evolution of the evaluation processes throughout the CSF –Evaluation in the CSF I ( ) and CSF II( ) –Evaluation in the CSF III ( ) C. Evaluation in the NSRF General contents of the presentation

38 Evaluation in the NSRF Evaluation in the NSRF

39 Strategic purpose: Operational Programmes of three large Thematic Agendas  knowledge  science,  technology  innovation  promotion of high and sustained levels of economic and socio- cultural development and  of territorial qualification EDUCATION and QUALIFICATION Through: Equal opportunities. Structural Funds and Cohesion Fund Increased efficiency and quality of public institutions. Promoted by Short presentation of the NSRF

40 National strategic goals and priorities NSRF OP Renewed Lisbon Agenda National Public Policies EU Strategic Guidelines Growth and Employment OUTPUTSOUTPUTS Short presentation of the NSRF

41 Thematic Operational Agendas NSRF Operational Programmes Human Potential Factors of Competitivity Short presentation of the NSRF Enhancement of the Territory

42 Articulation of instruments of national public intervention ENDS PNACE PNPOT National Strategy For Energy National Employment Plan National Programme for Inclusion National Plan for Equality Cultural Development Plan National Strategic Plan for Tourism Technological Plan NSRF Territorial Development Plans Autonomous Regions Plans Nat. Strat. Rural Develop. Plan / FEADER National Nature and Biodiversity Conservation Strategy National Strategy for the Oceans PEAASAR Cities Policy Nat. Forestry Plan National Water Plan: Hydrographic Basins and Efficient Use of Water National Coastal Zone Integrated Strategy National Waste Plan PNAC Short presentation of the NSRF

43 Characteristics Before the start of the programming period During the programming period After the programming period Ex ante Ex post On going TIMING OBJECTIVE To improve the quality, efficacy, efficiency and coherence of the intervention of the funds and of the strategy and execution of the OP. Evaluation

44 Strategic Operational NATURE RESPONSI BILITY EC Ex-post MS Ex-anteOn-going Analysis of the evolution of an OP or group of OP in relation to the community and national priorities To support the monitoring of an OP Characteristics Environmental Strategic Evaluation of the effects of the Interventions on the Environment

45 Institutional Framework Evaluation – Responsibility

46 Institutional Framework On-going Evaluation Responsibility Strategic Monitoring Financial and Operational Monitoring Management, Evaluation, Internal Control

47  To ensure the efficient use of the Evaluations  To ensure that the evaluations are integrated and considered as a management tool during the implementation of the OP Evaluation Plan Covers the whole programming period  Nature of the evaluations  Description of the coordination and articulation mechanisms  An indicative list of Evaluation exercises  Foreseeable calendar

48 Evaluation Plan Includes  Strategic evaluations  Operational evaluations Level :  NSRF  OP/Fund Parts: I - Coordination II – Evaluation activities and reports

49 From the Mid Term Review to the on-going Evaluation Regulation no allows for a NEW CONCEPT of Evaluation: More flexible In line with the needs of the policy decision-making process and with the more efficient management of resources

50 More Flexible By Topic By OP By Priority By Actions Major projects For all OP For one OP At NSRF level From the Mid Term Review to the on-going Evaluation

51 Two closely related concepts, but with different objectives and functions Monitoring analysis, monitoring and verification of results Evaluation analysis and interpretation of the information obtained through the monitoring, and other sources in order to find and explain the effects of the interventions. From the Mid Term Review to the on-going Evaluation

52 Monitoring Evaluation Provides quality information and respective analyses Helps the decision-making process From the Mid Term Review to the on-going Evaluation

53 Regular monitoring Alerts for the need for Evaluation Regular follow up of the Evaluation Ensures implementation of reccommmendations From the Mid Term Review to the on-going Evaluation

54 Evaluation in the NSRF Practical applications Evaluation in the NSRF Practical applications

55 Reflections of the Evaluation of the CSF III on the NSRF CSF IIINSRF Insufficient concentration of financing options on the key areas corresponding to the major development problems of the country Reduction in the no. of operational interventions More refined criteria in the prioritization of projects Insufficient strategic alignment of operational instruments and of projects supported On-going strategic monitoring mechanisms Consolidation of rationality centers Articulation between strategic objectives and financing models

56 Reflections of the Evaluation of the CSF III on the NSRF CSF IIINSRF Difficulty in making the innovative potential of agents emerge Dissemination of good practices, monitoring systems, Evaluation and benchmarking, inter-institutional coordination and innovative approaches Insufficient attention to bolstering the institutional capacity of the Public Administration Notion of Strategic State: Strategic planning Inter-sectorial coordination Monitoring and Evaluation Insufficient focus on the quality of the effects, efficiency and sustainability of co-financed operations Quality of the expense: Relevance of the investments and economic and financial sustainability Results to be achieved and the effects this will bring

57 SWOT Analysis of the Ex-ante Evaluation of the NSRF What is it? It defines a strategy through the relationships existing between the strong and weak points with the most important trends noted in the global external context, whether due to the economy, legal impositions, etc.

58 SWOT Analysis of the Ex-ante Evaluation of the NSRF The term SWOT comes from the initials of the words:  S trengths  W eaknesses  O pportunities  T hreats What is it?

59 SWOT Analysis of the Ex-ante Evaluation of the NSRF Weaknesses Strengths External context Threats Opportunities Strategy Internal context

60 SWOT Analysis of the Ex-ante Evaluation of the NSRF What is it? Opportunities Threats Limiting exogenous factors (external analysis) Strong Points Weak Points Existing applications On-going dynamics (internal analysis) For a good strategist there are no threats, only opportunities

61 SWOT Analysis of the Ex-ante Evaluation of the NSRF How? Strong Points Weak Points Opportunities Threats Build a table with the four elements: with the strong and weak points on one side and the opportunities and threats on the other Identify the key elements which help to establish priorities and take strategic decisions

62 SWOT Analysis of the Ex-ante Evaluation of the NSRF OpportunitiesThreats Broadening of the process of integration of Iberian economies Completion of the High Speed Lisbon – Madrid project Financial difficulties in realising a fundamental project for the interna- tional connectivity of the Portuguese economy – high speed trains Extending the market ado to companies which until now have been more focussed on the domestic market (potentially interesting for Portuguese SME in industry and services), within the space of proximity which is the Spanish economy Growing affirmation of Spain as an European force, affecting the balanced development of Luso-Spanish trade and the maintenance of national decision- making centres in strategic sectors Analysis of the opportunities and threats which confront the development process, as well as of the strengths and weaknesses of the Portuguese situation.

63 StrengthsWeaknesses Environment and natural heritage Generalised coverage of the population in terms of water supply Insufficient levels of service in basic areas with emphasis on the drainage and treatment of waste waters Great diversity of natural heritage with high conservational value; 22% of the national territory is calssified with the status of nature protection and conservation Insufficient protection and enhancement of natural heritage, frequently associated to the lack of knowledge for the management of protected species and habitats SWOT Analysis of the Ex-ante Evaluation of the NSRF

64 SMART Criteria in the preparation of the Indicator File The definition of the objectives may follow SMART criteria  S pecific  M easurable  A ttainable  R ealistic  T imely What is it?

65  S pecific  They should clearly define what is intended  What?  Why?  How? SMART Criteria in the preparation of the Indicator File What is it?

66  M easurable  If it cannot be measured, it cannot be managed!!  The objectives should be quantifiable SMART Criteria in the preparation of the Indicator File What is it?

67  A ttainable  The objectives should be achievable SMART Criteria in the preparation of the Indicator File What is it?

68  R ealistic  Objectives should be defined taking in to account the effort required to achieve them SMART Criteria in the preparation of the Indicator File What is it?

69  T imely  Deadline - clarifies when it is intended for SMART Criteria in the preparation of the Indicator File What is it?

70 Indicator File SMARTSMART SMART Criteria in the preparation of the Indicator File Identification Planning and Execution Characterization

71 Appreciation of the Quality of the Evaluation Mid Term Evaluation CSF III ExcellentGoodSufficient Unacceptable 1. Information needs are satisfied 2.Pertinence of the scope of the Evaluation exercise 3. Appropriate character of the Methodology 4.Reliability of the Data 5.Validity of the Analysis 6.Credibility of the Results 7.Impartiality of the Conclusions 8.Utility of the Recommendations 9.Clarity of the Report Qualitative Appreciation Criteria

72 Thank you for your attention! Thank you for your attention!


Download ppt "Evaluation within the context of the intervention of the Structural Funds in Portugal IFDR 1 October 2008."

Similar presentations


Ads by Google