Explanation on the Purpose The quality and adequacy of projects and fanding activities go hand in hand with the quality and adequacy of the programme in which the project is part of. A regional or sectoral development programme (as e.g. the EU Structural Funds) aims at fostering inter-regional convergence and/or stabilising important regional growth poles for the national economy. The purpose is not to simply subsidise institutions, business and individuals, Funding is the means and not the end. Programme evaluation has to consider the respective quality, adequacy and implementation of a programme. Therefore the criteria applied in programme evaluation need also be translated into the quality assessment of the projects materialising the programme. Hence, as a truism, a project can only be as good as the programme and the programme can only be as good as the projects.
1. Ex-ante Evaluation
Structural Funds interventions are implemented in accordance to Operational Programmes (or Single Programming Documents) which must be (1) based on a reliable qualitative and quantitative analysis of the socio-economic situation of the region including its SWOTs and previous experiences, (2) the determination of a target- oriented development strategy, (3) the choice of appropriate measures to materialise the strategy, (4) suitable projects and (5) sound provisions governing financing, responsibilities, evaluation etc.. The assessment of the consistency of the above listed contents of an OP (or SPD respectively) is the core rationale of the Ex-ante evaluation. Further tasks to be carried out in the context of the Ex- ante Evaluation are (7) the quantification of the objectives and (8) the estimation of the overall socio-economic impact.
Objectives of the Ex-ante Evaluation Assessment of the appropriateness of the plan in addressing the regional development problems; Assessment of the consistency of analysis and concept; Definition of suitable indicators and quantification of objectives;
Objectives... Analysis of the adequacy of the implementation and monitoring arrangements; and Impact assessment
Assessment of Appropriateness Lessons learned from past experiences; Difficulties and shortcomings experienced and to be tackled; Assessment of the validity of data used in previous programmes; Consideration of thematical evaluations; Consideration of experiences in other programmes; and Verification of the socio-economic analysis and the consistency of SWOTs
Consistency of SWOTs S trengths are static features to be supported W eaknesses are static features to be reduced O pportunities are dynamic features to be reaped T hreats are dynamic features to be forestalled
Consistency of Analysis and Strategic Concept Are the strategic objectives well specified ? Do the strategic objectives sufficiently respond to the needs identified in the socio-economic analysis? Are the proposed measures justified by the strategy? Is the strategy sufficiently coordinated with other related supra-national or national policies/programmes (nat. regional policy, NAP, CIs, Objective 3 etc.) ?
Suitable Indicators (examples) Basic indicators: Programme indicators: Measure related indicators: Performance indicators: Gross regional value added Supported investment No. of supported SMEs Disbursement/ allocated funds
Quantification of Objectives Related to INPUTS-OUTPUTS-RESULTS- IMPACTS Determination of a realistic quantification of indicators Comparison with results achieved in former programme periods Proxy-indicators to monitor intangible effects
Appropriateness of the Implementation System Management and implementation responsibilities; Transparency; Efficiency and effectiveness of inter-institutional co- ordination; Competitiveness in project choice; Accountability in line with national and community regulations (competition policy, additionality, horizontal priorities, innovation, NAP etc.); and Partnership in programming and monitoring.
Impact assessment Micro-economic net effects (notably employment, productivity, regional GDP); Macro-economic net effects for large programmes (macro-econometric estimations and simulations on short-term demand, long-term supply, labour market and government sector)
2. Mid-term Evaluation and its Update
Mid-term Evaluation Update of the analysis of relevance; Update of the analysis of consistency of priorities; Analysis of the validity of the objectives; Comparison of plan and achievements (vertical and horizontal); Assessment of the implementation performance in terms of financial and administrative management; Contribution to the decision on the allocation of the Performance Reserve;
Mid-term Evaluation... The responsibility of the mid-term evaluation is with the member state; The evaluation is to be carried out by an independent assessor; An update of the mid-term evaluation is to be elaborated one year before termination of the CSF/SPD; and The evaluations are accompanied and approved by the monitoring committees.
Analysis at Programme Level Were recommendations from former evaluations sufficiently considered? Have the socio-economic problems (justifying the intervention) changed? Is the SWOT analysis adequate? Is the Programme still consistent and coherent? Has the programme been implemented professionally?
Analysis at Priority/Measure Level Financial implementation analysis Effectiveness analysis (plan vs. achievement) Efficiency analysis (unit cost comparisons) Impact analysis (GDP, net employment) Cf. EU Commission/DG Regio: Working Paper 8
Update Mid-term Evaluation The role of the updated mid-term evaluation is to confirm the implementation of recommendations put forward in the mid- term evaluation and to cover fields of analysis which could not be evaluated earlier. Cf. EU Commission/DG Regio: Working Paper 9
3. Ex-post Evaluation
The Ex-post evaluation is carried out under the responsibility of the EU Commission. It is elaborated as a global evaluation for each of the regional(Objective 1 and 2) and sectoral (Objective 3) programmes covering all targeted member states. The main purpose is to assess the quality, effectiveness and impacts of the programmes. It should clearly reveal the strengths and weaknesses of the planning process, the strategic approach and implementation. Lessons learned from this exercise should contribute to better policy formulation in current and future programming periods.
Scope of the Ex-post Evaluation Analysis of the relevance of strategy and the expenditure at EU, national and regional levels; Effectiveness analysis; Efficiency and impact assessment; Assessment of the delivery system; Analysis of the Community value added; and Lessons learned for the future and policy recommendations.
Relevance of Strategy and Expenditure Are the strategies adopted by the regions sufficiently relevant? Are the strategies appropriately implemented in practice? Has there been any deviation of the interpretation of the strategy (new direction)? Is the philosophy of the strategy still in line with that of the prior evaluations? Were the funds accordingly allocated (dispersion vs. Concentration? Were the changes of the strategy relevant or arbitrary? Was the aid intensity relevant? The degree of coherence in the approaches across all CSFs/SPDs of Objective 1 or Objective 2 respectively?
Effectiveness Analysis Comparison of plan and realisation in terms of policy priorities (e.g. private sector support, innovation, business related infrastructure, HRD etc.); Assessment of the formerly expected and realised performance of the policy instruments (e.g. financial engineering)
Efficiency and impacts Value for money assessment; Assessment of the net-effects (employment, productivity, regional GDP etc.); and Macro-econometric model estimations for major programmes (notably Objective 1)
Delivery System Assessment of the institutional management and implementation capacity; Functioning of partnership; Sufficiency of transparency; Functioning of the monitoring system; Functioning of the project selection procedures; Appropriateness of the practical provisions to ensure the legal underpinnings; and Functioning of the quality control mechanisms
Community Value Added Economic and social convergence; Internal and external Cohesion; and Institutional improvements (e.g. better coordination)
`Lessons Learned` and Policy Recommendations Conceptual level and supra-national regulations (EU regional policy formulation); Justification of the financial contributions and/or the level of additionality respectively; and Implementation level (management, M&E partnership etc.)
Project Assessment within Structural Funds Programmes Rules and funding regulations laid down in the Programme Complement Supra-national, national and regional funding regulations Assessment of the scope and aim of the application Control/audit Funding approval … Proof of fund utilisation
Criteria listed the Programme Complement The PC includes the specific funding regulations and criteria for all measures of the programme. Any institution/individual applying for funding needs to comply to the provisions made in the PC. The PC defines The valid funding provisions The thematical/sectoral scope of funding The pre-conditions of funding (e.g. specific target groups) Eligible expenditure Ceiling of the subsidy Period of validity Geographical coverage
Funding regulations Regulations of the EC (e.g. Council Regulation 1260/1999) European competition law (e.g. Article 92) National competition law Regional funding provisions Note: all regulations need to be co-ordinated
Project assessment grid (I) Is the project thematically suitable for funding under the respective measure? Does the project contribute to the quantified objectives of the measure? Is the project idea sufficiently original? Does the project suggest synergy effects for the programme? Is the required co-financing adequate?
Project assessment grid (II) Is the applicant (or the group of applicants) part of the target group? Is the applicant (or the group of applicants) trustworthy …
Controlling/auditing Internal controlling of the managing authority notifying fraud and irregularities Monitoring of the project progress by the Managing Authority Controlling and issuing of expenditure notes by the Paying Authority 5% random audits carried out by the Independent Auditing Unit