1. Results of the midterm evaluation exercise on the Leader + programme for Portugal Special focus on evaluating innovation Pedro Afonso Fernandes (CIDEC – Lisbon - Portugal) Expert meeting on Guidance for Common Monitoring and Evaluation Framework Seminar on monitoring and evaluation of the LEADER approach DG AGRI, Brussels – Room LOI 130B 25 September 2006

2. Pedro Afonso Fernandes (pedro.afonso@cidec.pt)2 Innovation on evaluating innovation The innovative starting point The 3 × 3 evaluation matrix Specific evaluation tools Deal with lack of self-evaluation Recommendations on evaluation procedures and tools

3. Pedro Afonso Fernandes (pedro.afonso@cidec.pt)3 The innovative starting point (I) Evaluating a innovative programme such as LEADER+ requires a innovative starting point at … two levels: Community level: Guidelines produced by STAR committee (Doc. VI/43503/02-Rev.1, January 2002)  Detailed Common Evaluation Questions and Criteria that cover a wide range of evaluation topics:  Implementation of the LEADER+ method  Implementation of the 3 actions (integrated pilot strategies, co-operation and networking)  Impact – Overall objectives of the Structural Funds  Impact – Specific objectives of LEADER+ (Value Added)  Financing, management and evaluation

4. Pedro Afonso Fernandes (pedro.afonso@cidec.pt)4 The innovative starting point (II) National level: the midterm evaluator (CIDEC) developed specific evaluation questions in order to cover the main concerns of the national authority (IDRHa – Institute of Rural Development and Hydraulics), namely:  Evaluation of the LAG’s functioning and practices  Evaluation of local strategies’ appropriateness and sustainability  Evaluation of the monitoring and information systems Note: the specific (national) evaluation questions complement the DG AGRI’s Common Questions  Those additional questions reinforce the evaluation of LEADER’s value added (e.g. development of competences at local level, strategies sustainability, diversification effects)

5. Pedro Afonso Fernandes (pedro.afonso@cidec.pt)5 The 3 × 3 evaluation matrix Evaluating LEADER is a challenge that results from the cross of 3 levels of intervention with 3 types of action Action 1: local strategies Action 2: co- operation Action 3: networking National level (IDRHa and A.3 partners) Local level (LAG) Project level (promoters) NO TARGETS

6. Pedro Afonso Fernandes (pedro.afonso@cidec.pt)6 Specific evaluation tools (I) Multiple methodological tools were mobilized in order to deal with the 3 × 3 matrix Action 1: local strategies Action 2: co- operation Action 3: networking National level (IDRHa and A.3 partners) Interviews Local level (LAG) Inquiry Case Studies Inquiry Case Studies Inquiry Project level (promoters) Inquiry Case Studies Inquiry Case Studies -

7. Pedro Afonso Fernandes (pedro.afonso@cidec.pt)7 Specific evaluation tools (II) Case studies:  26 LAG covered (one half of the total)  Interview with LAG’s coordinator  Contact with 2 projects from Action 1 and interview  Analysis of the administrative processes (dossiers) of 10 projects (9 from Action 1 and one from Action 2)  Telephonic contact with some local partners  Documental analysis (e.g. local strategies, criteria to select projects)  Quantitative treatment of qualitative information  Detailed report

8. Pedro Afonso Fernandes (pedro.afonso@cidec.pt)8 Specific evaluation tools (III) Promoters Inquiry:  Main objective: find values for a set of indicators specially developed by the midterm evaluator with the support of the national authority (IDRHa)  The inquiry was applied with 9 different questionnaires according to the type of project  Good response rate (51%) and reliability of the data  Quality control procedures: statistical tests that compared the data from the inquiry with the data from the case studies  Impact and results indicators were favoured in order to quantify the LEADER’s value added

9. Pedro Afonso Fernandes (pedro.afonso@cidec.pt)9 Specific evaluation tools (IV) Examples of impacts and results measured by the Promoters Inquiry for Portugal:  63% of the projects incorporated technologies (43% ITC)  40% of the projects are innovative (product or process)  10% of the projects involved universities or polytechnics  60% of the “productive” projects mobilized local raw materials and/or services  70% of jobs created were for women  LEADER+ stimulates the development of competences on 40% of the promoters inquired and …  … the development of new different activities (no financed) on 12% of the projects – Diversification effects

10. Pedro Afonso Fernandes (pedro.afonso@cidec.pt)10 Deal with lack of self-evaluation Only 5 on 26 LAG studied by the midterm evaluator have permanent self-evaluation practices Even if the LAG has this kind of practices, an independent evaluator are not mobilized typically The midterm evaluator deals with those problems by involving LAG’s representatives and (selected) promoters and partners in the case studies’ works  In the future, national (or even European) authorities should develop a vademecum or guide to favour the self-evaluation by the Local Action Groups

11. Pedro Afonso Fernandes (pedro.afonso@cidec.pt)11 Recommendations on evaluation procedures and tools (I) At European level, it is important to develop the Common Evaluation Questions and Criteria, that is, “The Starting Point” by:  Incorporating new questions/criteria, namely, suggested by midterm evaluators  Simplifying some topics (e.g. the CEQ 4.1 “To what extent has LEADER+ contributed to promote and disseminate new integrated approaches to rural development (…)” was already covered by the action-specific CEQ)  Developing a Common Set of Indicators that would facilitate the impact and results evaluation at European and National levels (the Portuguese set already developed could be a “starting point” for that huge task)

12. Pedro Afonso Fernandes (pedro.afonso@cidec.pt)12 Recommendations on evaluation procedures and tools (II) At National level, it is important:  To establish targets (quantified goals) for local strategies, co-operation and networking  To develop Complementary Evaluation Questions, Criteria and Indicators in order to reinforce the evaluation of LEADER’s value added and incorporate national concerns  To develop a mix of evaluation tools that could solve the puzzle of the 3 × 3 Matrix  Case studies should be privileged because of the innovative local-based nature of LEADER’s projects  Project inquiries are essential to collect quantitative data, namely impact and results indicators