1. 2_Monitoring and Evaluation of CAP 2014 – 2020 Approach of PII
Jela Tvrdonova 2015

2. Contents
Why do we need a common approach to M&E of common agriculture policy?
Who is responsible?
How does the policy and the legal framework look like?
How is the M&E system built up and what guidance is provided?
Evaluation questions
Indicators
Evaluation approach
What lessons can be drawn from the assessment of impacts in rural development of the current programming period?

3. Need for coordination and collaboration
Who is responsible?
One policy = one evaluation
→ CAP common impacts
Pillar I
European
Commission
Pillar II
Member
States
When?
Who?
How?
Need for coordination and collaboration
Challenges of the CMEF & Ongoing Evaluation
4/5 June 2009

4. Monitoring & Evaluation
For the CMEF (Common Monitoring and Evaluation Framework) relates only to rural development.
For there will be one monitoring and evaluation framemork for the whole CAP (Art. 110 of CAP Horizontal Regulation proposal).
Important components will be (among others) indicators and common EQ

5. Legal framework for the evaluation
RDR, Art.75: Monitoring and evaluation system aims at:
Improving the quality of the design and the implementation programmes;
Demonstrating the progress and achievements of rural development policy;
Assessing the impact, effectiveness, efficiency and relevance of the RD policy interventions.
Objectives of monitoring and evaluation of the CAP

6. Technical Handbook of the M&E framework of the CAP
Provides information on:
Objectives and purpose of M&E,
Framework for M&E of the CAP (intervention logic and indicators),
Actors and responsibilities in M&E (incl. Expert group of M&E the CAP),
Data sources,
Rural development specificities for monitoring (MC, AIR, PF) and evaluation (EP, ex ante, CEQ, ex post), including links to guidance documents already published,
Use of M&E information,
Future development,
Annexes: pillar I and II indicators fiches.

7. CMES as part of CMEF
CAP Common Monitoring and Evaluation Framework (1306/2013, Art. 110)
Common Monitoring and Evaluation System
(1305/2013, Art. 67 and 808/2014. Art.14)
Pillar I specific elements
Common elements
Pillar II specific elements
Common context
Common context indicators
Common evaluation
questions
RD priorities/specific
objectives
Pillar II measures
Pillar I specific objectives
Pillar I instruments
Intervention logic:
Overall CAP policy objectives
Common impacts
Pillar II result /target
indicators
Pillar II output indicators
Performance framework
Common impact indicators
Pillar I result indicators
Pillar I output indicators
Data sources
Operation database
Responsibility
for evaluation:
DG Agri
Responsibility
for evaluation:
MAs

8. Elements of the CMES (Reg.808/2014, Art. 14.1)
Intervention logic showing interactions between priorities, focus areas and measures,
Common context (including RD related impact indicators), result/target, output indicators a performance review indicators,
Common evaluation questions,
Fiches for common context, impact, result/target and output
indicator
Guidance on use and establishment of proxy indicators
Guidance on the Indicator Plan
Guidance on performance review and reserve
CEQ related to:
→focus areas of RD priorities
→other aspect of RDP (RDP synergies and TA)
→Union level objectives

9. Elements of the CMES (Reg.808/2014, Art. 14.1)
Data collection, storage and transmission
Reporting on monitoring and evaluation activities
Evaluation plan
Ex ante, AIR and 2019 and ex post evaluation
Support to actors in M&E to fulfil their obligations
Common data sources: FADN, Eurostat, others
(updated tables for CCI), II, RI.
Operations database and electronic transmission of
monitoring data (OI)
Guidance on monitoring covering elements included
in the AIR
Guidance on the Evaluation Plan preparation
and implementation
Guidance on the ex ante evaluation of RDP
Other guidance documents

10. Intervention logic
Starting point for evaluation (ex ante, evaluation plan for evaluation during the programming period, ex post)
Composed of hierarchy of objectives (CAP, RD priority, focus areas, measures, operations)
Linked to financial allocations (inputs)
Linked to hierarchy of indicators

11. Pillar I Intervention logic

12. Pillar II Intervention logic

13. 1. Fostering knowledge transfer and innovation in agriculture, forestry and rural areas
1A Fostering innovation, cooperation, and the development of the knowledge base in rural areas
1B Strengthening the links between agriculture, food production and forestry and research and innovation, including for the purpose of improved environmental management and performance
1C Fostering lifelong learning and vocational training in the agricultural and forestry sectors
RD Focus areas
Relevant measures
Art. 16 Advisory services, farm management and farm relief services
Art. 15 Knowledge transfer and information actions
Art. 36 Co-operation 13

14. RDP Intervention Logic
An Intervention logic will be drawn up for each RD priority showing their contributions to selected Focus Areas (including potential contribution of particular measure to several focus areas).
A basic intervention logic is proposed by the EC covering the most commonly expected combinations ( see following slides)
MAs have the flexibility to develop a specific intervention logic appropriate to their territory and its needs.

15. Ex ante evaluation (Art. 8.1 (a) and Art.77)
Evaluation plan (Art. 8.1 (a) )
Common evaluation questions

16. Evaluation Questions (Pillar II)
Common Evaluation Questions: Horizontal and focus area evaluation questions:
define the focus of evaluations in relation to policy objectives, and;
help to demonstrate the progress, impact, achievements, effectiveness, efficiency and relevance of rural development policy.
Programme-specific evaluation questions
Policy objectives
Evaluation questions
Indicators

17. Common evaluation questions – 30
Focus area related – 18 questions
Other aspects of RDP:
Operational performance - 1
Technical assistance - 1
National rural networks - 1
Horizontal evaluation questions EU
CAP objectives - 4
Challenges of the CMEF & Ongoing Evaluation
4/5 June 2009

18. “Horizontal” Evaluation Questions
Linked to the overall policy objectives and other aspects of RDPs in order to demonstrate the achievements
EU 2020 objectives
CAP objectives
RD cross-cutting priorities (environment, CC, innovation)
National Rural Networks
Technical Assistance
Operational performance (synergies)

19. “Horizontal” Evaluation Questions
Capture the contribution of the programme towards the overall policy objectives in terms of impacts.
Answered with the means of common impact indicators, common context indicators and complementary result indicators (and additional information when necessary)
Reporting in the AIR in 2019 and in the ex post evaluation.

20. Horizontal Evaluation Questions Example
CAP objective
Fostering the competitiveness of agriculture
To what extent has the RDP contributed to the CAP objective of fostering the competitiveness of agriculture?
JUDGEMENT CRITERIA
The agricultural entrepreneurial income has increased
The agricultural factor income has increased
Agricultural productivity has increased
COMMON RD INDICATOR
Agricultural entrepreneurial income
Agricultural factor income
Agricultural productivity

21. FA-related Evaluation Questions
Linked to the objectives of the Focus Areas in order to demonstrate the achievements towards the policy objectives
Capture the contribution of the interventions under each FA in terms of programme results
Answered with the means of result indicators (and additional information when necessary)
Reporting in the AIRs in 2017 and 2019 and in the ex post evaluation

22. FA-related Evaluation Questions Example
Focus Area 4c: Preventing soil erosion and improving soil management
To what extent have RDP interventions supported the prevention of soil erosion and improvement of soil management?
JUDGEMENT CRITERIA
Soil erosion has been prevented
Soil management has improved
COMMON RD INDICATOR
% of Agricultural land under management contracts to improve soil management (Result indicator)
% of forestry land under management contracts to improve soil management (Result indicator)
ADDITIONAL INFORMATION
Additional information on soil erosion of the land under management contracts

23. Common indicators Programme-specific indicators (RD)
Impact indicators (16) refer to benefits both
at the level of measure or RDP/policy
Result indicators – PI (15)
and PII complementary results (6)
measure direct and immediate effects of RDP/policy
Output indicators – PI (22) and PII (27) measure
activities implemented within RDP/policy
Financial (input) indicators refer to resources
allocated to measures
Common context indicators (45) refer to
RDP context and include impact indicators

24. Context indicators
The Common Context Indicators (+ common impact indicators) of : compulsory minimal set for the SWOT analysis, the RDP strategy and assessment of impacts
45 common context indicators
13 of 16 common impact indicators
Socio economic; Sectorial; Environmental
Used in territorial description and SWOT, evaluations
Included in RDP via structured table (all required!) - the use of the common context indicators will be taken into account for the approval of RDPs by the EC
Base on the availble data, sources from EUROSTAT and other EU-level data sources, OR from national/regional sources. (Proxies may be used at regional level)

25. Context (Pillar II) and impact indicators
EU price variability
Agriculture trade balance
Consumer price evolution
of food products
Challenges of the CMEF & Ongoing Evaluation
4/5 June 2009

26. Impact indicators Cover both Pillar 1 and Pillar 2
Some are more relevant for Pillar 1 (e.g. trade related)
Some are more relevant for Pillar 2 (e.g. territorial development)
As far as possible existing datasets (EUROSTAT, Farm Structure Survey, FADN etc.) available at EU, national and/or regional level will be used for quantification.

27. Results indicators 25 result indicators
Show direct achievements of policy
Linked to Focus Areas of P2-5
Mix of:
targets (19) - monitoring
complementary result indicators (6) - evaluation
Requirement to flag operations contributing to Focus Areas with complementary result indicators
Challenges of the CMEF & Ongoing Evaluation
4/5 June 2009

28. Target indicators
At least one quantifiable target indicator is required for each Focus Area.
Mostly based on result indicators (19),
Some are closer to output indicators – monitoring (e.g. Priority 1).
Show direct achievements of policy
Linked to Focus Areas of P2-5
The target indicators will be reported on annually in the AIRs using:
Direct monitoring data,
Estimates based on coefficients supplied in the guidance (e.g. to estimate the production of renewable energy from new investments).
In few cases the calculation of the target indicators will be conducted by the evaluator when assessing the achievements of RDP (e.g. water and energy savings)

29. Output indicators 26 indicators at measure level:
Output indicators - for all measures (total public expenditures)
Output indicators linked to several measures (area under…)
Output indicators linked to specific measures (support for Leader start up)
Only a selection in the Indicator plan (Planned Outputs to be quantified)
New indicators for NRN, Risk Management….

30. Programme Specific Indicators
(Article 50(2) 1303/2013; draft RD IA Annex I Part 1 Point 4(a)(i))
Design in SMART way,
Shall:
provide additional info specific to RDP territory e.g.
Definition of rural area
Forestry sector
describe issues where common data lacking e.g.
Innovation
Short supply chains
Local markets
support and justify particular interventions e.g.
Thematic sub-programmes
Programme specific Focus Areas

31. Methodological approach (PII)
Setting up the
M&E system
- Evaluation Plan
- Governance
- ToR
- Prepare EQ and
indicators
Preparing and
Structuring
- Intervention logic
- Establish
methodology
- Identify indicators
Observing
- Collection
of data
and qualitative
information
Analysing
- Process and
synthetise data
and information
- Calculating
Net effects
Judging
- Conduct
Assessment
of impacts
- Answer EQ
- Conc&Rec
Reporting
and
disseminating

32. “Standard” AIR – every year from 2016
Reporting
“Standard” AIR – every year from 2016
Enhanced AIR 2017
Enhanced AIR 2019

33. “Standard” AIR – every year from 2016
(Art /2013 and /2013)
Key information on implementation of the programme and its priorities (Financial and monitoring)
Issues which affect the performance of the programme and the measures taken
Steps taken to implement technical assistance and programme publicity requirements
Actions taken to fulfil ex ante conditionalities (in 2017 and in where relevant)
+ Annex template on implementation of the financial instruments

34. Content of "standard" AIR (reporting on evaluation activities)
EP modifications
Evaluation activities undertaken
Activities related to provision and management of data
List of completed evaluations
Summary of completed evaluations
Communication activities
Follow-up of evaluation results

35. Enhanced AIR 2017 Information from evaluation activities on:
Reporting and quantification of programme achievements, in particular through assessment of the complementary result indicators, and relevant evaluation questions.
Description of implementation of sub-programmes
Implementation of actions to take into account the principles set out in art 6, 7 and 8 CPR (Art. 50 CPR)
a) Promotion of equality between men and women and non- discrimination
b) Sustainable development (Art. 8 CPR)
c) The role of the partners referred to in Article 5 CPR in the implementation of the programme and preparation of the progress report

36. Enhanced AIR 2019 Information from evaluation activities on:
Reporting and quantification of programme achievements, in particular through assessment of the complementary result indicators, and relevant evaluation questions.
Reporting on interim impact of the RDP: contribution to programme and EU strategy and objectives, in particular through assessment of the programme's net contribution to changes in CAP impact indicator values, and relevant evaluation questions.
Progress made in ensuring integrated approach

37. Thank you for your attention!