1. Tools for evaluating local policies in Europe

2. Policies, programmes, projects…

3. Project - definition
A project is a series of activities aimed at bringing about clearly specified objectives within a defined time-period and with a defined budget.
A project is defined in terms of a hierarchy of objectives (inputs, activities, results, purpose and overall objective) plus a set of defined assumptions and a framework for monitoring and evaluating project achievements (indicators and sources of verification).

4. A project should also have:
Clearly identified stakeholders, including the primary target group and the final beneficiaries;
Clearly defined coordination, management and financing arrangements;
A monitoring and evaluation system (to support performance management);
An appropriate level of financial and economic analysis, which indicates that the project’s benefits will exceed its costs.

5. Project Cycle Management (PCM)

6. Project Cycle Management - management activities and decision-making procedures used during the life-cycle of a project.

7. Three main principles of the process
Decision making criteria and procedures are defined at each phase.
Phases are progressive – each phase should be completed for the next to be succeded.
Results of the process are taken into account in next programming period.

8. Programming the situation is analysed
a review of socio-economic indicators
to identify the main objectives and sector priorities

9. Identification stakeholder analysis
problem analysis, including scoping of crosscutting issues (e.g. gender, governance, environment)
assessment of other planned initiatives, and assessment of lessons learned
preliminary objectives and strategy analysis
preliminary assessment of: resource and cost parameters, management, coordination and financing arrangements, economic/financial, environmental, technical and social sustainability issues

10. Formulation
To transform issues from the „identification” stage in a more detailed way.
To prepare a detailed project design, including the management and coordination arrangements, financing plan, cost-benefit analysis, risk management, monitoring, evaluation and audit arrangements.

11. Implementation - beginning
Make contracting arrangements
Mobilise resources
Establish working relationship with stakeholders
Review and revise project plan

12. Implementation – the main part
Selection and location of resources, including personnel
Implement activities and deliver results
Monitor and review progress
Revise operational plans in light of experience
Report on progress

13. Implementation - phasing out
Pass all responsibilities to local partners
Ensure maintenance plans are in place
Ensure relevant skills are effectively transferred

14. Evaluation according to Project Cycle Management (PCM)

15. Evaluation
assessment, as systematic and objective as possible, of an ongoing or completed project, programme or policy, its design, implementation and results.

16. Principles of evaluation
Impartiality and independence from the programming and implementation functions;
Credibility - use of appropriately skilled and independent experts and the transparency of the evaluation process, including wide dissemination of results;
Participation of stakeholders to ensure different perspectives and views are taken into account;
Usefulness of the evaluation findings and recommendations, through timely presentation of relevant, clear and concise information to decision makers.

17. Purposes of evaluation
Planning/efficiency - ensuring a justification for a policy /programme/project and that resources are efficiently used.
Accountability - how far a programme/project has achieved its objectives, how well it has used its resources and what has been its impact.
Implementation - improving the performance of programmes/projects and the effectiveness of how they are delivered and managed.
Knowledge production – understanding what works (for whom) and why (and in what contexts).
Institutional strengthening - improving and developing capacity among programme participants and their networks and institutions.

18. Evaluation Criteria used by the EC (1)
Relevance: appropriateness of project objectives to the problems that it was supposed to address, and to the physical and policy environment. Quality of project preparation design.
Efficiency: have the project results been achieved at reasonable cost? in terms of quality, quantity and time, and the quality of the results achieved.
Comparing alternative approaches to achieving the same results, to see whether the most efficient process has been adopted.

19. Evaluation Criteria used by the EC (2)
Effectiveness: how results contibuted to the Project Purpose, and how Assumptions have affected project achievements? Specific assessment of the benefits to target groups (women/men, vulnerable groups: children, the elderly, disabled).
Impact: The effect of the project on its wider environment, and its contribution to the wider policy or sector objectives.
Sustainability: likelihood of benefits produced by the project to continue to flow after external funding has ended.

20. Evaluation report should
mirror the structure of the main evaluation criteria,
take into account the nature of the project, the stage at which the evaluation is carried out, and the users for whom the report is prepared.

21. Decision options – time and objectives
continue project implementation as planned, to re-orient/restructure the project, or, in the worst case, to stop the project (mid-term evaluation);
modify the design of future projects or programmes in light of lessons learned (ex-post evaluation);
to modify policies, co-operation strategies, and subsequent programming or identification exercises - in the case of sector, thematic or cross-sector evaluations.

22. Choosing methods and techniques
All methods and techniques have both strengths and weaknesses; often they are used in circumstances that are far from ideal for their application. They should be chosen and applied in a manner that exploits their virtues and recognises their limitations.
It is best to apply methods and techniques in combination as part of any particular evaluation assignments. Relying on a single method or technique will be weaker than obtaining multiple perspectives.

23. Methods and techniques (1)
BENEFICIARY SURVEYS
CASE STUDIES
COST BENEFIT ANALYSIS
COST EFFECTIVENESS ANALYSIS
DELPHI SURVEY
EXPERT PANELS
FOCUS GROUPS
IMPACT EVALUATION

24. Methods and techniques (2)
INTERVIEWS
MODELS
MULTI-CRITERIA ANALYSIS
OBSERVATION TECHNIQUES
PRIORITY EVALUATION METHOD
REGRESSION ANALYSIS
SWOT ANALYSIS

25. What about data? (1)
Data are never pure or naturally occurring, they need to be produced. Evaluators need to know from where their data comes and what decisions have made in the course of their production. The strength of the arguments and conclusions depend on the strengths and characteristics of the data being used.
Different partners have to be willing to share information and all stakeholders need to be convinced that they are going to get something out of an evaluation before they give access with any enthusiasm to any information they hold.

26. What about data? (2)
Quantitative data to provide overviews (a comparative perspective), and qualitative data able to capture subtleties, people's experience and judgements.
Indicators and monitoring systems are a powerful adjunct to evaluation. Often evaluators depend on monitoring systems which are indicator based.
Over-elaborate indicator systems will be counterproductive. While there is a temptation to measure everything, this should be resisted. This can be costly and the results useless.
Indicators do not tell you everything about performance. Often they are a prompt to understand why and how something is happening. This is the role of evaluation.

27. Case studies
in-depth study of a phenomenon in a natural setting, drawing on a multitude of perspectives.
multiple data collection methods (both qualitative and quantitative), or derive from multiple accounts of different actors in the setting.
individuals, programmes, organisations, projects, groups of people or decision-making processes
where there is more than a single focus or unit of analysis.

28. Case studies – purposes
Illustration
exploration
critical analysis
analysis of implementation
analysis of the impacts.

29. Case studies – data
multiple data sources, such as interviews, observations over time, statistics, physical information, etc.
data cross-checked in order to ensure its coherence.
The notion of "context" - the impacts of a specific project on the beneficiaries are influenced by a large number of external factors.

30. Case studies – main steps
Selection of cases to study
Data collection and process
Case report
Account

31. Strengths and limitations
a view of processes and complexities
generalisation?
costs – number of observations
credibility – doubts if not implemented correctly

32. Student task (1)
find an example of a project co-financed by the EU funds in your neighbourhood (implemented by public or private entity or NGO institution),
Describe the situation on the local level regarding to this aspect.
Detailed description of the project: what kind of fund, program, time, total value, participation of the EU, etc.

33. Student task (2)
Contact directly a person taking part in the project, or the project originator, or a person benefiting from the project effects and find out:
what are origins of the project?
did the whole society accepted the project idea or not?
are the local users of the project effects or the local society aware of the fact that it was co-financed by the EU funds?
were there any problems during the project implementation?
are the project effects consistent with those expected in the beginning?
overall assessment: was the project worth of doing? Any changes in the future?
any plans for the future? Is further development possible?

34. Regional policy in micro perspective
Regional policy in micro perspective

35. References
EuropeAID. (2004). Aid Delivery Options Project Cycle Management (PCM) - Project Approach Guidelines, Project Cycle Management Guidelines (Vol. 1, pp. 158). Brussels: EuropeAid Cooperation Office, European Commission.
EC 2013: Evalsed Sourcebook: Methods and Techniques
EC 2013: EVALSED: The resource for the evaluation of Socio-Economic Development.