1. Indicators, Monitoring, Evaluation

2. Project Cycle Management
Basic PCM Training Course

3. The Logframe Matrix Project Description Verifiable Indicators
Sources of Verification
Assumptions
Overall objective
Project purpose
Outputs
Activities
means
costs
Pre-conditions

4. Levels of the project description
Overall objective
Relate to the wider sectoral or national objective, to which the project will contribute;
Project purpose
The sustainable flow of benefits to the project’s target groups(s);
Results
The goods and services delivered by the project;
Activities
What the project does to produce the results (work program/tasks). 15

5. Indicators – What are they?
An indicator is a unit of measurement
E.g. kilometres of road, population numbers
An indicator tells us something about changes in a system
e.g. Financial indicators (stock markets up / down)
An indicator helps to quantify and simplify phenomena and helps us understand complex realities
E.g. Blue Flag for bathing water quality
An Indicator is not a Target!

6. Characteristics of a good indicator
The Objectively Verifiable Indicators (OVI) should be:
Specific : including (where appropriate) clear targets;
Measurable: in terms of quality and/or quantity;
Available : at an acceptable cost;
Relevant : to what they are supposed to measure (horizontal logic);
Timely : produced in due time to be useful for project management.

7. Sources of Verification (I)
The logical framework requires that one spells out how the indicators will be collected :
How/through which methods ?
By whom?
When/how often?
Admin records
Activities, resources and costs required to collect this information must be included in the project design
Project managers should focus on verifying the delivery of results and their contribution to the purpose

8. Sources of Verification (II)
Specialised
surveys
Cost
Interviews with
beneficiaries
Adapted available
statistics
Available
statistics
Management
reports
Administrative
reports
Complexity 20

9. Describing Indicators
Indicators help to define: 
Q Quality
Q Quantity
T Time
P Place

10. Defining an Objective:
Define objective: Educational status improved
Specify quality: Number of graduates completing degree courses increased
Specify quantity: Number of graduates completing degree courses increased from 500 to 1000
Specify timeframe: Number of graduates completing degree courses increased from 500 to 1000 per year by the year 2007
Specify Place: Number of graduates from Tartu Technical College completing degree courses increased from 500 to 1000 per year by the year 2007

11. Indicators, Projects and Programmes
 Global objectives are expressed in terms of impacts (e.g. a reduction in unemployment among the previously long term unemployed).
 Specific objectives are expressed in terms of results (e.g. the improvement, through training, of the employability of the long term unemployed);
Operational objectives are expressed in terms of outputs (e.g. the provision of training courses to the long-term unemployed);

12. Note! Indicators require good data, based on monitoring
Indicators are not targets!
Indicators can help to reduce uncertainty, but not eliminate it
Sets of indicators evolve over time
Sets of indicators are seldom, if ever, complete
Indicators can help influence behaviour.
Indicators must be able to take into account different locations, people, cultures and institutions
Different people from different places have different values

13. Criteria for Indicator Selection
For Policy relevance and utility for users
an indicator :
• should provide a representative picture
should be simple, easy to interpret and able to show trends over time;
must be responsive to changes
can provide a basis for international, regional comparisons;
must have a comparative threshold or reference value/Baseline against to assess the significance of the values.

14. Criteria for Indicator Selection
Analytical soundness
be theoretically well founded in technical and scientific terms;
be based on international standards and international consensus about its validity;
lend itself to being linked to economic models, forecasting and information systems.
Measurability
The data required to support the indicator should be:
readily available or made available at a reasonable cost/benefit ratio;
adequately documented and of known quality;
updated at regular intervals in accordance with reliable procedures.

15. Setting up a system of indicators
Baseline
Context Indicators
Monitoring Indicators
Example: Scottish Executive ESF & ERDF Indicators
COM WP No 3 Section D, p. 11

16. Indicators for ESF Projects - Scotland
Project Level Core Indicators
Number of beneficiaries of ESF assistance, show separately:
% of persons aged receiving assistance
% of persons in this group unemployed for less than 6 months before assistance began
% of persons aged 25 and over receiving assistance
% of this group unemployed for less than 12 months before assistance began
% of those who complete their course
% of those leaving ESF funded training for positive outcomes, split by end destination:
into full or part-time employment
self employment
full time FE/training/other Government training schemes
% of leavers who gain a qualification or part qualification
% of beneficiaries who receive assistance specifically geared towards self employment
Number of existing companies given direct assistance from ESF
% of parents with children under 5, in employment 6 months after ESF assistance

17. Indicators for ERDF projects - Scotland
Projects in Scotland; Project Level Core Indicators
Indicator Name
Assistance to Business
Number of businesses assisted and instances of assistance, showing SMEs separately.
Increase in sales in assisted businesses.
Hectares of serviced land created or enhanced, split by greenfield / brownfield sites.
M2 of business space created or enhanced - occupied after 18 mths/3 yrs.
Private sector finance levered in by new assisted projects.
Value of investment in R&D by assisted SMEs.
Number of patents/intellectual property rights registrations by assisted SMEs.
Number of organisations taking up e-commerce trading.
Increase in visitor numbers.
Number and M2 of new training/learning facilities constructed or upgraded.

18. Indicators ERDF Projects - Scotland
Jobs and Employment
Gross new jobs created.
Number of jobless people securing employment.
Number of young people under 25 years of age placed in employment.
Number of gross jobs created through self-employment.
Gross number of jobs safeguarded.
  Social Inclusion and Equal Opportunities
Gross number of assisted organisations introducing active equal opportunity policies.
Number of childcare facilities and places created.
Number of organisations achieving recognised quality awards.
 Environment
K/W of new renewable energy infrastructure installed.
Number of SMEs undertaking environmental management.
Number of strategic environmental/forestry partnerships funded.
Hectares of natural habitat under management.
Increase in volume of waste recycled or reused.

19. Indicators Progress Chart
Scottish Executive Environment Group
Indicators of Sustainable Development for
Scotland:
Progress Report 2004
February Paper 2004/3

20. Headline Indicators National Assessment UK
Regional Quality of Life Counts – 2002

22. Monitoring & Evaluation

23. Some Principles of M&E
Keep the needs of the information user clearly in mind. Who is this information for? How will they use it?
More information is not better information
A balance of quantitative and qualitative information is usually best
Collecting and using information requires time and resources – allow for this in the project design

24. Monitoring vs Evaluation
Who?
Internal management responsibility – all levels
Usually incorporates external inputs (objectivity)
When?
Ongoing
Periodic – mid-term, completion, ex-post
Why?
Check progress, take remedial action, update plans
Learn broad lessons applicable to other programs/projects, policy review, etc
Focus on
Inputs, activities, outputs
Results, purpose, overall objective

25. Definition of Monitoring
A regular reporting mechanism, preparing and submitting analytical assessment to project management and other involved parties
The objective is to enable focused management decisions in order to allow adjustments and to ensure that the projects remains on course and reach their objectives.
Monitoring enables management to identify and assess potential problems and success of a program or project. It provides the basis of corrective actions, both substantive and operational, to improve the program or project design, manner of implementation and quality of results.
In addition, it enables the reinforcement of initial positive results.
Monitoring records and reports inputs, activities, outputs & problems (project monitoring) or even measures the extent to which expected project benefits are being achieved (performance monitoring).
The monitoring system must provide project managers with complete and accurate project information. Data must be presented in a comprehensive and useful way and reports should be distributed to all stakeholders at the same time, in order to avoid delays in information sharing.

26. Monitoring of Projects & Programmes
Monitoring needs a formal structure which sets out:
What is to be monitored?
How will it be done?
Who will do it?
When will it be carried out?
Where and how is information to be recorded?
Baselines data and project Indicators

27. Why Monitor ? To keep track of where you are
To identify any significant changes in the project
It assists in communications
It is good management practice!
It is, in some cases, an EU requirement
To help you keep track of where you are with a project in relation to where you planned to be. Hence the benefit of the logframe in identifying milestones. What outputs do we anticipate in relation to inputs and when are they expected?
In Module 5 we looked at the Gantt chart which sets out the time line.
Need to be aware of the critical path. Delays off the critical path are not initially project threatening – delays on the critical path definitely are!

28. Indicators & Project Responsibilities

29. Project Monitoring and Indicators

30. Activity Planning

31. Using Activity Plan to Monitor

32. What is to be monitored 1? Examples in a Project Beneficiary details
Staffing details
Project activity progress / milestones / deadlines vs plan
Quality checks
Attendance records
Finance – income
Finance – costs vs budget

33. What is to be monitored 2? Examples in a Programme Outputs Results
No of people trained
No. courses completed / % passing approved test
No of people on study tours / placements
Results
Higher quality of project fiches
Improved contracting rate
Impact
Improved absorption of EU funds

34. Data needed Inputs Monitoring of the project or programme
Outputs Monitoring of the project or programme
Results Evaluation of the project or programme
Impact Evaluation of the project or programme

35. Identifying Indicators for Monitoring
How to measure what we get is what we ordered and pay for?
Project duration
Deliverables
Activity plan /
milestones / phases
Activities
Reporting incl handover certificates etc
Indicators and sources of verification
Baseline indicators
Performance indicators for managing the project
Monitoring
Staff
Recording
Reporting
Communicating

36. Practical exercise to identify the information required to monitor a project
Case study.

37. Monitoring data exercise
 A project is working directly with young people with a disability to improve confidence, assertiveness and presentation. Each young person will have a mentor – an older person with a disability who is successful in their chosen employment. At the same time the project is raising awareness with employers of the value of employing young people with a disability. The ultimate aim is for the young people to get jobs in the open labour market. The project is funded by the ESF and in order to be eligible all the beneficiaries must be aged between 16 and 24, be unemployed and have a disability.
What information would be required for monitoring?

38. Evaluation

39. Definition of Evaluation?
A time-bound assessment that systematically and objectively assesses the relevance, performance, success and effectiveness of ongoing and completed programmes and projects.
The primary purpose of evaluation is to provide lessons that are incorporated into the decision-making processes of new projects and programs.
Emphasise the ‘Time’ related nature of evaluation, I.e. the time continuum of
Inputs activities outputs results impact
An evaluation is a systematic review of project performance by monitoring the progress of the project towards meeting its purpose through analyzing its
Relevance
Efficiency
Effectiveness
Impact and
Sustainability
These are covered in the next slides

40. Why Evaluate? Reporting of results to: Funders Project promoter
Local partners
Peer organisations
Transnational audiences
Beneficiaries
Who is the evaluation for?
Different audiences will have different expectations. An evaluation designed to meet the needs of the promoting organisation’s own management and staff (perhaps to find out what went wrong!) will be very different to one designed to serve as a promotional tool.
On the other hand, a good evaluation, carried out and written with integrity, should be multi functional and is likely to be of value both internally and externally – although it will probably need to be adapted to different audiences. For example:
Funders: Focus on value for money. Evaluation of costs against measurable contribution to policy objectives or programme objectives
Promoter: Identification of new market opportunities or unmet needs.
Identification of operational or administrative problems as well as recording and validating the project's achievements.
Local partners: Interested in the benefits to the local economy and to their own strategic objectives
Peer organisations: Interested in the possibility of duplicating or adapting to their needs. Looking for relevance to the needs of their clients and an honest account of its costs and difficulties
Transnational audiences (donor funds?): looking at impact and benefit to transnational activities (INTERREG?). Suitability of promoting organisation as a future partner?
Beneficiary group: Interested in the project’s impact on individual people or organisations in order to assess potential benefits to themselves

41. Project Evaluation
Project evaluation is a structured process comprising:
Collection of relevant information
Assessment of this information against agreed criteria
Reporting of the evaluation results
Incorporation of these results into future planning

42. ***** **** * *** ** Project Evaluation Gathering information Method
Quantitative
Qualitative
Record keeping
*****
Participant observation
**** *
Self administered questionnaires
*** **
Individual interviews
Group discussion
Method
Advantages
Disadvantages
Record keeping
Structured and easily available
Produces data which can be easily analysed
Feelings, impressions, ideas, may not be picked up
Data may be insufficient to answer key questions.
Participant observation
Examines actual behaviour rather than subjects’ views about their behaviour
Requires high level of observer skill
Danger of observer bias
Observation may affect subject’s behaviour (Hawthorne effect?)
Self administered questionnaires
Cheaper than interviews allowing for large sample
Easy to analyse if properly designed
Less flexible than interviews
Subjects need literacy skills
May have poor response rate
Individual interviews
Structured questons can yield quantitative data (yes/no or scores) but with the flexibility to probe and clarify
Time consuming
Interviewers must be skilled and acceptable to the subjects
Analysis of notes can be difficult.
Group discussion
May draw out shared ‘negative’ views
Offers a way for those with poor literacy to contribute
Needs a facilitator AND a note taker
A few speakers can dominate
Unsuitable for probing individual views

43. Relationship to the Logframe
Level of Logframe
Type of information
Timing of collection
Measurement concept
Overall Objectives
Substantive development change to specific sector
Project completion and ex-post
Sustainability
Impact
Project purpose
Realistic and sustainable change to target group specific to project intervention
Mid-term and project completion
Relevance
Effectiveness
Results
Quantitative and qualitative measures of physical progress
Quarterly/yearly
Efficiency
Costs
Measures based upon project budget
Weekly/monthly
Economy
Self explanatory and can be related to the log frame work from Module 5 and the SWOT exercise in Module 2

44. Programme Evaluation Ex Ante (Before) Mid Term (Mid way)
Can the programme achieve the results set out in the plan?
Mid Term (Mid way)
How are we doing and what changes are needed?
Ex Post (After)
What did we achieve vs plan?

45. Evaluation and Indicators
Relevance
Efficiency
Effectiveness
Impact
Sustainability
Relevance to objectives and to in-country/beneficiary needs. Does the project make sense in relation these needs?
Particularly important in relation to ex-ante evaluation in the context of judging the appropriateness of the policies or projects.

46. Evaluation and Indicators
Socio-economic problems
Impact
Needs
Results
Programme
Objectives
Inputs
Operations
Outputs
Efficiency
Relevance
Effectiveness
Sustainability
Evaluation

47. One for the road! It is not necessarily a crime to be running late…………
……..It is always a crime not to know!
You can only manage the work that remains to be done!
Getting people to think ahead
is 50% of the benefit of project planning
A successful project is one that where outputs are delivered on time, on budget, at the required quality…….
…….and no one died of a heart attack