1. Monitoring and Evaluation

2. Session Objective
To increase participants understanding of the concepts used in designing M&E Frameworks and Plans
To build participants competence in designing Program M&E Plans

3. Expected Results
At the end of the sessions participants will know about Program frameworks, M&E frameworks and the difference between the two frameworks
Participants will be able to identify and select appropriate indicators for a program.
Participants will be able to produce a program monitoring and evaluation framework.

4. Key Terms for Monitoring and Evaluation

5. Objectives Explain the difference between monitoring and evaluation.
Introduce the most common M&E terms.
Review examples of each term.
In this session, we will cover the following:
Read session objectives.

6. Monitoring versus Evaluation
Data collected on program activities
Ongoing, routine
Focus on activities and output, compared to target
Are we doing the work we planned?
Evaluation
Data collected to answer specific questions
Periodic
Focus on outcome, impact
How effective were our activities?
THIS SLIDE IS ANIMATED
Let’s begin by comparing monitoring and evaluation.
CLICK to display Monitoring.
Monitoring is the routine collection and analysis of program data, specifically on program activities. Data collection is ongoing (weekly, monthly, quarterly, semiannually, etc.).
CLICK to display question.
Monitoring compares results to the original targets to answer the question: Are we doing the work we planned? It can alert us to problems early because we continually review program data.
CLICK to display Evaluation.
Evaluation answers specific questions about our performance. It is in-depth analysis that tells us if we should continue or improve our activities.
Evaluation is less frequent and answers the question: How effective were our activities? Sometimes monitoring leads us to an evaluation. We may see a trend in the routine data that we cannot explain. So we could design an evaluation to help us understand what is going on.

7. Monitoring or Evaluation?
Local researchers conduct a study to determine if there are more people with possible TB symptoms coming to clinics as a result of a media campaign to promote TB screening.
Evaluation
THIS SLIDE IS ANIMATED
Read example and ask participants if this is monitoring or evaluation.
CLICK to reveal answer.

8. Monitoring or Evaluation?
A district manager reports on how many nurses were trained on interpersonal communication skills for her quarterly donor report.
Monitoring
THIS SLIDE IS ANIMATED
Read example and ask participants if this is monitoring or evaluation.
CLICK to reveal answer.

9. It Starts with QUESTIONS
Monitoring and evaluation answer different questions.
If we do not ask good questions about our activities, we will not get useful data!
Our monitoring and evaluation efforts are only as good as the questions we need to answer. Let’s look again at the questions we wrote on the flipchart.
Point to each question and ask participants to identify the question as a monitoring or an evaluation question.
We can see that there are endless questions we could ask about our activities. How should we prioritize what we ask? We will talk more about this later in the training. For now, just think about the variety of questions we could ask and the variety of information we could get about our activities.

10. What is a GOAL?
The ultimate result of efforts at a broad, population level.
Achieved over the long term (years) and through combined efforts of multiple programs.
THIS SLIDE IS ANIMATED.
Goals are meant to have a very broad impact across a very broad population. They usually take a very long time to achieve and need several types of activities to do so. It usually takes more than ACSM to achieve, but ACSM certainly contributes. Here are some examples.
CLICK to display examples.
Example 1: This shows a large population.
Example 2: This goal will need several programs to be accomplished.
Example 3: It will take many years to reduce stigma.
Decrease morbidity and mortality due to TB in Country X.
Reduce prevalence of TB by 50%.
Eliminate stigma of TB in our communities.

11. What is an OBJECTIVE? OBJECTIVES GOAL activities
How the results of your short-term program activities contribute to the big goal.
Several objectives can relate to the same goal.
THIS SLIDE IS ANIMATED.
So we have a long-term, broad goal and we also have our short-term, specific activities. They do not seem very connected unless there is an objective to link them together.
CLICK to add picture. CLICK again to display bullets.
The objective is the result of our short-term program activities, and contributes to the larger goal. In other words, the objectives say what we will do and when. The activity is how we will do it and the goal is why we will do it.
You can have several objectives for the same goal or several activities for the same objective.

12. Objective Examples
Aggressively advocate to increase budget by 8% each year for the next four years.
Double the percentage of secondary school students who can correctly identify TB symptoms by 2015.
Design and pilot a treatment support program for newly released prisoners with TB by 2015.
Here are some examples of ACSM objectives.
CLICK to display examples.

13. INPUTS Resources needed to plan and implement XX
“Raw materials” of an XX project
Examples
Money
Staff
Policies, guidelines
Equipment
Partners
THIS SLIDE IS ANIMATED.
Inputs are the raw materials you need to conduct your activities.
CLICK to display examples.
These include items such as money, staff, policies, training materials, etc.
When developing your ACSM and M&E plans, it is often helpful to determine which inputs you already have and which inputs you still need. You may need to do some advocacy to get the inputs you need (such as money or formal permission from the Ministry of Health).

14. ACTIVITIES The work that we do, what we implement
Also called “processes”
Examples
Training events
Meetings
Events
Outreach
Home visits
THIS SLIDE IS ANIMATED.
We all know what activities are. This is the work that we do. Sometimes activities are called “processes.”
CLICK to display examples and read.

15. OUTPUTS Immediate results of activities
What we can measure/count right after the activity
Examples
Number of people trained
Number of brochures produced
Number of policymakers reached with advocacy activity
THIS SLIDE IS ANIMATED.
Outputs are the immediate results of our activities. They are the proof that our activity happened. Often they are expressed as a number or amount. Outputs can be measured or counted right after our activity is complete.
CLICK to display examples and read.
Can anyone give me an example of an output for one of your ACSM activities?
Solicit 1-2 examples.

16. OUTCOMES “Ripple effects” of XX activities
What changes after outputs are produced
Examples
Increased funding for TB after lobbying meeting
Short: Improved attitudes toward TB patients among DOTS nurses after a training
Medium: Increased satisfaction of TB clients
Long: TB clients stay in treatment longer
THIS SLIDE IS ANIMATED.
Outputs lead to outcomes, which are the “ripple effects” of our activities, or what happens as a result of our activities.
CLICK to display examples.
In the first example, what would be the outputs?
Solicit answers. (Meeting held, number of meeting participants.)
Outcomes can be short-, medium-, or long-term effects. In the second example, we can clearly see how outcomes lead to each other over time. Example #2 is a good example of a ripple effect of an interpersonal communication training.
Read example #2 out loud and briefly discuss.

17. INDICATORS
How we define our activities, outputs, or outcomes
Signs or evidence we watch for to see if we have reached them
ACTIVITY: Meeting with Finance Minister and NTP Director to lobby for more funding for NTP
OUTPUT:
Number of officials attending the meeting
INDICATOR:
Number of officials attending the meeting compared to number invited
OUTCOME:
Increased funding
Percentage of NTP budget covered by the Ministry of Health
Indicators are signs or evidence of our activities, outputs, and outcomes. They “indicate” that something has happened. Indicators are what we watch for. For example, if you are yawning at this point in the session, that is an indicator that you are tired or my presentation is boring.
We will talk more about indicators later, but here are examples to introduce the concept.
Review examples.

18. IMPACT More related to goal Very broad-scale result over long term
Examples
Higher rate of treatment success
Reduction in deaths among MDR-TB patients
THIS SLIDE IS ANIMATED.
Impact is what happens over the longer term and is usually related to goals. Impact is measured at a broader scale and requires data beyond the scope of our project.
CLICK to display examples and read.
The difference between outcomes and impact is how confident we are that the changes are a direct result of our activities and not the result of other factors. This is very difficult to prove without expensive research.

19. Introduction…
Write the administrative/engineering/health problem addressed by a major public health program in your country
Write at up to three specific objectives of a engineering/health program that addresses the above mentioned health problem.
Write down two indicators the program mentioned above uses to monitor it progress or performance towards its objectives.

20. The Project cycle

21. Project Cycle Jan Davis and Robert Lambert, Engineering in Emergencies, p.63.

22. What is Program Monitoring, Evaluation?
Monitoring is the routine process of data collection and measurement of progress toward program objectives.
Evaluation is the use of social research methods to systematically investigate a achievement of a program’s results

23. Differences and Links between Monitoring and Evaluation
SLIDE 7
Monitoring
Evaluation
A continuous process
A specific activity or moment
To provide information to day-to-day decision making (adjustments)
To provide recommendations to strategic decision-making processes
It is carried out by the project team
For the project team (to adapt and improve the impacts) and the donors (to follow the progress)
It is carried out by an evaluation team (internal or external to the project team)
For the project team and the donors (lesson learned)
The monitoring system should provide information for evaluations

24. Comparison Between M&E
Item
Monitoring
Evaluation
Frequency
Regular, ongoing
Episodic
Main action
Keeping track/oversight
Assessment
Basic purpose
Improving efficiency
Adjusting work plan
Improve effectiveness, impact, future programming
Focus
Inputs/outputs, process outcomes, work plans
Effectiveness, relevance, efficiency, impact, sustainability
Information sources
Routine systems, field visits, stakeholder meetings, output reports, rapid assessments
Same plus
Surveys (pre-post project)
Special studies
Undertaken by
Project/program managers
Community workers
Supervisors
Community (beneficiaries)
Funders
Other Stakeholders
External evaluators
Adapted from UNICEF, A UNICEF Guide for Monitoring and Evaluation: Making a Difference? New York, 1991, p.3

25. MONITORING
MONITORING is the continuous, systematic and critical review of operations in order to measure their evolution and adjust them according to circumstances and project’s objectives.
Monitoring must be systematic.
Explain the difference between “monitoring” & “assessment” thoroughly.
It is probable that the assessment, the project design, the monitoring & the impact study shall be conducted by different individuals – hence the importance of clearly defining the desired impact, the objective, the monitoring & impact indicators from the onset (project conception – logical framework).
Monitoring with or without indicators.
Monitoring of the project’s progress &/or of its activities’ impact.
Define ahead what one wishes to monitor & why.

26. Building a Monitoring system

27. BUILDING A MONITORING SYSTEM
Intervention objectives

28. What should a M&E System Measure?
The Results Chain
Indicative Example:
Outcomes
Impact
Outputs
Activities
Inputs
Effects on dimensions of well-being
Improve literacy
Access to, use of, and satisfaction with services
School enrollment rates
Goods and services produced
Number of schools built; textbooks, etc.
Tasks undertaken to transform inputs to outputs
Building of schools Distribution of textbooks, etc.
Financial, human and material resources
Spending on primary education
Source: Adapted from ADB (2006) Introduction to Results Management, p. 7
World Bank (2001) PRSP Sourcebook, p. 108.

29. Some examples of outcome indicators
Improved road network
% of roads in good condition; average travel time
Improved public sector performance
% of population satisfied with public services; tax collection (% of GDP)
Improved health services
Utilization rate of healthcare centers
Increased tourism
Number of tourists visiting the region
Source: ADB Practice Note on Results-based Country Strategies and Programs, Annex 2.

30. Key Questions What is the purpose of carrying out M&E
Who needs, uses M&E Information
Who carries out M&E?
How is M&E carried out?
When should M&E be carried out?

31. Improve program implementation
What is the purpose….?
Improve program implementation
Data on program progress and implementation
Improve program management and decision making
Inform future programming
Inform stakeholders
Accountability (donors, beneficiaries)
Advocacy

32. Who needs, uses M&E Information?
Managers
Donors
Governments
Technocrats
Communities
Beneficiaries
To Improve program implementation…
To Inform and improve future programs
Inform stakeholders

33. Who conducts M&E….? Program implementer Stakeholders Beneficiary
Remember ..
M&E Technical skills
Participatory process

34. How to carry out M&E…? Key Features
Program Framework: Analyze and systematically lay out program elements
Identify key elements to monitor and evaluate.
Determine and describe the measures to be used for monitoring and evaluation
Develop M&E Framework and action plans, including data collection and analysis, reporting and dissemination of findings.
This morning, we began discussing #1, we are now going to talk about #2, and this afternoon Linda will discuss #3.

35. What do you know about your program….?
Program Framework
What do you know about your program….?

36. Program Framework do ……………..achieve!
Systematic lay out of the program elements and path showing what the program plans to:
do ……………..achieve!

37. Population, system level factors that cause the public health problem
Program Framework
Based on a theoretical, empirical model, or general understanding
Public health Problem
Population, system level factors that cause the public health problem
Action/interventions that can change the factors and ultimately alleviate/eliminate the problem

38. Improved Health Status
Results Framework
Improved Health Status
Impact:
Improved (Sustained) Use of Key Health Services and Practices/Behaviors
Strategic Objective:
Increased quality of…
Increased availability/ access to…
Improved social / policy environment…
Intermediate Result:
The defined SO, IRs, and strategies should finally be filled in on this framework.
Strategies:
Strategies:
Strategies:
Strategies (Sub IR):

39. Example: Result Framework for a Family Planning Project
Example: Result Framework for a Family Planning Project
GOAL: REDUCED FERTILITY
SO: Increased FP use and improved FP/RH practices
Increased knowledge of, improved attitudes toward, and acceptance of key services and behavior
Increased quality of FP counseling and services for
Increased availability/access to FP/RH
Improved social and policy environment for FP
Strategies:
 Increase availability of educational materials at clinic and community level
Community mobilization (using PRA and PDI) including men
Implement mass media strategy
Mobilize opinion leaders at national and local level
Design/ implement supportive supervision System
Train service providers (in-service in FP counseling and management of side effects)
Remodel clinic to allow for privacy
Design and implement quality improvement program
Strengthen logistics management
Mobilize private sector providers
Mobilize CHWs/CBDs
Encourage socially marketed pills
Advocate for community based distribution of pills
Promote addition of Depo injections to EPI outreach strategy
Pilot social marketing of pills

40. Basic Logic Model

43. Case 1: To decrease maternal mortality, a 10-year program plan to improve to train midwifes to Delivery and ANC services at health facilities, and to train and deploy CHWs to increase the community’s awareness about, and use of the improved services at the health facilities.
Case 2: To reduce high fertility, a 5-year program plans to work with the Government to change policies in order to allow and promote use of modern family planning methods, train family planning providers to provide better FP services, and to launch public campaigns that promote family planning methods.
Case 3: To reduce HIV infection among adolescents, a five-year program plans to implement income generation activities for the youth, provide and promote universal secondary education, and build adolescent-friendly reproductive health service delivery points.

44. Exercise
Identify and state is the Public Health problem implied in the case study.
What are population level factors will the program target to change in order to alleviate the public health problem
Prepare a Program Framework for the scase study

45. Monitoring and Evaluations Framework

46. M&E Questions Monitoring questions What is being done? By whom?
Target population?
When?
How much?
How often?
Additional outputs?
Resources used? (Staff, funds, materials, etc.)

47. M&E Questions Evaluation Questions?
Is the content of the intervention or the activity being delivered as planned?
Does the content of the intervention or the activity reflect the requisite standards?
Have the intervention achieved the expected results?

48. What do we need to answer these questions…
What do we need to answer these questions…? INDICATORS …to take measurements.

49. Indicators: Definition
Markers that help to measure change by showing progress towards meeting objectives
Observable, measurable, and agreed upon as valid markers of a less well-defined concept or objective
Indicators differ from objectives in that they address specific criteria that will be used to judge the success of the project or program.
Indicators are more specifically defined than objectives, since they define the program attributes with a focus on expected effects translated into specific measures providing the basis collecting valid and reliable information for evaluation. ( CDC ‘Framework for Evaluation’ pg 15). Indicators spell out & provide guidance for periodic monitoring and non-periodic assessment of higher level outcomes (results). They are tools that tell the story of program progress and success.
For example: increases in the contraceptive prevalence rate is an indicator for the increased use of family planning in a country.
Other examples include:
Indicators for measuring program activities such as program capacity to delivery services, participation rate, levels of client satisfaction, or amount of intervention exposure.
Indicators for measuring program effects, such as changes in a particular behavior, community norms, or health status.
See comment for examples

50. Type and Level of Each Indicator
Input/Process (Monitoring)
Outcome / Impact (Evaluation)
Level
Global level
Country level
Program level

51. Exercise: Group work
Use your case study and identify at least two indicators for program monitoring and two indicators for program evaluation.

52. What Is a Good Indicator?
Valid: Measures the effect it is supposed to measure
Reliable: Gives same result if measured in the same way
Precise: Is operationally defined so people are clear about what they are measuring
Timely: Can be measured at an interval that is appropriate to the level of change expected
Comparable: Can be compared across different target groups or project approaches
Please review these and make suggestions
Valid–For example, if we use the indicator “knows at least three modern methods of family planning,” this will give an indication over time of a changing level of knowledge. If, however, we are interested in whether people’s interest in using family planning is changing, this would NOT be a valid indicator.
Reliable–If we use the same indicator, it should be reliable when asked by different people during different survey rounds. However, if women (or interviewers) are not clear about the definition of “modern,” the validity may be compromised since different people may count different methods as modern.
Precise–As mentioned above, if we can clearly define our indicator by including a list of modern FP methods that are acceptable answers to be counted among the three, then the indicator is precise enough to determine whether the respondent can be counted among those who know three modern FP methods.
Timely–With a concerted education effort, we certainly expect to be able to see change in this indicator within a relatively short time. However, if for example we have a 2–3-year project and want to measure change in family size, the indicator (family size decreased) will not be observable within the life of the project.
Comparable–Our indicator on knowledge of three modern family planning methods is easily comparable across different groups. For example, it would be easy to compare whether husbands’ and wives’ knowledge levels are the same, or whether couples who received counseling vs. those who did not had the same knowledge level. In contrast, if we were to choose an indicator that is intervention-specific, such as those who receive counseling and know at least three modern methods, we could use this on the subgroup of people who received counseling but could not use this indicator with the population at large.

53. Criteria for Indicator Selection
Consistent with project design—measure the desired result
Useful—contributes to project design, management, and evaluation
Available
Affordable
Consistent with project design–This is related to validity. Is it measuring what we think it is measuring, and whether it is measuring what we think the project is impacting. If we are measuring change in knowledge because the project has an intensive community mobilization campaign, then the knowledge of three modern methods may be a good indicator. However, if the project is addressing quality of care and contraceptive availability, then change in knowledge may not be the best indicator, even if we are hoping that counseling on different methods is part of the quality improvement effort.
Useful–Indicators serve both to evaluate the impact of a project and to monitor its progress in order to make program adjustments. We may select some indicators that primarily serve for evaluation. However, it is also important to have indicators that will provide information for program adjustment. For example, the knowledge of modern family planning methods is a population-based indicator that is probably more useful to evaluate the impact of a community-mobilization campaign. However, an indicator measuring the availability of three different kinds of contraceptives in the health center could be used to evaluate the access and logistics system, but it also provides routine information that can be used to adjust the management and logistics systems if a problem is identified.
Available and Affordable–These criteria are essential and are often under-considered when planning. Information is useful, but it also costs time and money. We always need to balance the relative benefit of an indicator with the time and money it will cost to collect information on it. For example, if you are planning to do a baseline survey of mothers of children under 2 years old as part of a child survival project, you probably would not want to choose an indicator that asks about the percentage of men of reproductive age who know three modern methods of family planning. Even if you were targeting men and wanted some indicator of their level of involvement, if such an indicator is going to require that you survey men when all the other indicators target mothers, it is probably not worth it. An alternative might be the percentage of mothers of children under 2 years old (because this is the group you plan to survey) who report having discussed different contraceptive options with their husbands. While this is somewhat different, it might be valid as an indicator and a lot cheaper to collect.

54. Standard Indicators
Where possible, a project should select standard indicators.
They have been tested for validity and reliability.
They allow comparison between projects or sites.
They tend to be available for SOs and some IRs.
These are useful when project definitions and activities fit with what the standard indicators are measuring. This is more likely at the higher level of the results framework, where there is more consistency across projects and countries. The strategies and some of the IRs may be more project specific and therefore not easily measured by standard indicators.

55. How Many Indicators?
Choose at least one or two indicators per intermediate result, as well as the SO for evaluation purposes.
Choose one or two indicators per result for program monitoring.
Choose indicators that may be able to “cover” more than one element.
For management, think about basic activities that you need to monitor to judge if you are implementing activities as planned; include indicators that help you make decisions.
Evaluation indicators are the ones that will most likely be drawn from the standard indicators and will be used to draw conclusions across projects. To the extent possible, all the relevant core and standard indicators should be considered.
In some cases, by selecting one indicator, it may be possible to “cover” other project results. For example, no product stockouts during the past 6 months would imply that various elements of the logistic system were functioning. As such, it would not be necessary to measure separate elements (e.g., procurement, recordkeeping) unless they were necessary for management and supervision.
For another example, if we know that clients are returning to use services, we may assume that counseling for side effects and when to return is in fact occurring. If we are measuring new users as part of routine measurement toward our strategic objective, we do not need to try to collect information about increased knowledge of family planning among the catchment population, as they seem to know about and have an interest in getting services.

56. Exercise: Group work Refer to the indicators you selected..
Were the good indicators ?
Did you select a minimum number recommended given the type and size of your program?

57. M&E Framework

58. See comment for examples
Sample M&E Framework
Preventing Post Partum Hemorrhage :
Increase Active Management of the Third Stage of Labor
Result
Indicator
Definition
Data source
Collection Method
Freq- uency
Respons-ible Party
Active Management of the Third Stage of labor increased
Proportion of trained clinicians performing AMTSL to standard
# of trained midwives performing all steps of AMTSL on all patients/
AMTSL observa- tion checklist
Clinical observation
Annual
Zambia JHPIEGO staff
This is one indicator that was developed for the Ukraine Country Coordinating Mechanism (CCM) for their HIV/AIDS workplan submitted to the Global Fund for HIV/AIDS, TB and Malaria
See comment for examples

59. M&E Plan
The plan is a managerial tool that specifies the schedule, resources, responsibilities, for your M&E activities (data collection, data quality control, reporting, dissemination and use of data)
Note:
The plan should specify the time points when evaluations will be carried out, for example: Midterm, and End term.
Outcome/Impact evaluation is reserved for large longer term programs that can make impact at public health status level.
Your plan should include activities to monitor and evaluate the implementation, as planned, of the M&E plan.

60. Readiness for evaluation. Not all programs demand M&E at all levels
Readiness for evaluation. Not all programs demand M&E at all levels. Will talk about this further during sessions on developing and M&E plan.
Source: CDC. Global AIDS program monitoring and evaluation (M&E) field guide

61. Question
If funding for your case study program was cut off and the program closed in two years. What changes would you make to you M&E Plan?
ANSWER: Couple years of protection (CYP). This indicator was developed because it is not always feasible to do a population-based survey for a very large project. Formulas are used to estimate how many couples were protected from getting pregnant on the basis of the total distribution of different contraceptive methods. It makes the assumption that all contraceptives that are distributed are used, and it assumes a project can capture all contraceptives that are distributed (including through the private sector).

62. Sources of Information

63. THE END