Presentation is loading. Please wait.

Presentation is loading. Please wait.

Monitoring and Evaluation

Similar presentations


Presentation on theme: "Monitoring and Evaluation"— Presentation transcript:

1 Monitoring and Evaluation

2 Session Objective To increase participants understanding of the concepts used in designing M&E Frameworks and Plans To build participants competence in designing Program M&E Plans

3 Expected Results At the end of the sessions participants will know about Program frameworks, M&E frameworks and the difference between the two frameworks Participants will be able to identify and select appropriate indicators for a program. Participants will be able to produce a program monitoring and evaluation framework.

4 Key Terms for Monitoring and Evaluation

5 Objectives Explain the difference between monitoring and evaluation.
Introduce the most common M&E terms. Review examples of each term. In this session, we will cover the following: Read session objectives.

6 Monitoring versus Evaluation
Data collected on program activities Ongoing, routine Focus on activities and output, compared to target Are we doing the work we planned? Evaluation Data collected to answer specific questions Periodic Focus on outcome, impact How effective were our activities? THIS SLIDE IS ANIMATED Let’s begin by comparing monitoring and evaluation. CLICK to display Monitoring. Monitoring is the routine collection and analysis of program data, specifically on program activities. Data collection is ongoing (weekly, monthly, quarterly, semiannually, etc.). CLICK to display question. Monitoring compares results to the original targets to answer the question: Are we doing the work we planned? It can alert us to problems early because we continually review program data. CLICK to display Evaluation. Evaluation answers specific questions about our performance. It is in-depth analysis that tells us if we should continue or improve our activities. Evaluation is less frequent and answers the question: How effective were our activities? Sometimes monitoring leads us to an evaluation. We may see a trend in the routine data that we cannot explain. So we could design an evaluation to help us understand what is going on.

7 Monitoring or Evaluation?
Local researchers conduct a study to determine if there are more people with possible TB symptoms coming to clinics as a result of a media campaign to promote TB screening. Evaluation THIS SLIDE IS ANIMATED Read example and ask participants if this is monitoring or evaluation. CLICK to reveal answer.

8 Monitoring or Evaluation?
A district manager reports on how many nurses were trained on interpersonal communication skills for her quarterly donor report. Monitoring THIS SLIDE IS ANIMATED Read example and ask participants if this is monitoring or evaluation. CLICK to reveal answer.

9 It Starts with QUESTIONS
Monitoring and evaluation answer different questions. If we do not ask good questions about our activities, we will not get useful data! Our monitoring and evaluation efforts are only as good as the questions we need to answer. Let’s look again at the questions we wrote on the flipchart. Point to each question and ask participants to identify the question as a monitoring or an evaluation question. We can see that there are endless questions we could ask about our activities. How should we prioritize what we ask? We will talk more about this later in the training. For now, just think about the variety of questions we could ask and the variety of information we could get about our activities.

10 What is a GOAL? The ultimate result of efforts at a broad, population level. Achieved over the long term (years) and through combined efforts of multiple programs. THIS SLIDE IS ANIMATED. Goals are meant to have a very broad impact across a very broad population. They usually take a very long time to achieve and need several types of activities to do so. It usually takes more than ACSM to achieve, but ACSM certainly contributes. Here are some examples. CLICK to display examples. Example 1: This shows a large population. Example 2: This goal will need several programs to be accomplished. Example 3: It will take many years to reduce stigma. Decrease morbidity and mortality due to TB in Country X. Reduce prevalence of TB by 50%. Eliminate stigma of TB in our communities.

11 What is an OBJECTIVE? OBJECTIVES GOAL activities
How the results of your short-term program activities contribute to the big goal. Several objectives can relate to the same goal. THIS SLIDE IS ANIMATED. So we have a long-term, broad goal and we also have our short-term, specific activities. They do not seem very connected unless there is an objective to link them together. CLICK to add picture. CLICK again to display bullets. The objective is the result of our short-term program activities, and contributes to the larger goal. In other words, the objectives say what we will do and when. The activity is how we will do it and the goal is why we will do it. You can have several objectives for the same goal or several activities for the same objective.

12 Objective Examples Aggressively advocate to increase budget by 8% each year for the next four years. Double the percentage of secondary school students who can correctly identify TB symptoms by 2015. Design and pilot a treatment support program for newly released prisoners with TB by 2015. Here are some examples of ACSM objectives. CLICK to display examples.

13 INPUTS Resources needed to plan and implement XX
“Raw materials” of an XX project Examples Money Staff Policies, guidelines Equipment Partners THIS SLIDE IS ANIMATED. Inputs are the raw materials you need to conduct your activities. CLICK to display examples. These include items such as money, staff, policies, training materials, etc. When developing your ACSM and M&E plans, it is often helpful to determine which inputs you already have and which inputs you still need. You may need to do some advocacy to get the inputs you need (such as money or formal permission from the Ministry of Health).

14 ACTIVITIES The work that we do, what we implement
Also called “processes” Examples Training events Meetings Events Outreach Home visits THIS SLIDE IS ANIMATED. We all know what activities are. This is the work that we do. Sometimes activities are called “processes.” CLICK to display examples and read.

15 OUTPUTS Immediate results of activities
What we can measure/count right after the activity Examples Number of people trained Number of brochures produced Number of policymakers reached with advocacy activity THIS SLIDE IS ANIMATED. Outputs are the immediate results of our activities. They are the proof that our activity happened. Often they are expressed as a number or amount. Outputs can be measured or counted right after our activity is complete. CLICK to display examples and read. Can anyone give me an example of an output for one of your ACSM activities? Solicit 1-2 examples.

16 OUTCOMES “Ripple effects” of XX activities
What changes after outputs are produced Examples Increased funding for TB after lobbying meeting Short: Improved attitudes toward TB patients among DOTS nurses after a training Medium: Increased satisfaction of TB clients Long: TB clients stay in treatment longer THIS SLIDE IS ANIMATED. Outputs lead to outcomes, which are the “ripple effects” of our activities, or what happens as a result of our activities. CLICK to display examples. In the first example, what would be the outputs? Solicit answers. (Meeting held, number of meeting participants.) Outcomes can be short-, medium-, or long-term effects. In the second example, we can clearly see how outcomes lead to each other over time. Example #2 is a good example of a ripple effect of an interpersonal communication training. Read example #2 out loud and briefly discuss.

17 INDICATORS How we define our activities, outputs, or outcomes Signs or evidence we watch for to see if we have reached them ACTIVITY: Meeting with Finance Minister and NTP Director to lobby for more funding for NTP OUTPUT: Number of officials attending the meeting INDICATOR: Number of officials attending the meeting compared to number invited OUTCOME: Increased funding Percentage of NTP budget covered by the Ministry of Health Indicators are signs or evidence of our activities, outputs, and outcomes. They “indicate” that something has happened. Indicators are what we watch for. For example, if you are yawning at this point in the session, that is an indicator that you are tired or my presentation is boring. We will talk more about indicators later, but here are examples to introduce the concept. Review examples.

18 IMPACT More related to goal Very broad-scale result over long term
Examples Higher rate of treatment success Reduction in deaths among MDR-TB patients THIS SLIDE IS ANIMATED. Impact is what happens over the longer term and is usually related to goals. Impact is measured at a broader scale and requires data beyond the scope of our project. CLICK to display examples and read. The difference between outcomes and impact is how confident we are that the changes are a direct result of our activities and not the result of other factors. This is very difficult to prove without expensive research.

19 Introduction… Write the administrative/engineering/health problem addressed by a major public health program in your country Write at up to three specific objectives of a engineering/health program that addresses the above mentioned health problem. Write down two indicators the program mentioned above uses to monitor it progress or performance towards its objectives.

20 The Project cycle

21 Project Cycle Jan Davis and Robert Lambert, Engineering in Emergencies, p.63.

22 What is Program Monitoring, Evaluation?
Monitoring is the routine process of data collection and measurement of progress toward program objectives. Evaluation is the use of social research methods to systematically investigate a achievement of a program’s results

23 Differences and Links between Monitoring and Evaluation
SLIDE 7 Monitoring Evaluation A continuous process A specific activity or moment To provide information to day-to-day decision making (adjustments) To provide recommendations to strategic decision-making processes It is carried out by the project team For the project team (to adapt and improve the impacts) and the donors (to follow the progress) It is carried out by an evaluation team (internal or external to the project team) For the project team and the donors (lesson learned) The monitoring system should provide information for evaluations

24 Comparison Between M&E
Item Monitoring Evaluation Frequency Regular, ongoing Episodic Main action Keeping track/oversight Assessment Basic purpose Improving efficiency Adjusting work plan Improve effectiveness, impact, future programming Focus Inputs/outputs, process outcomes, work plans Effectiveness, relevance, efficiency, impact, sustainability Information sources Routine systems, field visits, stakeholder meetings, output reports, rapid assessments Same plus Surveys (pre-post project) Special studies Undertaken by Project/program managers Community workers Supervisors Community (beneficiaries) Funders Other Stakeholders External evaluators Adapted from UNICEF, A UNICEF Guide for Monitoring and Evaluation: Making a Difference? New York, 1991, p.3

25 MONITORING MONITORING is the continuous, systematic and critical review of operations in order to measure their evolution and adjust them according to circumstances and project’s objectives. Monitoring must be systematic. Explain the difference between “monitoring” & “assessment” thoroughly. It is probable that the assessment, the project design, the monitoring & the impact study shall be conducted by different individuals – hence the importance of clearly defining the desired impact, the objective, the monitoring & impact indicators from the onset (project conception – logical framework). Monitoring with or without indicators. Monitoring of the project’s progress &/or of its activities’ impact. Define ahead what one wishes to monitor & why.

26 Building a Monitoring system

27 BUILDING A MONITORING SYSTEM
Intervention objectives

28 What should a M&E System Measure?
The Results Chain Indicative Example: Outcomes Impact Outputs Activities Inputs Effects on dimensions of well-being Improve literacy Access to, use of, and satisfaction with services School enrollment rates Goods and services produced Number of schools built; textbooks, etc. Tasks undertaken to transform inputs to outputs Building of schools Distribution of textbooks, etc. Financial, human and material resources Spending on primary education Source: Adapted from ADB (2006) Introduction to Results Management, p. 7 World Bank (2001) PRSP Sourcebook, p. 108.

29 Some examples of outcome indicators
Improved road network % of roads in good condition; average travel time Improved public sector performance % of population satisfied with public services; tax collection (% of GDP) Improved health services Utilization rate of healthcare centers Increased tourism Number of tourists visiting the region Source: ADB Practice Note on Results-based Country Strategies and Programs, Annex 2.

30 Key Questions What is the purpose of carrying out M&E
Who needs, uses M&E Information Who carries out M&E? How is M&E carried out? When should M&E be carried out?

31 Improve program implementation
What is the purpose….? Improve program implementation Data on program progress and implementation Improve program management and decision making Inform future programming Inform stakeholders Accountability (donors, beneficiaries) Advocacy

32 Who needs, uses M&E Information?
Managers Donors Governments Technocrats Communities Beneficiaries To Improve program implementation… To Inform and improve future programs Inform stakeholders

33 Who conducts M&E….? Program implementer Stakeholders Beneficiary
Remember .. M&E Technical skills Participatory process

34 How to carry out M&E…? Key Features
Program Framework: Analyze and systematically lay out program elements Identify key elements to monitor and evaluate. Determine and describe the measures to be used for monitoring and evaluation Develop M&E Framework and action plans, including data collection and analysis, reporting and dissemination of findings. This morning, we began discussing #1, we are now going to talk about #2, and this afternoon Linda will discuss #3.

35 What do you know about your program….?
Program Framework What do you know about your program….?

36 Program Framework do ……………..achieve!
Systematic lay out of the program elements and path showing what the program plans to: do ……………..achieve!

37 Population, system level factors that cause the public health problem
Program Framework Based on a theoretical, empirical model, or general understanding Public health Problem Population, system level factors that cause the public health problem Action/interventions that can change the factors and ultimately alleviate/eliminate the problem

38 Improved Health Status
Results Framework Improved Health Status Impact: Improved (Sustained) Use of Key Health Services and Practices/Behaviors Strategic Objective: Increased quality of… Increased availability/ access to… Improved social / policy environment… Intermediate Result: The defined SO, IRs, and strategies should finally be filled in on this framework. Strategies: Strategies: Strategies: Strategies (Sub IR):

39 Example: Result Framework for a Family Planning Project
Example: Result Framework for a Family Planning Project GOAL: REDUCED FERTILITY SO: Increased FP use and improved FP/RH practices Increased knowledge of, improved attitudes toward, and acceptance of key services and behavior Increased quality of FP counseling and services for Increased availability/access to FP/RH Improved social and policy environment for FP Strategies:  Increase availability of educational materials at clinic and community level Community mobilization (using PRA and PDI) including men Implement mass media strategy Mobilize opinion leaders at national and local level Design/ implement supportive supervision System Train service providers (in-service in FP counseling and management of side effects) Remodel clinic to allow for privacy Design and implement quality improvement program Strengthen logistics management Mobilize private sector providers Mobilize CHWs/CBDs Encourage socially marketed pills Advocate for community based distribution of pills Promote addition of Depo injections to EPI outreach strategy Pilot social marketing of pills

40 Basic Logic Model

41

42

43 Case 1: To decrease maternal mortality, a 10-year program plan to improve to train midwifes to Delivery and ANC services at health facilities, and to train and deploy CHWs to increase the community’s awareness about, and use of the improved services at the health facilities. Case 2: To reduce high fertility, a 5-year program plans to work with the Government to change policies in order to allow and promote use of modern family planning methods, train family planning providers to provide better FP services, and to launch public campaigns that promote family planning methods. Case 3: To reduce HIV infection among adolescents, a five-year program plans to implement income generation activities for the youth, provide and promote universal secondary education, and build adolescent-friendly reproductive health service delivery points.

44 Exercise Identify and state is the Public Health problem implied in the case study. What are population level factors will the program target to change in order to alleviate the public health problem Prepare a Program Framework for the scase study

45 Monitoring and Evaluations Framework

46 M&E Questions Monitoring questions What is being done? By whom?
Target population? When? How much? How often? Additional outputs? Resources used? (Staff, funds, materials, etc.)

47 M&E Questions Evaluation Questions?
Is the content of the intervention or the activity being delivered as planned? Does the content of the intervention or the activity reflect the requisite standards? Have the intervention achieved the expected results?

48 What do we need to answer these questions…
What do we need to answer these questions…? INDICATORS …to take measurements.

49 Indicators: Definition
Markers that help to measure change by showing progress towards meeting objectives Observable, measurable, and agreed upon as valid markers of a less well-defined concept or objective Indicators differ from objectives in that they address specific criteria that will be used to judge the success of the project or program. Indicators are more specifically defined than objectives, since they define the program attributes with a focus on expected effects translated into specific measures providing the basis collecting valid and reliable information for evaluation. ( CDC ‘Framework for Evaluation’ pg 15). Indicators spell out & provide guidance for periodic monitoring and non-periodic assessment of higher level outcomes (results). They are tools that tell the story of program progress and success. For example: increases in the contraceptive prevalence rate is an indicator for the increased use of family planning in a country. Other examples include: Indicators for measuring program activities such as program capacity to delivery services, participation rate, levels of client satisfaction, or amount of intervention exposure. Indicators for measuring program effects, such as changes in a particular behavior, community norms, or health status. See comment for examples

50 Type and Level of Each Indicator
Input/Process (Monitoring) Outcome / Impact (Evaluation) Level Global level Country level Program level

51 Exercise: Group work Use your case study and identify at least two indicators for program monitoring and two indicators for program evaluation.

52 What Is a Good Indicator?
Valid: Measures the effect it is supposed to measure Reliable: Gives same result if measured in the same way Precise: Is operationally defined so people are clear about what they are measuring Timely: Can be measured at an interval that is appropriate to the level of change expected Comparable: Can be compared across different target groups or project approaches Please review these and make suggestions Valid–For example, if we use the indicator “knows at least three modern methods of family planning,” this will give an indication over time of a changing level of knowledge. If, however, we are interested in whether people’s interest in using family planning is changing, this would NOT be a valid indicator. Reliable–If we use the same indicator, it should be reliable when asked by different people during different survey rounds. However, if women (or interviewers) are not clear about the definition of “modern,” the validity may be compromised since different people may count different methods as modern. Precise–As mentioned above, if we can clearly define our indicator by including a list of modern FP methods that are acceptable answers to be counted among the three, then the indicator is precise enough to determine whether the respondent can be counted among those who know three modern FP methods. Timely–With a concerted education effort, we certainly expect to be able to see change in this indicator within a relatively short time. However, if for example we have a 2–3-year project and want to measure change in family size, the indicator (family size decreased) will not be observable within the life of the project. Comparable–Our indicator on knowledge of three modern family planning methods is easily comparable across different groups. For example, it would be easy to compare whether husbands’ and wives’ knowledge levels are the same, or whether couples who received counseling vs. those who did not had the same knowledge level. In contrast, if we were to choose an indicator that is intervention-specific, such as those who receive counseling and know at least three modern methods, we could use this on the subgroup of people who received counseling but could not use this indicator with the population at large.

53 Criteria for Indicator Selection
Consistent with project design—measure the desired result Useful—contributes to project design, management, and evaluation Available Affordable Consistent with project design–This is related to validity. Is it measuring what we think it is measuring, and whether it is measuring what we think the project is impacting. If we are measuring change in knowledge because the project has an intensive community mobilization campaign, then the knowledge of three modern methods may be a good indicator. However, if the project is addressing quality of care and contraceptive availability, then change in knowledge may not be the best indicator, even if we are hoping that counseling on different methods is part of the quality improvement effort. Useful–Indicators serve both to evaluate the impact of a project and to monitor its progress in order to make program adjustments. We may select some indicators that primarily serve for evaluation. However, it is also important to have indicators that will provide information for program adjustment. For example, the knowledge of modern family planning methods is a population-based indicator that is probably more useful to evaluate the impact of a community-mobilization campaign. However, an indicator measuring the availability of three different kinds of contraceptives in the health center could be used to evaluate the access and logistics system, but it also provides routine information that can be used to adjust the management and logistics systems if a problem is identified. Available and Affordable–These criteria are essential and are often under-considered when planning. Information is useful, but it also costs time and money. We always need to balance the relative benefit of an indicator with the time and money it will cost to collect information on it. For example, if you are planning to do a baseline survey of mothers of children under 2 years old as part of a child survival project, you probably would not want to choose an indicator that asks about the percentage of men of reproductive age who know three modern methods of family planning. Even if you were targeting men and wanted some indicator of their level of involvement, if such an indicator is going to require that you survey men when all the other indicators target mothers, it is probably not worth it. An alternative might be the percentage of mothers of children under 2 years old (because this is the group you plan to survey) who report having discussed different contraceptive options with their husbands. While this is somewhat different, it might be valid as an indicator and a lot cheaper to collect.

54 Standard Indicators Where possible, a project should select standard indicators. They have been tested for validity and reliability. They allow comparison between projects or sites. They tend to be available for SOs and some IRs. These are useful when project definitions and activities fit with what the standard indicators are measuring. This is more likely at the higher level of the results framework, where there is more consistency across projects and countries. The strategies and some of the IRs may be more project specific and therefore not easily measured by standard indicators.

55 How Many Indicators? Choose at least one or two indicators per intermediate result, as well as the SO for evaluation purposes. Choose one or two indicators per result for program monitoring. Choose indicators that may be able to “cover” more than one element. For management, think about basic activities that you need to monitor to judge if you are implementing activities as planned; include indicators that help you make decisions. Evaluation indicators are the ones that will most likely be drawn from the standard indicators and will be used to draw conclusions across projects. To the extent possible, all the relevant core and standard indicators should be considered. In some cases, by selecting one indicator, it may be possible to “cover” other project results. For example, no product stockouts during the past 6 months would imply that various elements of the logistic system were functioning. As such, it would not be necessary to measure separate elements (e.g., procurement, recordkeeping) unless they were necessary for management and supervision. For another example, if we know that clients are returning to use services, we may assume that counseling for side effects and when to return is in fact occurring. If we are measuring new users as part of routine measurement toward our strategic objective, we do not need to try to collect information about increased knowledge of family planning among the catchment population, as they seem to know about and have an interest in getting services.

56 Exercise: Group work Refer to the indicators you selected..
Were the good indicators ? Did you select a minimum number recommended given the type and size of your program?

57 M&E Framework

58 See comment for examples
Sample M&E Framework Preventing Post Partum Hemorrhage : Increase Active Management of the Third Stage of Labor Result Indicator Definition Data source Collection Method Freq- uency Respons-ible Party Active Management of the Third Stage of labor increased Proportion of trained clinicians performing AMTSL to standard # of trained midwives performing all steps of AMTSL on all patients/ AMTSL observa- tion checklist Clinical observation Annual Zambia JHPIEGO staff This is one indicator that was developed for the Ukraine Country Coordinating Mechanism (CCM) for their HIV/AIDS workplan submitted to the Global Fund for HIV/AIDS, TB and Malaria See comment for examples

59 M&E Plan The plan is a managerial tool that specifies the schedule, resources, responsibilities, for your M&E activities (data collection, data quality control, reporting, dissemination and use of data) Note: The plan should specify the time points when evaluations will be carried out, for example: Midterm, and End term. Outcome/Impact evaluation is reserved for large longer term programs that can make impact at public health status level. Your plan should include activities to monitor and evaluate the implementation, as planned, of the M&E plan.

60 Readiness for evaluation. Not all programs demand M&E at all levels
Readiness for evaluation. Not all programs demand M&E at all levels. Will talk about this further during sessions on developing and M&E plan. Source: CDC. Global AIDS program monitoring and evaluation (M&E) field guide

61 Question If funding for your case study program was cut off and the program closed in two years. What changes would you make to you M&E Plan? ANSWER: Couple years of protection (CYP). This indicator was developed because it is not always feasible to do a population-based survey for a very large project. Formulas are used to estimate how many couples were protected from getting pregnant on the basis of the total distribution of different contraceptive methods. It makes the assumption that all contraceptives that are distributed are used, and it assumes a project can capture all contraceptives that are distributed (including through the private sector).

62 Sources of Information

63 THE END


Download ppt "Monitoring and Evaluation"

Similar presentations


Ads by Google