Presentation is loading. Please wait.

Presentation is loading. Please wait.

Monitoring and Evaluation

Similar presentations


Presentation on theme: "Monitoring and Evaluation"— Presentation transcript:

1 Monitoring and Evaluation
Charles Katende PhD. Director of Monitoring, Evaluation and Research JHPIEGO (An affiliate of John Hopkins University)

2 Session Objective To increase participants understanding of the concepts used in designing M&E Frameworks and Plans To build participants competence in designing Program M&E Plans

3 Expected Results At the end of the sessions participants will know about Program frameworks, M&E frameworks and the difference between the two frameworks Participants will be able to identify and select appropriate indicators for a program. Participants will be able to produce a program monitoring and evaluation framework.

4 Introduction… Write the health problem addressed by a major public health program in your country Write at up to three specific objectives of a public health program that addresses the above mentioned health problem. Write down two indicators the program mentioned above uses to monitor it progress or performance towards its objectives.

5 What is Program Monitoring, Evaluation?
Monitoring is the routine process of data collection and measurement of progress toward program objectives. Evaluation is the use of social research methods to systematically investigate a achievement of a program’s results

6 Key Questions What is the purpose of carrying out M&E
Who needs, uses M&E Information Who carries out M&E? How is M&E carried out? When should M&E be carried out?

7 Improve program implementation
What is the purpose….? Improve program implementation Data on program progress and implementation Improve program management and decision making Inform future programming Inform stakeholders Accountability (donors, beneficiaries) Advocacy

8 Who needs, uses M&E Information?
Managers Donors Governments Technocrats Communities Beneficiaries To Improve program implementation… To Inform and improve future programs Inform stakeholders

9 Who conducts M&E….? Program implementer Stakeholders Beneficiary
Remember .. M&E Technical skills Participatory process

10 How to carry out M&E…? Key Features
Program Framework: Analyze and systematically lay out program elements Identify key elements to monitor and evaluate. Determine and describe the measures to be used for monitoring and evaluation Develop M&E Framework and action plans, including data collection and analysis, reporting and dissemination of findings. This morning, we began discussing #1, we are now going to talk about #2, and this afternoon Linda will discuss #3.

11 What do you know about your program….?
Program Framework What do you know about your program….?

12 Program Framework do ……………..achieve!
Systematic lay out of the program elements and path showing what the program plans to: do ……………..achieve!

13 Population, system level factors that cause the public health problem
Program Framework Based on a theoretical, empirical model, or general understanding Public health Problem Population, system level factors that cause the public health problem Action/interventions that can change the factors and ultimately alleviate/eliminate the problem

14 Improved Health Status
Results Framework Improved Health Status Impact: Improved (Sustained) Use of Key Health Services and Practices/Behaviors Strategic Objective: Increased quality of… Increased availability/ access to… Improved social / policy environment… Intermediate Result: The defined SO, IRs, and strategies should finally be filled in on this framework. Strategies: Strategies: Strategies: Strategies (Sub IR):

15 Example: Result Framework for a Family Planning Project
Example: Result Framework for a Family Planning Project GOAL: REDUCED FERTILITY SO: Increased FP use and improved FP/RH practices Increased knowledge of, improved attitudes toward, and acceptance of key services and behavior Increased quality of FP counseling and services for Increased availability/access to FP/RH Improved social and policy environment for FP Strategies:  Increase availability of educational materials at clinic and community level Community mobilization (using PRA and PDI) including men Implement mass media strategy Mobilize opinion leaders at national and local level Design/ implement supportive supervision System Train service providers (in-service and preservice in FP counseling and management of side effects Remodel clinic to allow for privacy Design and implement quality improvement program Strengthen logistics management Mobilize private sector providers Mobilize CHWs/CBDs Encourage socially marketed pills Advocate for community based distribution of pills Promote addition of Depo injections to EPI outreach strategy Pilot social marketing of pills

16 Basic Logic Model

17

18

19 Case 1: To decrease maternal mortality, a 10-year program plan to improve to train midwifes to Delivery and ANC services at health facilities, and to train and deploy CHWs to increase the community’s awareness about, and use of the improved services at the health facilities. Case 2: To reduce high fertility, a 5-year program plans to work with the Government to change policies in order to allow and promote use of modern family planning methods, train family planning providers to provide better FP services, and to launch public campaigns that promote family planning methods. Case 3: To reduce HIV infection among adolescents, a five-year program plans to implement income generation activities for the youth, provide and promote universal secondary education, and build adolescent-friendly reproductive health service delivery points.

20 Exercise Identify and state is the Public Health problem implied in the case study. What are population level factors will the program target to change in order to alleviate the public health problem Prepare a Program Framework for the scase study

21 Monitoring and Evaluations Framework

22 M&E Questions Monitoring questions What is being done? By whom?
Target population? When? How much? How often? Additional outputs? Resources used? (Staff, funds, materials, etc.)

23 M&E Questions Evaluation Questions?
Is the content of the intervention or the activity being delivered as planned? Does the content of the intervention or the activity reflect the requisite standards? Have the intervention achieved the expected results?

24 What do we need to answer these questions…
What do we need to answer these questions…? INDICATORS …to take measurements.

25 Indicators: Definition
Markers that help to measure change by showing progress towards meeting objectives Observable, measurable, and agreed upon as valid markers of a less well-defined concept or objective Indicators differ from objectives in that they address specific criteria that will be used to judge the success of the project or program. Indicators are more specifically defined than objectives, since they define the program attributes with a focus on expected effects translated into specific measures providing the basis collecting valid and reliable information for evaluation. ( CDC ‘Framework for Evaluation’ pg 15). Indicators spell out & provide guidance for periodic monitoring and non-periodic assessment of higher level outcomes (results). They are tools that tell the story of program progress and success. For example: increases in the contraceptive prevalence rate is an indicator for the increased use of family planning in a country. Other examples include: Indicators for measuring program activities such as program capacity to delivery services, participation rate, levels of client satisfaction, or amount of intervention exposure. Indicators for measuring program effects, such as changes in a particular behavior, community norms, or health status. See comment for examples

26 Type and Level of Each Indicator
Input/Process (Monitoring) Outcome / Impact (Evaluation) Level Global level Country level Program level

27 Exercise: Group work Use your case study and identify at least two indicators for program monitoring and two indicators for program evaluation.

28 What Is a Good Indicator?
Valid: Measures the effect it is supposed to measure Reliable: Gives same result if measured in the same way Precise: Is operationally defined so people are clear about what they are measuring Timely: Can be measured at an interval that is appropriate to the level of change expected Comparable: Can be compared across different target groups or project approaches Please review these and make suggestions Valid–For example, if we use the indicator “knows at least three modern methods of family planning,” this will give an indication over time of a changing level of knowledge. If, however, we are interested in whether people’s interest in using family planning is changing, this would NOT be a valid indicator. Reliable–If we use the same indicator, it should be reliable when asked by different people during different survey rounds. However, if women (or interviewers) are not clear about the definition of “modern,” the validity may be compromised since different people may count different methods as modern. Precise–As mentioned above, if we can clearly define our indicator by including a list of modern FP methods that are acceptable answers to be counted among the three, then the indicator is precise enough to determine whether the respondent can be counted among those who know three modern FP methods. Timely–With a concerted education effort, we certainly expect to be able to see change in this indicator within a relatively short time. However, if for example we have a 2–3-year project and want to measure change in family size, the indicator (family size decreased) will not be observable within the life of the project. Comparable–Our indicator on knowledge of three modern family planning methods is easily comparable across different groups. For example, it would be easy to compare whether husbands’ and wives’ knowledge levels are the same, or whether couples who received counseling vs. those who did not had the same knowledge level. In contrast, if we were to choose an indicator that is intervention-specific, such as those who receive counseling and know at least three modern methods, we could use this on the subgroup of people who received counseling but could not use this indicator with the population at large.

29 Criteria for Indicator Selection
Consistent with project design—measure the desired result Useful—contributes to project design, management, and evaluation Available Affordable Consistent with project design–This is related to validity. Is it measuring what we think it is measuring, and whether it is measuring what we think the project is impacting. If we are measuring change in knowledge because the project has an intensive community mobilization campaign, then the knowledge of three modern methods may be a good indicator. However, if the project is addressing quality of care and contraceptive availability, then change in knowledge may not be the best indicator, even if we are hoping that counseling on different methods is part of the quality improvement effort. Useful–Indicators serve both to evaluate the impact of a project and to monitor its progress in order to make program adjustments. We may select some indicators that primarily serve for evaluation. However, it is also important to have indicators that will provide information for program adjustment. For example, the knowledge of modern family planning methods is a population-based indicator that is probably more useful to evaluate the impact of a community-mobilization campaign. However, an indicator measuring the availability of three different kinds of contraceptives in the health center could be used to evaluate the access and logistics system, but it also provides routine information that can be used to adjust the management and logistics systems if a problem is identified. Available and Affordable–These criteria are essential and are often under-considered when planning. Information is useful, but it also costs time and money. We always need to balance the relative benefit of an indicator with the time and money it will cost to collect information on it. For example, if you are planning to do a baseline survey of mothers of children under 2 years old as part of a child survival project, you probably would not want to choose an indicator that asks about the percentage of men of reproductive age who know three modern methods of family planning. Even if you were targeting men and wanted some indicator of their level of involvement, if such an indicator is going to require that you survey men when all the other indicators target mothers, it is probably not worth it. An alternative might be the percentage of mothers of children under 2 years old (because this is the group you plan to survey) who report having discussed different contraceptive options with their husbands. While this is somewhat different, it might be valid as an indicator and a lot cheaper to collect.

30 Standard Indicators Where possible, a project should select standard indicators. They have been tested for validity and reliability. They allow comparison between projects or sites. They tend to be available for SOs and some IRs. These are useful when project definitions and activities fit with what the standard indicators are measuring. This is more likely at the higher level of the results framework, where there is more consistency across projects and countries. The strategies and some of the IRs may be more project specific and therefore not easily measured by standard indicators.

31 How Many Indicators? Choose at least one or two indicators per intermediate result, as well as the SO for evaluation purposes. Choose one or two indicators per result for program monitoring. Choose indicators that may be able to “cover” more than one element. For management, think about basic activities that you need to monitor to judge if you are implementing activities as planned; include indicators that help you make decisions. Evaluation indicators are the ones that will most likely be drawn from the standard indicators and will be used to draw conclusions across projects. To the extent possible, all the relevant core and standard indicators should be considered. In some cases, by selecting one indicator, it may be possible to “cover” other project results. For example, no product stockouts during the past 6 months would imply that various elements of the logistic system were functioning. As such, it would not be necessary to measure separate elements (e.g., procurement, recordkeeping) unless they were necessary for management and supervision. For another example, if we know that clients are returning to use services, we may assume that counseling for side effects and when to return is in fact occurring. If we are measuring new users as part of routine measurement toward our strategic objective, we do not need to try to collect information about increased knowledge of family planning among the catchment population, as they seem to know about and have an interest in getting services.

32 Exercise: Group work Refer to the indicators you selected..
Were the good indicators ? Did you select a minimum number recommended given the type and size of your program?

33 M&E Framework

34 See comment for examples
Sample M&E Framework Preventing Post Partum Hemorrhage : Increase Active Management of the Third Stage of Labor Result Indicator Definition Data source Collection Method Freq- uency Respons-ible Party Active Management of the Third Stage of labor increased Proportion of trained clinicians performing AMTSL to standard # of trained midwives performing all steps of AMTSL on all patients/ AMTSL observa- tion checklist Clinical observation Annual Zambia JHPIEGO staff This is one indicator that was developed for the Ukraine Country Coordinating Mechanism (CCM) for their HIV/AIDS workplan submitted to the Global Fund for HIV/AIDS, TB and Malaria See comment for examples

35 M&E Plan The plan is a managerial tool that specifies the schedule, resources, responsibilities, for your M&E activities (data collection, data quality control, reporting, dissemination and use of data) Note: The plan should specify the time points when evaluations will be carried out, for example: Midterm, and End term. Outcome/Impact evaluation is reserved for large longer term programs that can make impact at public health status level. Your plan should include activities to monitor and evaluate the implementation, as planned, of the M&E plan.

36 Readiness for evaluation. Not all programs demand M&E at all levels
Readiness for evaluation. Not all programs demand M&E at all levels. Will talk about this further during sessions on developing and M&E plan. Source: CDC. Global AIDS program monitoring and evaluation (M&E) field guide

37 Question If funding for your case study program was cut off and the program closed in two years. What changes would you make to you M&E Plan? ANSWER: Couple years of protection (CYP). This indicator was developed because it is not always feasible to do a population-based survey for a very large project. Formulas are used to estimate how many couples were protected from getting pregnant on the basis of the total distribution of different contraceptive methods. It makes the assumption that all contraceptives that are distributed are used, and it assumes a project can capture all contraceptives that are distributed (including through the private sector).

38 Sources of Information

39 THE END


Download ppt "Monitoring and Evaluation"

Similar presentations


Ads by Google