Presentation on theme: "Monitoring and Evaluation"— Presentation transcript:
1 Monitoring and Evaluation Charles Katende PhD.Director of Monitoring, Evaluation and ResearchJHPIEGO (An affiliate of John Hopkins University)
2 Session ObjectiveTo increase participants understanding of the concepts used in designing M&E Frameworks and PlansTo build participants competence in designing Program M&E Plans
3 Expected ResultsAt the end of the sessions participants will know about Program frameworks, M&E frameworks and the difference between the two frameworksParticipants will be able to identify and select appropriate indicators for a program.Participants will be able to produce a program monitoring and evaluation framework.
4 Introduction…Write the health problem addressed by a major public health program in your countryWrite at up to three specific objectives of a public health program that addresses the above mentioned health problem.Write down two indicators the program mentioned above uses to monitor it progress or performance towards its objectives.
5 What is Program Monitoring, Evaluation? Monitoring is the routine process of data collection and measurement of progress toward program objectives.Evaluation is the use of social research methods to systematically investigate a achievement of a program’s results
6 Key Questions What is the purpose of carrying out M&E Who needs, uses M&E InformationWho carries out M&E?How is M&E carried out?When should M&E be carried out?
7 Improve program implementation What is the purpose….?Improve program implementationData on program progress and implementationImprove program management and decision makingInform future programmingInform stakeholdersAccountability (donors, beneficiaries)Advocacy
8 Who needs, uses M&E Information? ManagersDonorsGovernmentsTechnocratsCommunitiesBeneficiariesTo Improve program implementation…To Inform and improve future programsInform stakeholders
9 Who conducts M&E….? Program implementer Stakeholders Beneficiary Remember ..M&E Technical skillsParticipatory process
10 How to carry out M&E…? Key Features Program Framework: Analyze and systematically lay out program elementsIdentify key elements to monitor and evaluate.Determine and describe the measures to be used for monitoring and evaluationDevelop M&E Framework and action plans, including data collection and analysis, reporting and dissemination of findings.This morning, we began discussing #1, we are now going to talk about #2, and this afternoon Linda will discuss #3.
11 What do you know about your program….? Program FrameworkWhat do you know about your program….?
12 Program Framework do ……………..achieve! Systematic lay out of the program elements and path showing what the program plans to:do ……………..achieve!
13 Population, system level factors that cause the public health problem Program FrameworkBased on a theoretical, empirical model, or general understandingPublic health ProblemPopulation, system level factors that cause the public health problemAction/interventions that can change the factors and ultimately alleviate/eliminate the problem
14 Improved Health Status Results FrameworkImproved Health StatusImpact:Improved (Sustained) Use of Key Health Services and Practices/BehaviorsStrategic Objective:Increased quality of…Increased availability/ access to…Improved social / policy environment…Intermediate Result:The defined SO, IRs, and strategies should finally be filled in on this framework.Strategies:Strategies:Strategies:Strategies (Sub IR):
15 Example: Result Framework for a Family Planning Project Example: Result Framework for a Family Planning ProjectGOAL: REDUCED FERTILITYSO: Increased FP use and improved FP/RH practicesIncreased knowledge of, improved attitudes toward, and acceptance of key services and behaviorIncreased quality of FP counseling and services forIncreased availability/access to FP/RHImproved social and policy environment for FPStrategies: Increase availability of educational materials at clinic and community levelCommunity mobilization (using PRA and PDI) including menImplement mass media strategyMobilize opinion leaders at national and local levelDesign/ implement supportive supervision SystemTrain service providers (in-service and preservice in FP counseling and management of side effectsRemodel clinic to allow for privacyDesign and implement quality improvement programStrengthen logistics managementMobilize private sector providersMobilize CHWs/CBDsEncourage socially marketed pillsAdvocate for community based distribution of pillsPromote addition of Depo injections to EPI outreach strategyPilot social marketing of pills
19 Case 1: To decrease maternal mortality, a 10-year program plan to improve to train midwifes to Delivery and ANC services at health facilities, and to train and deploy CHWs to increase the community’s awareness about, and use of the improved services at the health facilities.Case 2: To reduce high fertility, a 5-year program plans to work with the Government to change policies in order to allow and promote use of modern family planning methods, train family planning providers to provide better FP services, and to launch public campaigns that promote family planning methods.Case 3: To reduce HIV infection among adolescents, a five-year program plans to implement income generation activities for the youth, provide and promote universal secondary education, and build adolescent-friendly reproductive health service delivery points.
20 ExerciseIdentify and state is the Public Health problem implied in the case study.What are population level factors will the program target to change in order to alleviate the public health problemPrepare a Program Framework for the scase study
22 M&E Questions Monitoring questions What is being done? By whom? Target population?When?How much?How often?Additional outputs?Resources used? (Staff, funds, materials, etc.)
23 M&E Questions Evaluation Questions? Is the content of the intervention or the activity being delivered as planned?Does the content of the intervention or the activity reflect the requisite standards?Have the intervention achieved the expected results?
24 What do we need to answer these questions… What do we need to answer these questions…? INDICATORS …to take measurements.
25 Indicators: Definition Markers that help to measure change by showing progress towards meeting objectivesObservable, measurable, and agreed upon as valid markers of a less well-defined concept or objectiveIndicators differ from objectives in that they address specific criteria that will be used to judge the success of the project or program.Indicators are more specifically defined than objectives, since they define the program attributes with a focus on expected effects translated into specific measures providing the basis collecting valid and reliable information for evaluation. ( CDC ‘Framework for Evaluation’ pg 15). Indicators spell out & provide guidance for periodic monitoring and non-periodic assessment of higher level outcomes (results). They are tools that tell the story of program progress and success.For example: increases in the contraceptive prevalence rate is an indicator for the increased use of family planning in a country.Other examples include:Indicators for measuring program activities such as program capacity to delivery services, participation rate, levels of client satisfaction, or amount of intervention exposure.Indicators for measuring program effects, such as changes in a particular behavior, community norms, or health status.See comment for examples
26 Type and Level of Each Indicator Input/Process (Monitoring)Outcome / Impact (Evaluation)LevelGlobal levelCountry levelProgram level
27 Exercise: Group workUse your case study and identify at least two indicators for program monitoring and two indicators for program evaluation.
28 What Is a Good Indicator? Valid: Measures the effect it is supposed to measureReliable: Gives same result if measured in the same wayPrecise: Is operationally defined so people are clear about what they are measuringTimely: Can be measured at an interval that is appropriate to the level of change expectedComparable: Can be compared across different target groups or project approachesPlease review these and make suggestionsValid–For example, if we use the indicator “knows at least three modern methods of family planning,” this will give an indication over time of a changing level of knowledge. If, however, we are interested in whether people’s interest in using family planning is changing, this would NOT be a valid indicator.Reliable–If we use the same indicator, it should be reliable when asked by different people during different survey rounds. However, if women (or interviewers) are not clear about the definition of “modern,” the validity may be compromised since different people may count different methods as modern.Precise–As mentioned above, if we can clearly define our indicator by including a list of modern FP methods that are acceptable answers to be counted among the three, then the indicator is precise enough to determine whether the respondent can be counted among those who know three modern FP methods.Timely–With a concerted education effort, we certainly expect to be able to see change in this indicator within a relatively short time. However, if for example we have a 2–3-year project and want to measure change in family size, the indicator (family size decreased) will not be observable within the life of the project.Comparable–Our indicator on knowledge of three modern family planning methods is easily comparable across different groups. For example, it would be easy to compare whether husbands’ and wives’ knowledge levels are the same, or whether couples who received counseling vs. those who did not had the same knowledge level. In contrast, if we were to choose an indicator that is intervention-specific, such as those who receive counseling and know at least three modern methods, we could use this on the subgroup of people who received counseling but could not use this indicator with the population at large.
29 Criteria for Indicator Selection Consistent with project design—measure the desired resultUseful—contributes to project design, management, and evaluationAvailableAffordableConsistent with project design–This is related to validity. Is it measuring what we think it is measuring, and whether it is measuring what we think the project is impacting. If we are measuring change in knowledge because the project has an intensive community mobilization campaign, then the knowledge of three modern methods may be a good indicator. However, if the project is addressing quality of care and contraceptive availability, then change in knowledge may not be the best indicator, even if we are hoping that counseling on different methods is part of the quality improvement effort.Useful–Indicators serve both to evaluate the impact of a project and to monitor its progress in order to make program adjustments. We may select some indicators that primarily serve for evaluation. However, it is also important to have indicators that will provide information for program adjustment. For example, the knowledge of modern family planning methods is a population-based indicator that is probably more useful to evaluate the impact of a community-mobilization campaign. However, an indicator measuring the availability of three different kinds of contraceptives in the health center could be used to evaluate the access and logistics system, but it also provides routine information that can be used to adjust the management and logistics systems if a problem is identified.Available and Affordable–These criteria are essential and are often under-considered when planning. Information is useful, but it also costs time and money. We always need to balance the relative benefit of an indicator with the time and money it will cost to collect information on it. For example, if you are planning to do a baseline survey of mothers of children under 2 years old as part of a child survival project, you probably would not want to choose an indicator that asks about the percentage of men of reproductive age who know three modern methods of family planning. Even if you were targeting men and wanted some indicator of their level of involvement, if such an indicator is going to require that you survey men when all the other indicators target mothers, it is probably not worth it. An alternative might be the percentage of mothers of children under 2 years old (because this is the group you plan to survey) who report having discussed different contraceptive options with their husbands. While this is somewhat different, it might be valid as an indicator and a lot cheaper to collect.
30 Standard IndicatorsWhere possible, a project should select standard indicators.They have been tested for validity and reliability.They allow comparison between projects or sites.They tend to be available for SOs and some IRs.These are useful when project definitions and activities fit with what the standard indicators are measuring. This is more likely at the higher level of the results framework, where there is more consistency across projects and countries. The strategies and some of the IRs may be more project specific and therefore not easily measured by standard indicators.
31 How Many Indicators?Choose at least one or two indicators per intermediate result, as well as the SO for evaluation purposes.Choose one or two indicators per result for program monitoring.Choose indicators that may be able to “cover” more than one element.For management, think about basic activities that you need to monitor to judge if you are implementing activities as planned; include indicators that help you make decisions.Evaluation indicators are the ones that will most likely be drawn from the standard indicators and will be used to draw conclusions across projects. To the extent possible, all the relevant core and standard indicators should be considered.In some cases, by selecting one indicator, it may be possible to “cover” other project results. For example, no product stockouts during the past 6 months would imply that various elements of the logistic system were functioning. As such, it would not be necessary to measure separate elements (e.g., procurement, recordkeeping) unless they were necessary for management and supervision.For another example, if we know that clients are returning to use services, we may assume that counseling for side effects and when to return is in fact occurring. If we are measuring new users as part of routine measurement toward our strategic objective, we do not need to try to collect information about increased knowledge of family planning among the catchment population, as they seem to know about and have an interest in getting services.
32 Exercise: Group work Refer to the indicators you selected.. Were the good indicators ?Did you select a minimum number recommended given the type and size of your program?
34 See comment for examples Sample M&E FrameworkPreventing Post Partum Hemorrhage :Increase Active Management of the Third Stage of LaborResultIndicatorDefinitionData sourceCollection MethodFreq- uencyRespons-ible PartyActive Management of the Third Stage of labor increasedProportion of trained clinicians performing AMTSL to standard# of trained midwives performing all steps of AMTSL on all patients/AMTSL observa- tion checklistClinical observationAnnualZambia JHPIEGO staffThis is one indicator that was developed for the Ukraine Country Coordinating Mechanism (CCM) for their HIV/AIDS workplan submitted to the Global Fund for HIV/AIDS, TB and MalariaSee comment for examples
35 M&E PlanThe plan is a managerial tool that specifies the schedule, resources, responsibilities, for your M&E activities (data collection, data quality control, reporting, dissemination and use of data)Note:The plan should specify the time points when evaluations will be carried out, for example: Midterm, and End term.Outcome/Impact evaluation is reserved for large longer term programs that can make impact at public health status level.Your plan should include activities to monitor and evaluate the implementation, as planned, of the M&E plan.
36 Readiness for evaluation. Not all programs demand M&E at all levels Readiness for evaluation. Not all programs demand M&E at all levels. Will talk about this further during sessions on developing and M&E plan.Source: CDC. Global AIDS program monitoring and evaluation (M&E) field guide
37 QuestionIf funding for your case study program was cut off and the program closed in two years. What changes would you make to you M&E Plan?ANSWER: Couple years of protection (CYP). This indicator was developed because it is not always feasible to do a population-based survey for a very large project. Formulas are used to estimate how many couples were protected from getting pregnant on the basis of the total distribution of different contraceptive methods. It makes the assumption that all contraceptives that are distributed are used, and it assumes a project can capture all contraceptives that are distributed (including through the private sector).