1. Introduction to Monitoring and Evaluation

2. Learning Objectives By the end of the session, participants will be able to: Define program components Define key concepts in monitoring and evaluation (M&E) Know the basic purpose of M&E 2

3. Program Components Inputs  Processes  Outputs  Outcome  Impact 3

4. Program Components: Inputs Inputs: Resources used in a program/project, such as money, staff, curricula, and materials Examples: –Funding: Government, World bank, Global Fund, PMI & other donors –Commodities: ACTs, ITNs, RDTs, laboratory supplies 4

5. Program Components: Processes Processes: Activities/services that the program/project undertakes/provides to accomplish its objectives, such as outreach, materials distribution, workshops, and training –Ex. Training of health personnel and program staff 5

6. Program Components: Outputs Outputs: Direct products or deliverables of the program, such as intervention sessions completed, people reached, and materials distributed; direct deliverables of the service delivery environment, such as access, quality, and client satisfaction. Examples: –Number of ITNs distributed –Number of houses sprayed with IRS –Number of ACTs distributed 6

7. Program Components: Outcomes Outcomes: Program results that occur at the population level both immediately and some time after the activities are completed, such as changes in knowledge, attitudes, beliefs, skills, behaviors Examples: –ITN coverage and use –Knowledge about ACTs –Adherence to treatment strategy (usage of combination therapies rather than monotherapies) 7

8. Program Components: Impacts Impacts: Long-term results of one or more programs over time, such as changes in disease incidence, fertility, morbidity, and mortality rates –Changes in malaria morbidity and mortality 8

9. Example from an Malaria Program Project / Program Level Population Level Resources Staff Drugs, LLINs, RDTS Supplies Equipment Activities: Trainings, events Distribution of supplies Services: ACTs and LLINs distributed, Staff trained Intermediate Coverage of interventions Impact Inputs Processes/ Activities Outputs Outcomes Long-term Morbidity Mortality 9

10. Brainstorming Activity What is monitoring? What is evaluation? 10

11. Definition: Monitoring and Evaluation MONITORING is a process of measuring progress towards program/project objectives through tracking activities conducted, resource utilization, and the outputs generated EVALUATION is a process of determining systematically and objectively the relevance, effectiveness and impact of interventions in relation to their objectives 11

12. Monitoring vs. Evaluation Monitoring is tracking program/ project deliverables Focus on performance of the implementing agencies Evaluation is assessing the effectiveness of the program/project Focus on the results expected to be achieved in the target population 12

13. Changes occur with adequate intervention efforts and time Status at the start of program Time since intervention began: more time …bigger changes Increased complexity of study design 13

14. Illustration of Program Monitoring Program start Program end Program Indicator: # of ITNs Distributed Time 14

15. Outcome Evaluation Program Start Time Program End With program Program Indicator: ITN Ownership 15

16. Impact Evaluation With program Without program Attribution “Impact” Program Indicator: Inpatient malaria deaths 16 Program Start Time Program End

17. Differentiating Actual Impact from Bias Intervention start End TIME-> Outcome Actual program effect Bias Overall change Outcome With program (Impact) Outcome Without program 17

18. Monitoring & Evaluation Pipeline Some All Most Few Levels of Monitoring & Evaluation Efforts Number of Projects Measuring each Component Resources Staff Funds Materials Facilities Supplies ITNs distributed Houses sprayed Staff trained ITN coverage and use IRS coverage Diagnostic and Treatment coverage Inputs Outputs Short-term and intermediate effects Long-term effects Outcomes Impact 18 Morbidity Mortality

19. Role of M&E in Program Life Cycle 19

20. Monitoring Questions Were inputs made available to program/ project in the quantities and at the time specified by the program/project work plan? Were the scheduled activities carried out as planned? How well were they carried out? Did the expected changes occur at the program/project level, in terms of people reached, materials distributed? 20

21. Evaluation Questions Did the expected change occur at the population level (not necessarily attributable to program/project)? How much change occurred? Did the target population benefit from the program and at what cost? Impact Evaluation: Can improved health outcomes be attributed to program efforts? 21

22. When should evaluations be planned? Before programs are implemented 22

23. Class Activity: Is it Monitoring or Is it Evaluation? 1.The MOH wants to know if the programs being carried out in Region A are increasing ITN use among pregnant women and children under five in that region 2.The Government wants to know how many RDT have been used in health facilities for a year 3.The NMCP would like to see if there have been changes in the under five mortality rate since the implementation of their LLIN campaign 3 years ago 23

24. Class Discussion: Is it Impact Evaluation? 1.The Government wants to know if its campaign to distribute LLINs has reduced the prevalence of malaria in the distribution districts 2.The WHO wants to know if the all cause under-five mortality rate has increased or decreased in Country X since the year 2000 24

25. Summary Monitoring Routine & continuous Internal to program Regular Measures actual performance Tracks cost Done by those in program Evaluation Time bound External or internal Periodic assessement Impact Evaluation Evidence of changes due to program Rigorous and requires a design 25

26. References Patton, Michael Quinn. 1997. Utilization-Focused Evaluation. Thousand Oaks: Sage Publications MEASURE Evaluation, Addis Continental institute of Public Health. Introduction to M&E. Workshop on M&E of PHN Programs. January, 2012 Addis Ababa. Available at : https://www.cpc.unc.edu/measure/training/materials https://www.cpc.unc.edu/measure/training/materials 26

27. MEASURE Evaluation is a MEASURE program project funded by the U.S. Agency for International Development (USAID) through Cooperative Agreement GHA-A-00-08-00003-00 and is implemented by the Carolina Population Center at the University of North Carolina at Chapel Hill, in partnership with Futures Group International, John Snow, Inc., ICF Macro, Management Sciences for Health, and Tulane University. Visit us online at http://www.cpc.unc.edu/measure. 27