Monitoring and Evaluation in MCH Programs and Projects MCH in Developing Countries Feb 24, 2009

Session objectives : –Explain similarities and differences between monitoring and evaluation –Describe the major purposes, types and tools for MCH program evaluation –Contrast traditional evaluation approaches with participatory evaluation –Describe some practical lessons learned from a recent Timor-Leste program evaluation survey: drawing a representative cluster sample selecting households for a village survey

Objectives of both monitoring & evaluation: Improve –program planning –program management –program performance Assess program effects on beneficiaries Strengthen links with participant groups

Monitoring: Periodic, regular collection and analysis of selected indicators Determines whether key activities are being carried out as planned, and identifies corrections needed Early indication of progress, or a lack of progress Provides periodic oversight of activity implementation

Characteristics of Monitoring Can be carried out by any staff, not a specific evaluation staff person Informs a project about strengths & weaknesses of the strategies: do we need to refine them?

Is Monitoring Supervision?

We monitor an intervention We supervise an individual

Monitoring and Evaluation: how are they related? Monitoring: –the regular collection and analysis of selected indicators –conducted to determine whether key activities are being carried out as planned, and identify corrections needed Evaluation: –done at longer intervals than monitoring –conducted to provide an indication of success of a program, and identify problem areas

Why evaluate??

Characteristics of Evaluation Assesses the value of something Systematic review of evidence – gives a perspective on the ‘reality’ being examined Assess performance and effects of program efforts -- requires analysis  Process: organization and management  Outcome or impact: effects on beneficiaries Carried out selectively – costly and time consuming

Evaluation Provides Information on : Strategy – Are you doing the right things? Operations – Are you doing things right? Learning – Are there better ways?

Types of evaluation: Formative Summative Of relative need for a program Of the feasibility of a program design Of program performance or process Of direct effects or outcomes of a program Of long-term impact Internal or external

Internal Evaluation Staff know more ( history, organization, culture, problems, successes ) Can interpret findings more accurately Known to the program so less likely to be threatening, disruptive but: May be too close, hard to be ‘objective’ Part of power structure, may have personal agendas Likely not highly trained in evaluation

External evaluators Can take a ‘fresh’ look at the program Not personally involved; more ‘objective,’ not a part of the normal power structure Less chance of personal gain Well trained in evaluation methods, seen as ‘expert’ but: May not understand the program or people or setting involved May cause anxiety if not known and trusted

So – which to conduct: Internal or External evaluation?

The best of both: Participatory Evaluation Distinction between experts and layperson is de-emphasized and redefined Focus on –program participants, not donors –learning, not just accountability Flexible design, not predetermined Consultant is a facilitator, not the actual ‘evaluator’ Evaluation team is key to approach

What does an evaluation team do? Preparation: Establish purpose, methods (‘scope of work’) including the evaluation questions Develop data gathering instruments Develop team plan and itinerary Main activities: Visit main offices, field sites Gather data (document review; observations, group discussions, interviews; survey report)

Report preparation: Team discussion of findings, come to tentative conclusions and recommendations Draft main findings of the evaluation report Discuss draft of findings with program staff, other stakeholders Revise findings, recommendations as needed Finalize and submit report Share with staff, community, MOH, others

When in doubt, observe and ask questions. When certain, observe at length and ask many more questions! From Halcolm’s Evaluation Laws