1. Monitoring and Evaluation: A Logical Frame
Evaluating the Impact of Projects and Programs
Beijing, China, April 10-14, 2006
Shahid Khandker, WBI

2. What is M&E?
Monitoring and evaluation are tools that make it possible to identify and measure the results of projects, programs or policies

3. Why the Concern? To evaluate and adjust strategies and activities
To report on progress to interested parties, clients, taxpayers and the general public
To identify and share with others best practices and lessons learned
To improve the programming of new interventions and strategies

4. Monitoring Provides regular information on how things are working
Defn: A continuing function that uses:
A) systematic data collection and analysis of specific indicators of progress.
B) Provides management with indication of extent of progress towards goals
- achievement of deliverables
-use of resources.
C) Contributes to performance improvement.
D) Conducted by business unit.

5. Evaluation
Evaluation can only be done after a certain time and requires more thorough investigations
Conducted by independent evaluators.
Defn: A systematic and objective measurement of the results achieved by a project/program/policy in order to assess its relevance, the efficiency of implementation, its effectiveness, impact and sustainability.

6. Monitoring vs Evaluation
Monitoring assesses progress in implementation of ongoing programs
Evaluation provides a snapshot against some benchmarks at a point in time of programs that may or may not be continuing
Monitoring looks at progress relative to targets and assumes there is causality
Evaluation seeks to prove causality

7. Monitoring and Evaluation - Rationale
Holds implementers accountable for delivery of inputs
Provides basis for corrective action
Provides assessment of continued relevance
Evaluation
Accountability - was money well spent?
Learning - what could we do better next time?

8. Monitoring and Evaluation, and the Chain of Results
Resources are mobilized…
… to undertake activities...
… the direct results of which...
… must have effects...
This chain is based on a series of logical relationships (if… then) called the logframe
… and an impact on development
Evaluation
Monitoring

9. Logframe of a Project or Program
Resources are mobilised
… to undertake activities...
… the direct results of which...
… must have effects…
… and an impact on development
budget funds
building, training, delivery
amount of services provided, processes completed
Health, literacy
consumption, life expectancy, poverty
Allocation
Inputs
Outcomes
Impact
Outputs

10. What Type of Indicators Are Needed? -MDGs-
Eradicate extreme poverty and hunger
Achieve universal primary education
Promote gender equality and empower women
Reduce child mortality
Improve maternal health
Combat HIV/AIDS, malaria and other diseases
Ensure environmental sustainability
Develop a Global Partnership for Development

11. What Type of Indicators Are Needed
What Type of Indicators Are Needed? GOAL:Improve the health status of the population through improved water supply
Impact
Impact on living standards –
Lower poverty; prevalence of
malnutrition
5-10 yrs
Outcomes
Access, usage, and satisfaction of
beneficiaries - % hh with access to safe water supply; % hh with continuous water supply during the day
1-3 yrs
Outputs
Goods and services generated - Newly installed or rehabilitated water points;% wastewater treated
Annual
Inputs
Resources (financial, material and human) provided - Real investment in water supply and sanitation
Annual

12. From project to program – The new development strategy and its implications for M&E
Inputs
Impact
Outcomes
Outputs
Logframe
Outcome-based M&E
Monitoring the outcome and impact
of development across sectors
Focus on demand-side issues (data for decisions): logframe provides the foundation; largely project/program level: operationally-focused; tend to emphasize M&E at lower end of logframe [inputs/outputs); link to project design/implementation
Reflects a greater emphasis on: country-ownership, long-term capacity-building -also provides greater opportunities for civil society involvement and participatory M&E approaches. Link to PRSP
Two important dimensions to bring together are: i) vertically integrated M&E (based on logframe concepts of input, output, outcome, impact monitoring - the ‘silo’ approach and is very much the domain of the CPPR and ii) horizontally integrated monitoring of outcomes and impacts of national development strategies like PRSPs.
The ‘cross-sectoral) approach. It reflects a greater emphasis on: country-ownership (as against. Bank ownership), outcomes and impacts (instead of inputs and outputs), long-term capacity-building (as against short-term data needs). It also provides greater opportunities for civil society involvement and participative M&E approaches.
The ‘pilots’ should provide learning opportunities. For instance, in Tanzania there is the recently approved Social Fund which has an interesting hybrid M&E system. On the one hand it is establishing standard MIS-based reporting systems (although, even this is being partly fed with information generated out of participacive community-based needs assessment exercises). On the other hand, it is also contributing resources to the NSO to build up capacity to implement CWIQ service delivery surveys that will service the information needs of other programs as well as those of TASAF

13. Why Monitor? To ensure best use of scarce resources
To make user feedback integral to poverty reduction efforts
To mitigate the standard pitfalls in service provision to the poor
To enforce a concrete link between policy goals and policy inputs
To incorporate the many dimensions of poverty reduction into our decisions

14. What to Monitor
Identify few indicators, measure them well, and use the results for policy makers
A prioritized list of input, output, outcome and impact indicators for monitoring
Develop a data collection system to institutionalize a monitoring system: budget and administrative data, facility and other survey data.

15. How to Monitor
Manage a monitoring system that integrates a variety of different types of information: MIS, surveys and censuses, and participatory exercises
Draw up a monitoring matrix which identifies:
-Data sources for each indicator
-Frequency of measurements
-The organization responsible for collecting information.
Collect information in a sequence that maximizes the complementaries between different types of data.
Outputs:
-Annual progress report
-Database for continuous monitoring
-Desk studies

16. Purposes of an M&E Manager, Operational staff People in charge Decide
4 main
purposes
Know,
be informed
Manage, monitor
Researcher
Public,
beneficiaries,
contributors
Know,
understand
and learn lessons

17. For accountability purposes
Key M&E Products
Start
Before
During
At the end
Afterwards
Program
To check the design
To improve the
implementation
For accountability purposes
To assess
the impact

18. Components of a M&E Strategy
Outcome-based monitoring system
Complemented with systematic impact evaluation
Creating a feedback process
Building capacity for monitoring and evaluation
Promoting participation

19. Outcome-based Monitoring System
Setting goals and targets (including establishing the baseline) – where do we want to go?
Identifying indicators that can be used to measure progress towards goals
Collecting data - what progress is being made?
Providing feedback for decision making – what needs to be changed along the way

20. Monitoring System: Where to Start?
Stock taking:
-Who are the actors? What are their activities? What is their capacity? What are their roles and needs?
Review the actions currently undertaken to build or re-enforce capacity.
Review of needs in information
Review of the institutional framework.

21. Monitoring System: What Limitations?
A series of issues are of concern:
- Roles badly defined
- Lack of coordination
- Lack of reliability and relevance of information
- Difficulty in accessing information
- Long delays in production of information
- Lack of use of the data by the users

22. Example: Monitoring Poverty Reduction

23. Monitoring poverty reduction requires
Measuring poverty
Understanding poverty reduction strategies
Monitoring outcomes and impacts—goes beyond project-based monitoring
Evaluating m&e system and its capacity

24. Measuring poverty
Poverty is a multi-dimensional concept—income, consumption, health, education, water, housing, sanitation, capability and empowerment
Poverty is also dynamic—vulnerability and coping mechanism
Poverty is also gender, caste, ethnic, and geographical based

25. Poverty reduction strategies
Market matters; state matters too – they can’t be substitutes; a right balance is required.
Involve the poor in development process—poverty reduction requires both growth and targeted measures
Development strategy needs (a) policies to improve the investment climate to raise income and employment growth, and (b) policies to equip and empower people to participate in growth.

26. Better investment climate and better participation
Macroeconomic stability and openness
Good governance and strong institutions
Quality infrastructure
Growth reduces poverty– but not all growth is equally pro-poor
Empowerment through social, human, and infrastructural investment to enable poor to participate in growth

27. Setting up an M&E system for Poverty Monitoring
Defining information needs. Deciding on indicators
What tools do we need? What tools do we have?
Defining institutional functions (including feedback processes)
Defining a 5-10 year work program
Defining resource requirements

28. Measuring outcomes - What tools do we need? What tools do we have?
Tools for measuring the extent and depth of poverty (poverty analysis)
Tools for monitoring changes over time (poverty dynamics)
Tools for comparing differences over space (poverty mapping)
Participatory poverty monitoring (listening to the poor)

29. Measuring poverty and data quality
Tools: - A good household survey (there are various options - HBS, LSMS etc.)
Measuring poverty is a data hungry exercise (means and distributions)
Household Consumption (If you’re going to measure it - do it properly)
Sampling and non-sampling error
Monitoring over time compounds the error problem

30. Monitoring changes over time
Tools: Administrative data/MIS; Institution-based surveys; household surveys
Monitoring leading indicators (Service Delivery)
Key attributes of an ‘indicators survey
Simple to execute
Rapid reporting
Disaggregatable to low levels
Core Welfare Indicators Questionnaire (CWIQ)

31. Monitoring Spatial/regional differences
Main source of information on distributional outcomes - household surveys - permit only limited disaggregation.
Very large data sources (e.g. census) typically collect very limited information on welfare outcomes.
Impute a preferred measure of welfare (e.g. comprehensive real consumption) from household survey into census, using statistical prediction methods

32. What are Poverty Maps?

33. Participatory Poverty monitoring (listening to the poor)
Limitations of statistical information
Qualitative and participative approaches
Complementary not alternative approaches

34. Data to monitor investment climate and empowerment
Firm and institutional surveys
Good governance indicators
Empowerment indicators
Community participation indicators
Data on quality of infrastructure
Both quantitative and quality information required

35. Key messages
Monitoring poverty reduction goals goes beyond selection of indicators and measuring inputs and outputs
A range of different monitoring tools are required
Poverty reduction monitoring goals should include statistical capacity-building as a core element
Must be a national commitment to funding monitoring and statistical systems
Must ensure existence of appropriate incentives for good M&E?
Feedback into budgeting and policy process is essential