Presentation is loading. Please wait.

Presentation is loading. Please wait.

11 Results-based Monitoring and Evaluation in Bank projects HD Learning Week. November 6, 2006 Jody Zall Kusek & Mohamed Khatouri World Bank.

Similar presentations


Presentation on theme: "11 Results-based Monitoring and Evaluation in Bank projects HD Learning Week. November 6, 2006 Jody Zall Kusek & Mohamed Khatouri World Bank."— Presentation transcript:

1 11 Results-based Monitoring and Evaluation in Bank projects HD Learning Week. November 6, 2006 Jody Zall Kusek & Mohamed Khatouri World Bank

2 22 Session I- Overview of Results-Based Management Concepts Traditional Management Approach Focused mainly on inputs and activities. Results-Based Management Approach Focus on the results obtained rather than just on the inputs used or the activities conducted.  Managing for results. Use the information to improve decision making and steer development interventions towards clearly defined goals.  Results agenda. Focus on development outcomes to be achieved by Bank operations, which requires high quality project design and supervision. 1.Better indicators + Monitoring = Better results 2. High quality Project design (specific focus on outcomes) + High quality implementation = Better results Which statement is accurate ?

3 33 Socio- economic context: Results statement should reflect local needs and priorities Critical factors for defining results to be achieved Resources:Level of resources will impact on what can realistically be achieved Local Capacity: Existing skills, leadership, and management capacity will impact on what can be implemented to achieve expected results Timetable: Results framework must identify the results (changes) to be achieved in the life of the program Session I- Overview of Results-Based Management Concepts

4 44 Principles of Management for Development Results* Tools for performance measurement to increase the effectiveness of development interventions * Roundtable on Managing for Results in 2004 1. Focusing the dialogue on results at all phases of the development process 2. Aligning programming, M&E with results 5. Using results information for learning and decision making 4. Managing for, not by, results 3. Keeping measurement & reporting simple & cost-effective Results-based M&E Session I- Overview of Results-Based Management Concepts

5 55  Ensuring accountability  Improving internal management  Marketing successes Results are more important than processes: provide evidence that the program is producing longer term benefits. Focusing interventions/reallocating inputs to achieve results. Programs focused are better able to demonstrate results over time. Showing general program progressCapturing lessons-learned: Showing general program progress & sharing practices and innovations. Preoccupation with results is a global issue : demonstrate results against money spent Why is Results-Based M&E Important? Secure political & public support: “ If you can demonstrate results, you can win public support “ Better reporting: Programs have effective M&E system develop results-oriented reporting

6 66 Session I- Overview of Results-Based Management Concepts Regular collection and reporting of information to track whether actual results are being achieved as planned  Clarifies program objectives  Link inputs and activities to results to be achieved  Translates results to be achieved into performance indicators  Periodically collect data on the indicators and compare actual results with target  Reports progress and alerts management to problems in implementation Indicator Year 30% 40% 50% 12345 20 % Monitoring

7 77  Analyses why intended results were or were not achieved  Explores unintended results  Assesses causal contributions of activities to results and validate hypothesis  Examines implementation processes Analytical efforts to answer specific questions about performance of a program activities. Oriented to answering WHY? And HOW?  Provides lessons learned and recommendations for improvement With project With out project Indicator Year 30% 40% 50% 12345 15 % Evaluation Session I- Overview of Results-Based Management Concepts

8 88 Outcomes Outputs Long-term, widespread improveme nt in society Effects or behavior changes resulting from program outputs Products and services to be used to simulate the achievement of results Utilization of resources to generate products and services Resources committed to program activities Implementation Results Long-term Goal (Impact) ActivitiesInputs PLANING FOR RESULTS Attribution Gap Traditional M&E Results-based M&E Session I- Overview of Results-Based Management Concepts Results Building Blocs

9 99 Social Protecti on and labor  Increased coverage of crop-insurance programs Increased % of vulnerable population using saftey net programs Increased population is food secure Health  Doctors hired  Health workers trained Increased use of health clinics Improved maternal mortality Education  Teachers trained  Text Books provided Increased student completion rates Increase literacy rates Examples of Results Chain Long-Term Goal (Impact) OutcomesOutputs Session I- Overview of Results-Based Management Concepts

10 10 Session I- Overview of Results-Based Management Concepts 1. Information is available for parents about the importance of breast feeding 2. Children in Local Community healthier 3. Fewer children are having diarrhea diseases 4. Mothers Parents breast feeding rather than using formula 5. New funds available to implement a health project to reduce babies mortality rates 6. Implement information campaigns on the importance of breast feeding Example of Results Chain (Health Sector) Identify the sequence of inputs, activities, outputs, outcomes, and impacts

11 11  Responsive to  Responsive to information management and policy making of government institutions at the central and local level  M&E capacity in place for collecting, analyzing, and reporting performance information  Built around reporting requirements  Active involvement of customers and partners (planning, conducting, reviewing, & interpreting performance information)  Fully integrated into management systems  Fully integrated into the government’s existing management systems at the central and local level Critical Elements of an Effective M&E System Session I- Overview of Results-Based Management Concepts

12 12 Country’s National Poverty Reduction Strategy (PRSP) Provides a central framework for donor coordination PRSP Strategic Goals Outcomes Country Assistance Strategy (CAS) to support the national development strategy Country Assistance Strategy (CAS) to support the national development strategy CAS Operationalize the CAS Operationalize the CAS AAA Policy Dialogue Operations Projects/Programs Align Projects and Programs Results with CAS & PRSP Results Design Logic of Bank Operations Session I- Overview of Results-Based Management Concepts

13 13 Exercise 1. Develop Results Building Blocs Assignment Break out into groups of 4-5 persons Each group should develop a results building bloc (inputs, outputs, outcomes, & impacts) for at least 2 programs. Spend no more than 15 minutes per program Select one case to report on in the plenary session. Session I- Overview of Results-Based Management Concepts

14 14 Session II- Development of Results Framework Identifies the Program Development Objective (PDO) and Intermediate Outcomes which are critical to achieve it  Program’s strategy for achieving specific objective Illustrates the development hypothesis and the cause & effect relationships linking all levels of a program’s objectives (IF a and b happen THEN c should happen) Building consensus and ownership  An effective tool for: Strategic clarity and planning & managing for results Effective communication Results Framework

15 15 Session II- Development of Results Framework Reduction in the child mortality 1. Determine the key development hypothesis & its contribution to higher order impacts THEN incidence of preventable disease will be reduced  Causal Chain Hypothesis. Results are caused by some interventions IF… THEN  IF we build health centers THEN local population will use them to for health services  IF we train teachers on new pedagogic skills THEN teachers will use these skills to improve the quality of education IF expand child immunizations  Design with clear logic

16 16 2. Develop a hierarchy of objectives showing the causal linkages Reduction in Child Mortality Expanded health care system Improved nutrition Provide Funds Incidence of preventable disease reduced Increased immunizations Increased family health practices Expand facilities Public awareness 3. Determine what is possible in the operation (program or project) Vaccines availability  DESIGN. Project success depends on the development of the right activities & assumptions  IMPLEMENTATION. Each part of the results chain has a role to play in achieving the Project Development Objective  Design with clear logic (Cont.) Session II- Development of Results Framework Enhance regulations

17 17 Session II- Development of Results Framework Program Development Objective (PDO) The PDO describes the effect that the program ’ s outputs will have on the beneficiaries in terms of changed behavior or improved performance. It defines the program ’ s success.  The PDO outcomes are one step beyond the products of the activities. At the close of the program, what problem has been solved for the key recipient of the program outputs ? PDO example: “To achieve improved, expanded, sustainable utilization of basic social and economic services and to support a governance system where local government and communities can gradually become mutually accountable. Is the PDO precise ?  Focus on the outcome that the project can directly influence, given its duration, resources, and approach  Focus on the expected outcome for the primary beneficiaries  Keep the PDO statement clear, precise, and stated as realistic results— not overly broad or too ambitious

18 18 Example 1: To provide grant financing to educate “at risk” groups in the risks of HIV/AIDS. Example 2: To increase the awareness of the dangers of HIV/AIDS of “at risk” groups. Which outcome example describe what is going to be transformed?  PDO Outcomes. Capture single outcome by outcome statement Example: PDO objective To expand access and improve quality of primary education throughout the country. Outcomes to be achieved ? ……………… Program Development Objective (Cont.) Session II- Development of Results Framework  Focus on what is going to be transformed (not what is going to be done)

19 19  Avoid selection of too many indicators Less is Better!  Work with stakeholders and technical experts to determine which indicators are most important 7  Information about indicators should be easy to gather and useful for management decisions  Performance indicators should be identified at all levels of results chain. Indicator: A variable that provides accurate and reliable evidence about the achievement of a specific result Indicator: A variable that provides accurate and reliable evidence about the achievement of a specific result o“What gets measured gets done” o“If you don’t measure results, you can’t tell success from failure” Performance Indicators Session II- Development of Results Framework

20 20 8 Specific : Specific : measure as closely as possible what is intended to measure Measurable : Measurable : clear and unambiguous about what is measured Outcome Indicator Parents insure that 1.Increased utilization of clinics Children get treatment for 2. Increased use of malaria Malaria drugs Which indicator is more specific ? 2. % of health centers with availability of drugs 1.% of health centers without stock out of drugs x, y & z for more than a week at a time Which indicator is measurable ? Performance indicators should be SMART : Specific; Measurable; Attributable; Realistic; Targeted Criteria for Selecting Good Indicators Session II- Development of Results Framework

21 21 2. HIV prevalence among the total population 1. HIV prevalence among 15-24 year old pregnant women 1. Percent increase in employment 8 Realistic : Realistic : data obtained at reasonable cost with enough frequency Which indicator is more realistic? Targeted : Targeted : it should be specific about the targeted population/area 2. Percent increase in employment of graduate of technical training center X in the first year after completion of training. Which indicator is targeted ? 1. Life expectancy Attributable : attributable to the project ’ s or program ’ s efforts 2. % of children fully immunized at 1 year Which indicator is attributable ? Criteria for Selecting Good Indicators (Cont.) Session II- Development of Results Framework

22 22 PDO Outcomes Outcomes Indicators Intermediate Outcomes by Component Intermediate Outcome Indicators Increased coverage of clinics providing immuniatio PDO: Increased use of health facilities Improved quality of doctors/nurses % of under 5 yr covered by DPT immunization Component I- Extension of Quality Health Services % of facilities without 7-day stock outs of essential drugs (List of essential drugs defined) Sustained availability of essential drugs Component II. Development of Human resources Reduced shortage of human resources % of facilities with minimum staffing norms (List of minimum staffing defined) % of health deliveries carried out in public health facilities (%) Session II- Development of Results Framework EXAMPLE

23 23 Definition. Clear and precise definition of what will be measured and unity of measure. Definition. Clear and precise definition of what will be measured and unity of measure. Critical Assumptions. External factors that could significantly affect the achievement of the program results targets Critical Assumptions. External factors that could significantly affect the achievement of the program results targets 7 Baseline & Targets Baseline & Targets 4 Baseline. Value of the indicator at the beginning of the program. Used as a point of comparison when measuring progress toward a specific result 4 Target values. The intended value of the indicator at the end of a specified point in time, against which actual results will be measured Data Acquisition Method. Data source, frequency/schedule, and responsibility for its collection Data Acquisition Method. Data source, frequency/schedule, and responsibility for its collection Data Analysis & Reporting Method. Frequency/schedule, Analysis method, and responsibility of reporting. Data Analysis & Reporting Method. Frequency/schedule, Analysis method, and responsibility of reporting. Each Indicator must have a M&E plan Session II- Development of Results Framework

24 24 Indicators (Definition % unit) Base Line & Target Values (/Year) BL 1 2 3 4 Data Collection & Reporting Frequency /Schedule Instrument (Method) Responsible Party % of health deliveries carried out in public health facilities (%): 2528303540Annually (June) Routine administrati ve records Ministry of Health 10 Example of a M&E plan for one indicator Session II- Development of Results Framework

25 25 Impacts Interventions High level outcomes Processes, Policies, & Outputs Example: Provide essential obstetrical health services Example: Reduced maternal mortality Intermediate outcomes which the program is expect to influence directly through its defined interventions during the program period. Intermediate Outcomes Ways to Improve the development of Results Frameworks Example: Birth attended by skilled staff Session II- Development of Results Framework

26 26 Health services provided ( Heath workers trained, Health facilities built & rehabilitated, Pharmaceutical drugs purchased) Reduced Maternal Mortality Example: Health Sector Hygiene and sanitation practices Increased use of maternal health services Improved access to family planning Increased adoption of prevention methods Availability of ITNs Provision of contracep- tives and counseling to women and youth Increased coverage of Antenatal, new born, emergency obstetric, & post-natal care Improved Communi- cation networks & ambulances Information on family planning available Session II- Development of Results Framework

27 27 Key Informant Interviews Community Interviews Review of official records One-Time Survey Panel Surveys Impact Evaluation (Experimental Design) Census Statistical Emphasis Rigor/Difficulty (Cost/Time/skills ) Focus Group Interviews Formal Methods Rapid Appraisal Methods Direct Observation Validity Reliability Credibility  Choice depends 1. Issue to be examined 2. Quality of the information needed 3. Time frame in which information is needed 4. Cost Data Collection Methods Session II- Development of Results Framework

28 28 Key Steps for planning an Evaluation 1. Clarify the evaluation purpose and audience 2. Identify the evaluation questions 3. Select appropriate methods based on the questions to answer a.What % of women using health facilities? b.Why didn ’ t more parents send their kids to school? c.Did the CDD project contribute to the increase in income of local communities? Example. Which data collection method is more appropriate in providing an answer to each of the following questions? 4. Prepare data collection and analysis plan Session II- Development of Results Framework

29 29 7 Design/PAD Develop the results framework with SMART indicators Develop the M&E plan with baselines & targets PCN Develop causal chain Define appropriate PDO and outcomes to be achieved Completion (ICR) Review and validate the causal chain Report on the Project outcomes Implementation (ISR) -Report on outputs & outcomes -Adjust project design (as necessary to achieve outcomes) Focus on outcomes all phase of the project cycle Session II- Development of Results Framework

30 30 Assignment  Break out into groups of 4-5 person  Review the quality of the PDO statement and its corresponding outcome indicators for 2 projects  Report on one case in the plenary session Exercise 2. Assess the quality of the Project Development Objectives (PDO) & outcome indicators of selected projects Session II- Development of Results Framework


Download ppt "11 Results-based Monitoring and Evaluation in Bank projects HD Learning Week. November 6, 2006 Jody Zall Kusek & Mohamed Khatouri World Bank."

Similar presentations


Ads by Google