Presentation on theme: "Results-based Monitoring and Evaluation in Bank projects"— Presentation transcript:
1 Results-based Monitoring and Evaluation in Bank projects HD Learning Week. November 6, 2006Jody Zall Kusek & Mohamed KhatouriWorld Bank
2 Traditional Management Approach Session I- Overview of Results-Based Management ConceptsManaging for results. Use the information to improve decision making and steer development interventions towards clearly defined goals.Results-Based Management ApproachFocus on the results obtained rather than just on the inputs used or the activities conducted.Traditional Management ApproachFocused mainly on inputs and activities.Better indicators + Monitoring = Better resultsWhich statement is accurate ?2. High quality Project design (specific focus on outcomes) + High quality implementation = Better resultsResults agenda. Focus on development outcomes to be achieved by Bank operations, which requires high quality project design and supervision.
3 Critical factors for defining results to be achieved Session I- Overview of Results-Based Management ConceptsCritical factors for defining results to be achievedSocio- economic context:Results statement should reflect local needs and prioritiesLocal Capacity: Existing skills, leadership, and management capacity will impact on what can be implemented to achieve expected resultsResources: Level of resources will impact on what can realistically be achievedTimetable: Results framework must identify the results (changes) to be achieved in the life of the program
4 Principles of Management for Development Results* Session I- Overview of Results-Based Management ConceptsPrinciples of Management for Development Results*1. Focusing the dialogue on results at all phases of the development process5. Using results information for learning and decision making2. Aligning programming, M&E with results4. Managing for, not by, results3. Keeping measurement & reporting simple & cost-effectiveTools for performance measurement to increase the effectiveness of development interventionsResults-based M&E* Roundtable on Managing for Results in 2004
5 Why is Results-Based M&E Important? Session I- Overview of Results-Based Management ConceptsWhy is Results-Based M&E Important?Improving internal managementFocusing interventions/reallocating inputs to achieve results. Programs focused are better able to demonstrate results over time.Better reporting: Programs have effective M&E system develop results-oriented reportingMarketing successesCapturing lessons-learned: Showing general program progress & sharing practices and innovations.Secure political & public support: “If you can demonstrate results, you can win public support “Ensuring accountabilityPreoccupation with results is a global issue : demonstrate results against money spentResults are more important than processes: provide evidence that the program is producing longer term benefits.
6 Monitoring Session I- Overview of Results-Based Management Concepts Clarifies program objectivesMonitoringLink inputs and activities to results to be achievedTranslates results to be achieved into performance indicatorsPeriodically collect data on the indicators and compare actual results with targetRegular collection and reporting of information to track whether actual results are being achieved as plannedReports progress and alerts management to problems in implementationIndicatorYear30%40%50%1234520 %
7 Evaluation Session I- Overview of Results-Based Management Concepts Analyses why intended results were or were not achievedEvaluationExplores unintended resultsAssesses causal contributions of activities to results and validate hypothesisExamines implementation processesAnalytical efforts to answer specific questions about performance of a program activities.Oriented to answering WHY? And HOW?Provides lessons learned and recommendations for improvementWith projectWith out projectIndicatorYear30%40%50%1234515 %
8 Long-term Goal (Impact) Session I- Overview of Results-Based Management ConceptsResults Building BlocsAttribution GapPLANING FOR RESULTSLong-term Goal (Impact)OutcomesOutputsActivitiesInputsEffects or behavior changes resulting from program outputsLong-term, widespread improveme nt in societyProducts and services to be used to simulate the achievement of resultsUtilization of resources to generate products and servicesResources committed to program activitiesResultsImplementationTraditional M&EResults-based M&E
9 Examples of Results Chain Session I- Overview of Results-Based Management ConceptsExamples of Results ChainLong-Term Goal (Impact)OutcomesOutputsIncrease literacy ratesTeachers trainedText Books providedIncreased student completion ratesEducationImproved maternal mortalityIncreased use of health clinicsDoctors hiredHealth workers trainedHealthIncreased population is food secureIncreased % of vulnerable population using saftey net programsIncreased coverage of crop-insurance programsSocial Protection and labor
10 Session I- Overview of Results-Based Management Concepts Identify the sequence of inputs, activities, outputs, outcomes, and impactsExample of Results Chain (Health Sector)Information is available for parents about the importance of breast feedingChildren in Local Community healthierFewer children are having diarrhea diseasesMothers Parents breast feeding rather than using formulaNew funds available to implement a health project to reduce babies mortality ratesImplement information campaigns on the importance of breast feeding
11 Session I- Overview of Results-Based Management Concepts Critical Elements of an Effective M&E SystemActive involvement of customers and partners (planning, conducting, reviewing, & interpreting performance information)Fully integrated into the government’s existing management systems at the central and local levelResponsive to information management and policy making of government institutions at the central and local levelBuilt around reporting requirementsM&E capacity in place for collecting, analyzing, and reporting performance information
12 Design Logic of Bank Operations Session I- Overview of Results-Based Management ConceptsDesign Logic of Bank OperationsCountry’s National Poverty Reduction Strategy (PRSP) Provides a central framework for donor coordinationPRSPStrategic GoalsOutcomesAlign Projects and Programs Results withCAS & PRSP ResultsCountry Assistance Strategy (CAS) to support the national development strategyCASAAAPolicy DialogueOperations Projects/ProgramsOperationalize the CAS
13 Exercise 1. Develop Results Building Blocs Session I- Overview of Results-Based Management ConceptsExercise 1. Develop Results Building BlocsAssignmentBreak out into groups of 4-5 personsEach group should develop a results building bloc (inputs, outputs, outcomes, & impacts) for at least 2 programs. Spend no more than 15 minutes per programSelect one case to report on in the plenary session.
14 Session II- Development of Results Framework Program’s strategy for achieving specific objectiveIdentifies the Program Development Objective (PDO) and Intermediate Outcomes which are critical to achieve itIllustrates the development hypothesis and the cause & effect relationships linking all levels of a program’s objectives(IF a and b happen THEN c should happen)An effective tool for:Strategic clarity and planning & managing for resultsBuilding consensus and ownershipEffective communication
15 Session II- Development of Results Framework Causal ChainHypothesis. Results are caused by some interventions IF… THENIF we build health centersTHEN local population will use them to for health servicesIF we train teachers on new pedagogic skillsTHEN teachers will use these skills to improve the quality of educationDesign with clear logicReduction in the child mortality1. Determine the key development hypothesis & its contribution to higher order impactsTHEN incidence of preventable disease will be reducedIF expand child immunizations
16 Session II- Development of Results Framework Design with clear logic (Cont.)2. Develop a hierarchy of objectives showing the causal linkages3. Determine what is possible in the operation (program or project)Reduction in Child MortalityExpanded health care systemImproved nutritionIncidence of preventable disease reducedIncreased family health practicesIncreased immunizationsDESIGN. Project success depends on the development of the right activities & assumptionsVaccines availabilityPublic awarenessEnhance regulationsIMPLEMENTATION. Each part of the results chain has a role to play in achieving the Project Development ObjectiveExpand facilitiesProvide Funds
17 Session II- Development of Results Framework Program Development Objective (PDO)The PDO describes the effect that the program’s outputs will have on the beneficiaries in terms of changed behavior or improved performance. It defines the program’s success.Focus on the outcome that the project can directly influence, given its duration, resources, and approachFocus on the expected outcome for the primary beneficiariesAt the close of the program, what problem has been solved for the key recipient of the program outputs ?The PDO outcomes are one step beyond the products of the activities.Keep the PDO statement clear, precise, and stated as realistic results— not overly broad or too ambitiousPDO example: “To achieve improved, expanded, sustainable utilization of basic social and economic services and to support a governance system where local government and communities can gradually become mutually accountable.Is the PDO precise ?
18 Session II- Development of Results Framework Program Development Objective (Cont.)Focus on what is going to be transformed (not what is going to be done)Which outcome example describe what is going to be transformed?Example 1: To provide grant financing to educate “at risk” groups in the risks of HIV/AIDS.Example 2: To increase the awareness of the dangers of HIV/AIDS of “at risk” groups.PDO Outcomes. Capture single outcome by outcome statementExample: PDO objectiveTo expand access and improve quality of primary education throughout the country.Outcomes to be achieved ?……………… ………………
19 Session II- Development of Results Framework Performance Indicators“What gets measured gets done”“If you don’t measure results, you can’t tell success from failure”Indicator: A variable that provides accurate and reliable evidence about the achievement of a specific resultWork with stakeholders and technical experts to determine which indicators are most importantAvoid selection of too many indicators Less is Better!Information about indicators should be easy to gather and useful for management decisionsPerformance indicators should be identified at all levels of results chain.7
20 Specific: measure as closely as possible what is intended to measure Session II- Development of Results FrameworkCriteria for Selecting Good IndicatorsPerformance indicators should be SMART:Specific; Measurable; Attributable; Realistic; TargetedSpecific: measure as closely as possible what is intended to measureWhich indicator is more specific ?Outcome IndicatorParents insure that Increased utilization of clinicsChildren get treatment for Increased use of malariaMalaria drugsMeasurable: clear and unambiguous about what is measured% of health centers without stock out of drugs x, y & z for more than a week at a timeWhich indicator is measurable ?2. % of health centers with availability of drugs8
21 Attributable: attributable to the project’s or program’s efforts Session II- Development of Results FrameworkCriteria for Selecting Good Indicators (Cont.)Attributable: attributable to the project’s or program’s effortsWhich indicator is attributable ?1. Life expectancy2. % of children fully immunized at 1 yearRealistic: data obtained at reasonable cost with enough frequency1. HIV prevalence among year old pregnant womenWhich indicator is more realistic?2. HIV prevalence among the total populationTargeted: it should be specific about the targeted population/areaWhich indicator is targeted ?1. Percent increase in employment2. Percent increase in employment of graduate of technical training center X in the first year after completion of training.8
22 Intermediate Outcomes by Component Intermediate Outcome Indicators Session II- Development of Results FrameworkEXAMPLEPDO OutcomesOutcomes IndicatorsPDO: Increased use of health facilities% of under 5 yr covered by DPT immunizationIncreased coverage of clinics providing immuniatio% of health deliveries carried out in public health facilities (%)Improved quality of doctors/nursesIntermediate Outcomes by ComponentIntermediate Outcome IndicatorsComponent I- Extension of Quality Health Services% of facilities without 7-day stock outs of essential drugs (List of essential drugs defined)Sustained availability of essential drugsComponent II. Development of Human resourcesReduced shortage of human resources% of facilities with minimum staffing norms (List of minimum staffing defined)
23 Session II- Development of Results Framework Each Indicator must have a M&E planDefinition. Clear and precise definition of what will be measured and unity of measure.Baseline & TargetsBaseline. Value of the indicator at the beginning of the program. Used as a point of comparison when measuring progress toward a specific resultTarget values. The intended value of the indicator at the end of a specified point in time, against which actual results will be measuredData Acquisition Method. Data source, frequency/schedule, and responsibility for its collectionData Analysis & Reporting Method. Frequency/schedule, Analysis method, and responsibility of reporting.Critical Assumptions. External factors that could significantly affect the achievement of the program results targets7
24 Session II- Development of Results Framework Example of a M&E plan for one indicatorIndicators(Definition % unit)Base Line & Target Values (/Year)BLData Collection & ReportingFrequency/ScheduleInstrument (Method)Responsible Party% of health deliveries carried out in public health facilities (%):2528303540Annually (June)Routine administrative recordsMinistry of Health10
25 Session II- Development of Results Framework Ways to Improve the development of Results FrameworksHigh level outcomesImpactsExample:Reduced maternal mortalityIntermediate outcomes which the program is expect to influence directly through its defined interventions during the program period.Intermediate OutcomesDefineMissing MiddleOutcomesExample:Birth attended by skilled staffProcesses, Policies, & OutputsInterventionsExample:Provide essential obstetrical health services
26 Example: Health Sector Session II- Development of Results FrameworkExample: Health SectorReduced Maternal MortalityHygiene and sanitation practicesIncreased use of maternal health servicesImproved access to family planningIncreased adoption of prevention methodsAvailability of ITNsProvision of contracep-tives and counseling to women and youthIncreased coverage of Antenatal, new born, emergency obstetric, & post-natal careImproved Communi-cation networks & ambulancesInformation on family planning availableHealth services provided ( Heath workers trained, Health facilities built & rehabilitated, Pharmaceutical drugs purchased)
27 Session II- Development of Results Framework Data Collection MethodsRigor/Difficulty (Cost/Time/skills)Formal MethodsCensusImpact Evaluation (Experimental Design)PanelSurveysDirect ObservationRapid Appraisal MethodsOne-Time SurveyCommunity InterviewsReview of official recordsFocus GroupInterviewsValidity Reliability CredibilityKey Informant Interviews1. Issue to be examinedStatistical EmphasisChoice depends2. Quality of the information needed3. Time frame in which information is needed4. Cost
28 Session II- Development of Results Framework Key Steps for planning an Evaluation1. Clarify the evaluation purpose and audience2. Identify the evaluation questions3. Select appropriate methods based on the questions to answer4. Prepare data collection and analysis planExample. Which data collection method is more appropriate in providing an answer to each of the following questions?What % of women using health facilities?Why didn’t more parents send their kids to school?Did the CDD project contribute to the increase in income of local communities?
29 Session II- Development of Results Framework Focus on outcomes all phase of the project cyclePCNDevelop causal chainDefine appropriate PDO and outcomes to be achievedCompletion (ICR)Review and validate the causal chainReport on the Project outcomesDesign/PADDevelop the results framework with SMART indicatorsDevelop the M&E plan with baselines & targetsImplementation (ISR)Report on outputs & outcomesAdjust project design (as necessary to achieve outcomes)7
30 Session II- Development of Results Framework Exercise 2. Assess the quality of the Project Development Objectives (PDO) & outcome indicators of selected projectsAssignmentBreak out into groups of 4-5 personReview the quality of the PDO statement and its corresponding outcome indicators for 2 projectsReport on one case in the plenary session