1. Focusing on Development Outcomes in Projects SARAR M&E Workshop 20-21 June 2006 New Delhi, India

2. Outline 1.Management for results as an emerging practice 2.Processes to manage quality 3.Methodology to manage for results – outcome focused design and supervision

3. 1 - Management for Results is an emerging practice within the development community zCompared to Financial Management and Procurement, Management for Results is an emerging practice (within the last 10 years), now rooted in a broader agenda of the development community (Paris Declaration) zManagement for Results has been practiced in recent years under several titles (M&E, management by objectives, balanced scorecard, budget rationalization) zThe emphasis on Management for Results grows out of a need to improve development effectiveness. It will help the Bank and it’s development partners to tell their results story in a more convincing way

4. 2 - Quality Processes z Processes to manage quality z Country level z Portfolio level z Project level

5. The Bank’s view of what needs to be monitored now includes the entire results hierarchy zImpacts zOutcomes zOutputs zActivities zInputs Procurement & Disbursements Deliverables Effectiveness Efficiency Longer-term benefits Results Results Hierarchy

6. Key Events in the Bank related to Results Management zJan ‘96 - Performance indicators become mandatory in all new projects; zFeb ‘97 - Strategic Compact commits the Bank to 100% quality-at-entry by FY2001; zJune ‘97 - Revised supervision reporting system introduced, including indicators monitoring; zAug. ‘97 - Revised project documentation introduced for new operations (including the Logical Framework); z2002 - Results Secretariat unit formed within the Bank z2003 - Logical Framework ‘simplified’ to the Results Framework z2004 - Task Force on Organizational Effectiveness created z2005 – Paris Declaration on Aide effectiveness

7. Magnitude of the Results Management Task Bank-wide z6 Bank regions (AFR, EAP, ECA, LAC, MNA, SAR) z1282 new projects under preparation; z2372 existing projects under supervision; zUS$ 20-25 billion in new lending per year; the Bank also manages US$ 9 billion in trust funds

8. 1 2 3 4 5 6 7 CAS Completion Report Portfolio Performance (ARPP) National Development Strategy Poverty Reduction Strategy Country level Results Cycle

9. SAR Portfolio level quality standard zDefines portfolio management as a management function that seeks to maximize the overall quality of the portfolio by (i) carefully screening new projects prior to entry, and (ii) identifying projects that are performing poorly for early closure or restructuring.

10. Project level standards for SAR work z a. Standard for Quality-at-Entry (Design) yProject Concept Note (PCN) yProject Appraisal Document (PAD) z b. Standard for Project Supervision yProject status report (PSR) yProject completion report (PCR)

11. a. Quality-at-Entry Standard A fully satisfactory project contains a well conceived intervention model (i.e., storyline) which… zis designed to bring about specific, worthwhile and economic outcomes; zembodies a focused and realistic strategy; zcan be expected to document and achieve results (outputs and outcomes) within the time frame and resources allocated to the project.

12. Typical Problems, Quality-at-Entry: zBringing about specific, worthwhile, and economic outcomes: Many projects have difficulty bringing about identifiable outcomes because they are focused on other things: (i) the PDO is too high (i.e., CAS-level, long-term, sweeping, non-specific, non-attributable, unattainable), or (ii) the PDO is too low (i.e., output level, supply-driven, internally controlled by the project, or trivial). zEmbodying a focused and realistic strategy: Many projects lack sufficient focus, realism and clarity with respect to strategy; zDocumenting and achieving results within the time and resources available: Many projects do not document results adequately, and then have difficulty convincing observers that the intended results were achieved.

13. b. Quality in Supervision Standard For existing projects in the portfolio, high quality supervision is characterized by… zproactive management of emerging problems, and zconsistent, honest reporting focused on documenting and achieving the expected development outcomes. High quality supervision is accomplished without extension of the time frame or expansion of the funding available to the project. Quality supervision also meets or exceeds the requirements of the Bank’s current guidelines and procedures for projects under supervision.

14. Typical problems in Supervision: zSupervision plans often lack a strategic vision and a necessary focus on outcomes; zMany projects fail to use key performance indicators as a management tool during supervision; zToo many projects lack a focus on outcomes in supervision reporting (ISR, MTR, ICR)

15. 3 – Focus and methodology z It is important to focus on ‘outcomes’ z What is a focus on outcomes?

16. Terminology: “Results” “Impacts” “Outcomes” “Outputs”

17. It is important to focus on ‘outcomes’ ResultsOutcomes z Output, outcome or impact (intended or unintended, positive and/or negative) of a development intervention. z Specific concept: expected demand-side response needed to validate the causal chain z Success means progressive movement along the causal chain z Provides useful focus to the PDO

18. What is a focus on outcomes? zProvides a convincing storyline (outcome level); zProvides a clear and meaningful vision of success (objective statement); zAnswers the question: What visible changes in behavior can be expected among end users as a result of the project, thus validating the causal chain?

19. Outcomes appear on the demand side of the project, validating the causal chain Inputs Activities Outputs Outcomes Impacts Demand Supply Project Funding ($) Implementing Agency (Internal) (External) Project implementation environment Project Objective Project Deliverables Longer-term goals + -- ICR Efficiency Effectiveness Project Components

20. The previous Logical Framework has been ‘simplified’ to the Result Framework PDO Hierarchy of Objectives Key Performance Indicators (KPIs) Means of Verification Critical Assumptions Impact [IM] Input [IP] Level Outcome [OC] Output [OP] External Internal Result Framework

21. Outcome-focused projects utilize participatory design Output focusOutcome Focus

22. Using a storyline to structure a design concept: Crossing the River of Uncertainties PresentUnsatisfactorySituation FutureVision of Success Results

23. The parts of a storyline Present unsatisfactory situation Future vision of success

24. A dysfunctional storyline fails to provide a consistent approach for the delivery of results Results River of Uncertainties Present Unsatisfactory Situation Future Satisfactory Situation

25. Project Storyline of PCN/PAD ‘Crossing the River of Uncertainty’ A good storyline bridges key areas of uncertainty with a outcome-level approach Results Present Unsatisfactory Situation Future Satisfactory Situation Specify indicators to verify success IdentifyProblem DetermineStrategy Specify Vision of Success [OC] CountryContext SectorContext ProcessEnvironmentReportingEnvironment

26. Appendix: Project examples Rural water Rural health Transport sector

27. Rural water example (causal chain) –Increase in rural productivity and decrease in incidents of poor health caused by water-related diseases –Significant increase in the adoption and use of improved water supply, sanitation and hygiene practices by participating households (objective achieved) –Participating households gain access to completed water supply and sanitation facilities (output delivered) –Target population selects among available options for water supply and sanitation services to meet their needs at affordable prices – Awareness raising activities to promote new integrated approach –Integrated water supply, sanitation and hygiene services are made available at the county level –Capacity building for integrated delivery of water supply, sanitation and hygiene services at the county level Problem: Poor water supply, sanitation and hygiene practices limit the productivity of rural households and often adversely affect health Supply Demand

28. Rural water example storyline Problems being addressed Strategy to address the problems Vision of Success (objective expressed as expected outcome) Evidence of Success (KPIs) Poor water supply, sanitation and hygiene practices [OC] limit the productivity of rural households [IM] and often adversely affect health [IM]. The basic strategy is to expand access to improved water supply, sanitation and hygiene services in participating villages using an integrated three-in-one approach, participatory processes, and county-line agencies as service providers. Total project costs: US$ 75 M Significant increase in the adoption and use of improved water supply, sanitation and hygiene practices by participating households. -Evidence* that participating households have adopted the improved practices [OC], as expected. - Evidence* of reduced time spent in collection of water by households [OC]. ___ * Via independent technical and social audit teams.

29. Rural health example storyline Problems being addressed Strategy to address the problems Vision of Success (objective expressed as expected outcome) Evidence of Success (KPIs) - Use of health services by poor rural households is decreasing in key provinces due to a number of factors [OC]. The basic strategy is to try to make health care more affordable at the point of use through specific initiatives related to health financing, the quality of service available, and public health more generally. Total project costs: US$ 72.5 M To increase the use of health services by poor rural households in targeted provinces [OC]. - Evidence* of reduce percentage of poor households that cite the high out-of-pocket cost of health care as a reason for not accessing health services when needed [OC]. ___ * Via independent technical and social audit teams.

30. Transport sector example storyline Problem(s) being addressed Strategy to address the problem(s) Vision of Success (objective expressed as expected outcome) Evidence of Success (KPIs) Travel in the municipality is constrained by hilly terrain, traffic bottlenecks [OC] and associated slow travel times [OC]. - The project is guided by the municipal development strategy, which aims to emphasize the role of transport in improving core urban functions for the next 5 years. - A total of five infrastructure and TA components are included in the project. - Total project costs: US$ 200 M Traffic bottlenecks removed [OC] in order to improve core urban transport functions [IM] in the municipality. Evidence* that travel times have been reduced and associated traffic bottlenecks eliminated [OC], as expected. ___ * Via independent technical and social audit teams.

31. The End ‘Focusing on Development Outcomes in Projects’ presented by Charles G. Chandler Assumption Analysis, Inc. www.AssumptionAnalysis.com