Presentation on theme: "Social Development Department The World Bank Participatory Approaches in Impact Evaluation Asli Gurkan Social Development Department World Bank Dubai –"— Presentation transcript:
Social Development Department The World Bank Participatory Approaches in Impact Evaluation Asli Gurkan Social Development Department World Bank Dubai – Impact Evaluation workshop May 31-June 4
Demand for Good Governance Setting the Context Heightened attention to governance issues at the World Bank since adoption on GAC strategy Increasing emphasis on outcomes/impact to enhance development effectiveness More attention to transparency, access to information, citizen-participation at all levels in World Bank operations (mandatory with Investment Lending reform) Key focus of Social Development department- strengthening demand-side of governance (including PM&E methods)
Demand for Good Governance What is demand for good governance (DFGG)? The ability of citizens, civil society organizations and other non state actors to hold the state accountable and to make it responsive to their needs In turn, DFGG enhances the capacity of the state to become transparent, accountable and participatory in order to respond to these demands DFGG mechanisms can be initiated and supported by the state, citizens or both but very often they are demand-driven and operate from the bottom-up. The people have a right to know, a right to question, a collective Constitutional right to receive an answer.” Aruna Roy, MKSS Rajasthan, India
Demand for Good Governance Key Demand-side mechanisms to address governance challenges Transparency/Access to Information Consultation/Participation Grievance Redress Mechanisms Third party Monitoring/ Independent verification of outcomes Participatory Impact Assessments
Demand for Good Governance Why are participatory approaches important for IE? Ensures that impact indicators are aligned with people’s priorities Increase accountability to citizens Sustainability of projects/programs Expands pool of citizens who can continue monitoring once the project is closed and the M&E specialists have moved on Better risk assessment/mitigation for projects and programs
Demand for Good Governance 6 Possible methods/tools under PM&E: Visual techniques Activity monitoring chart Participatory Rural Appraisal Citizens report cards Community scorecards SARAR (participatory problem solving tool) Participatory Impact Assessments
Demand for Good Governance 7 What’s Participatory Impact Assessment? “Involves the adaptation of participatory tools combined with more conventional statistical approaches specifically to measure the impact of humanitarian assistance and development projects on people’s lives.” Source: Feinstein International Center: Participatory Impact Assessment” Guide for Practitioners
Demand for Good Governance 8 Participatory versus Conventional IE Source: Adapted from Deepa Narayan, World Bank.
Demand for Good Governance At the project level-3 key questions… 1.What changes have there been in the community since the start of the project? 2.Which of these changes are attributable to the project? 3.What difference have these changes made to people’s lives?
Demand for Good Governance Designing and Implementing PIA: Steps Stage 1: Define Questions Stage 2: Define the geographical and time-limits of the project Step 3: Identify locally defined impact indicators Step 4: Decide on ranking/scoring methods on and testing sampling methods Stage 5: Choose Sampling Methods Stage 6: Assess project attribution Stage 7: Triangulation Stage 8: Feedback and verify results with community Source: Feinstein International Center: Participatory Impact Assessment” Guide for Practitioners
Demand for Good Governance Step 2, Tool #1: Defining the project boundary: participatory mapping A map of Zipwa Site, ZimbabweCommunity members drawing a map in the sand
Demand for Good Governance Step 2- Tool # 2: Define the project period by timelines established by the communities Creating a timeline-- - Identify a Knowledgeable person (or persons) in a community -Ask them to describe the history of the community. -- In many rural communities, such descriptions usually refer to key events such as drought, periods of conflict or disease epidemics - The project start and end time should be related to these key events.
Demand for Good Governance STEP 3: Local Identification of Impact Indicators Impact indicators : -look at the end result of project activities on people’s lives. -Measure the fundamental assets, resources and feelings of people affected by the project. -can include household measures of income and expenditure, food consumption, health, security, confidence and hope. Quantitative: (income earned from crop sales, increased milk consumption by children) Qualitative:(improved skills, knowledge or social status, participation, security, dignity, social cohesion, wellbeing)
Demand for Good Governance Step 3: Indicators-identified by communities themselves… Ex: Drought projects in Zimbabwe and Niger Impact indicator by project M&E specialist Impact indicators by beneficiaries increased crop productionThe ability to pay for school fees using project derived income (education benefits) dietary diversityThe ability to make home improvements Improved skills and knowledge from the projects training activities Improved social cohesion Tips for practitioners: Make sure to capture the views of different groups of people within the community. (Women will often have different priorities and expectations of project impact than men.) Tips for practitioners: Make sure to capture the views of different groups of people within the community. (Women will often have different priorities and expectations of project impact than men.)
Demand for Good Governance project participants identify all the food sources that contribute to the household food basket. project participants identify all the food sources that contribute to the household food basket. Practitioner Tips- Where informants are literate you may choose to simply write the name of The indicator on a card. Practitioner Tips- Where informants are literate you may choose to simply write the name of The indicator on a card. Step 4: Methods for Scoring/Evaluation Tool #1: Scoring of Food sources using counters- Evaluating the impact of a community garden
Demand for Good Governance Step 4-Tool # 2: Impact calendars-post-harvest food balance Monthly household utilization of the harvested maize until depletion ( using 25 counters) exercise -done with project participants for the agricultural year before and after the project and again for the agricultural year. The exercise then repeated with community members who had not participated in the project
Demand for Good Governance Challenges with community-level PM&E approaches (from Bolivia and Nepal) Clash of incentives: pressure from donors to ‘prove’ impacts vs. adoption of a bottom-up, participatory approach based on ‘improving’ programs in ways that meet community needs and aspirations Resistance from the project teams to changing their existing M&E practices Project deadlines prevent project-staff from establishing a consistent PM&E practice Availability of PME expertise within the field staff to facilitate the PM&E exercises, Insufficient transfer of community-level PME skills to interest groups and grassroots organizations Lack of sufficient training/capacity-building programs Human resource problems and lack of capacity in analyzing qualitative data and reporting results.
Demand for Good Governance Possible recommendations to improve PM&E activities Develop ready-to-use templates, a detailed Community Researcher manual to improve the research data and reports. Conduct regular follow up visits to each case study sites to review the work of the Community researchers and provide feedback, advice and support Identify mentors, encourage the community researchers to phone their mentors on a regular basis to share their progress and any problems they had. Encourage the community researchers to contact each other regularly to share their experiences and reports. Promote “peer-to- peer learning’
Demand for Good Governance Key messages Keep participatory evaluation methodologies simple and practical Develop your methods, standardize and repeat. the more repetitions, or the larger the sample size, the more statistically reliable the results will be. Remember to field test your methods with community members before the assessment–most methods look easy on paper but require fine tuning once you start to use them in the field.
Demand for Good Governance Participatory Impact Monitoring Booklet I- V GTZ World Bank Participation and Civic Engagement Website: Feinstein International Center-Tufts University Participatory Impact Assessment: A Guide for PractitionersFeinstein International Center-Tufts University Participatory Impact Assessment: A Guide for Practitioners Useful links/Resources World Bank Social Accountability Sourcebook