Presentation on theme: "PNPM-Urban Process Evaluation : Key Findings and Considerations November 2011."— Presentation transcript:
PNPM-Urban Process Evaluation : Key Findings and Considerations November 2011
For discussion only Goals for process evaluation Assess implementation relative to program goals and objectives Identify what is working well and explanations for why Highlight areas needing improvement and explanations for why Although process evaluation is not impact evaluation, if and when possible, show associations between process and outcomes Highlight (subjective) “best practices” and suggest implications for design and/or areas for more investigation or experimentation
For discussion only Program objectives for PNPM-Urban Source: Review of World Bank PNPM-Urban III Project Action Document
For discussion only Evaluation focused on six themes emerging from the results framework To what extent is CDD realized in planning, decisionmaking and implementation? Are socialization activities sufficiently inclusive? Is PNPM-Urban meeting the needs of the poor? Assessing the role of CDD How does PNPM-Urban link with local government and with sectoral programs? Is there much collaboration / duplication? Links with local government How effective are the current facilitators and do they adequately meet the needs of the communities? What characteristics contribute to strong facilitators versus weak? Capacity building and effectiveness of the facilitators Overall how is the quality of the infrastructure investments? What factors contribute to high- and low-quality works? How cost effective are infrastructure investments? Quality of infrastructure How effective are the current governance structure/ control mechanisms that are in place? Are they adequate or could they be improved? If so, how? Control mechanisms/governance Do communities express a desire for larger grants?. How do experiences and outcomes vary under PAPG/ND and what are the differential impacts? Adequacy of block grants and insights from PAPG/ND
For discussion only Methodology Desk and Literature Review Subject Matter Expert Interviews Formative Research Site visits Review of PNPM-Urban MIS Data Analysis draws on multiple sources of information
For discussion only Focus of Data Gathering: Site Visits Goal was to design and implement efficient data collection effort under time and resource constraints, while capturing sufficient number and variation in sites, including 3 ND subjective perceptions of multiple, different stakeholders objective assessments of PNPM-Urban project output quality Final fieldwork design reflects important trade-offs 16 field sites to be covered in 5-6 week timeline priority focus on infrastructure simultaneous dispatch of four field teams led by RAND and SurveyMeter, with four engineering teams after heavy pre- field preparation + common initial training and 2 site visits Fieldwork commenced May 28 through June 25, 2011.
For discussion only Site Selection Randomly selected from IFLS communities (80% of national population) with projects in the MIS database > 2 years old Regional stratification by urban population/PNPM-Urban funds, wealth stratification by IFLS community wealth index
For discussion only 360-degree approach to data collection District Program manager Korkot / Askot PNPM-Urban Management Bappeda and district MPW leadership Kota Officials Government Facilitators Consultants/planners Local PNPM-Urban Camat, Lurah, RT/RW leaders NGO representatives Local Government (and other civil society) BKM leader Infrastructure KSM members PNPM-Urban community leaders Poor households in general Infrastructure project beneficiaries Community members
For discussion only 360-degree approach to data collection Key informant interviews PNPM-Urban management and government Local PNPM-Urban, government and other civil society Key informant interviews Rapid survey about BKM and KSM PNPM-Urban community leaders Rapid survey of poor households Male and female focus groups + interviews General poverty PNPM-Urban Infrastructure beneficiary interviews Community members Infrastructure assessments MIS data assessment Other Key informant interviews
For discussion only Key Findings 1: Community socialization and participation What is working well? Knowledge of processes among local leaders and individuals within PNPM is good Community institutions like the BKM largely function as intended High level of recognition of PNPM-Urban projects among community (although not always PNPM-Urban) High level of expressed satisfaction with project outputs among community and beneficiaries High levels of community contribution to implementation, although role of paid and unpaid labor appears to vary
For discussion only Key Findings 1: Community socialization and participation What are areas for improvement or consideration ? Little process awareness among community members: rarely critical but tend to approve /ratify rather than propose project. Decision-making tends to proceed by delegation. Stated preferences = demand for social and economic projects: respondents identified loans/training as most important needs Targeting the poor via infrastructure projects challenging, because of geographical spread Voluntary administrative burden is taxing for BKM and KSM, may affect selection of leaders Women’s participation in decisionmaking is relatively low, potentially due to exclusionary norms and protocols
For discussion only Key Findings 2: Links with Local Government What is working well? Involvement by local government varies across sites, from little substantive involvement (e.g., only signing off on projects) to substantial local government support (e.g., participating in decision making and oversight) Even in low-interaction sites, Musrenbang process is used effectively to convey information about efforts: there appears to be little or no duplication of activities or priorities. What are areas for improvement or consideration? Few respondents mentioned training for local governments Observe coordination, but no widespread collaboration or joint planning; some evidence of competitive feeling Support for more integration is not universal.
For discussion only Key Findings 3: Facilitators What is working well? Consensus view across respondents is that facilitators are playing a significant part in all stages of PNPM- Urban, and that facilitators’ contributions improve the quality of projects Rotation of facilitators limits cronyism and increases the exposure of skilled individuals What are areas for improvement or consideration? Facilitators universally appear overworked and overextended. The existing administrative burden on the facilitators for compliance purposes is substantial Most cannot contribute much to encouraging participation at the sub-Kelurahan level. Timing of rotations is disruptive to the project cycle
For discussion only Key Findings 4: Infrastructure Assessment What is working well? Assessments of PNPM-Urban project quality were high Subjective assessments compare PNPM-Urban project quality favorably to government projects; respondents attribute higher quality to participatory approach and avoidance of government management (including some government officials) Community reports being more likely to maintain PNPM- Urban projects (although too early to objectively determine) Average levels of quality inspection reported in MIS tally with average independent site assessments, suggesting standards are adequate by local professional standards
For discussion only Key Findings 4: Infrastructure Assessment What are areas for improvement or consideration? Most beneficiaries were not consulted or given opportunities to provide feedback or correction Although generally quality is perceived to be high, some respondents suggest voluntary contributions may lead to the use of lower quality materials Use of paid versus volunteer labor varies across sites, with some explanations e.g responding to the technical demands of the project, but reflects potential trade-off Actual projects can depart from plan and MIS reports (also related to control) without the appearance of reliable systematic protocols for reporting(e.g. cases of splitting projects, or bundling types of expenses together)
For discussion only Key Findings 5: Control Mechanisms and Governance – Funds Management What is working well? Very few reports of complaints related to misuse of funds Auditing is generally completed on schedule and uncovers instances of inadequate project performance, selection and implementation at an early stage by facilitators. Most sites reported relying on the combination of multiple layers of audit, social norms and community awareness of projects to ensure accountability What are areas for improvement or consideration? Difficult to assess extent of reporting bias Previously noted: heavy burden of compliance, issues of control (versus governance)
For discussion only Key Findings 5: Control Mechanisms and Governance – MIS What is working well? Relative to its initial goals, MIS performs well in making public a vast quantity of timely and usable data for tracking purposes Ability to analyze MIS has increased significantly thanks to motivated and highly-familiarized PNPM-Urban technical staff What are areas for improvement or consideration? Significant inherent challenges including harmonization of file formats across funders and cycles, training and motivation of data entry personnel at the community level Substantial room to realize full potential of MIS data for process evaluation: data quality and validation issues remain, many key outcomes e.g. poverty, population are only self-reported, users need significant guidance
For discussion only Key Findings 6: Block Grant Size and ND Model What is working well? Relatively few sites reported size was inadequate for needs (however, this may be biased by local expectations) Large % of ND site funding is spent on planning and marketing, suggesting recognition of problems of participation What are areas for improvement or consideration? Larger grants may be used to undertake more small projects in different areas, not realizing economies of scale/scope. (One ND site even desired more flexibility to do so) Concerns expressed that larger projects are harder to manage and difficult to distribute benefits equitably. ND site planning and marketing efforts not viewed as useful (although sample is small)
For discussion only Limitations of this study (planned and unplanned) Reiterate that focus is largely on infrastructure Short timeline precluded more intensive focus group discussions and community member interviews on specific issues that arose during evaluation e.g. expectations of community regarding delegated versus direct participation Resource and time constraints prevented larger sample of sites, including PAPG sites, potentially more meaningful sub-group analysis Cost-effectiveness analysis and analysis of sectoral program links could not be completed within the scope of the project MIS data provided useful insights into program at a high level but precluded robust regression and subgroup analysis; most analysis draws on subjective perceptions, should be interpreted as such.
For discussion only Implications for Design and Experimentation Implement universal invitations to meetings particularly for women. Consider alternative methods of eliciting preferences of poor households Assessing the role of CDD Continue to focus on training and capacity building, emphasizing collaboration rather than competition and potentially consider more structured interactions Links with local government Build more facilitator capacity through recruitment and/or training that includes interpersonal skills building, reconsider rotation schedules Capacity building and effectiveness of the facilitators Allow more direct channels of feedback for those likely to be affected. Improve balance between local responsiveness and control. Consider tradeoffs between community and hired labor. Quality of infrastructure Maintain multiple checks and balances with audit system incorporating local participation; increase attention to potential of MIS from bottom data entry to high-level database structure and user interafce Control mechanisms/governance Consider individual needs of communities and continue to monitor development of ND projects as they mature Adequacy of block grants and insights from PAPG/ND