2. Evaluating Social Fund Performance Across Countries International Conference on Social Funds June 6, 2000 Laura Rawlings, Lynne Sherburne-Benz, Julie Van Domelen World Bank

3. Today’s Presentation Preliminary Results of a multi-country study to assess household level impacts of Social Funds n Overview of Study n Poverty Targeting n Household and Facility Impacts of Education, Health, Water and Sanitation sub-projects n Short Break at 11:00 am n Cross-cutting Issues - Participation, Sustainability n Cost-efficiency n Discussion

4. Reasons for the Evaluation n Importance of Social Funds –Social Funds (SFs) have lasted longer than originally thought; many now in 3rd or 4th phases –Increasingly widespread with significant amount of resources; SFs currently operating in 50 countries. Bank support at around $3 billion, twice that when include resources from other agencies, governments –SFs show flexibility and dynamism in their evolution, now moving into supporting capacity building of local governments as well as local communities, services as well as infrastructure, centers of innovation n Little knowledge of direct SF impact on households or on facilities relative to comparators

5. Impact Evaluation Issues Addressed –Poverty targeting –Impacts –Sustainability –Cost efficiency Country Case Studies –Armenia –Bolivia –Honduras –Nicaragua –Peru –Zambia

6. Approach n Build on on-going work n Collaborate with SFs and Bank teams where interest high and/or evaluations planned n Find countries with household surveys already planned (and financed) n Experiment with analytical approaches –Different data availability –Different areas of emphasis

7. Data and Surveys Used Strive to combine 4 types of primary data 1. Household survey Household surveys with modifications or added module Oversample SF beneficiaries and creation of control groups 2. Facilities survey Same framework, adapted to each country (except Peru) Applied to social fund (treatment) and non-social fund (control) facilities Separate survey for each type of intervention 3. Qualitative assessments 4. Cost efficiency study

8. Joint Collaboration Broad Participation -World Bank and SFs n Bank : 2 Networks ( PREM and HD ), 2 Thematic Groups ( Impact Evaluation and Social Funds ), 3 Regions ( LAC, AFR, ECA ) n Countries : Social Funds, National Statistical Offices, Consulting firms, Universities, NGOs n Partners : Norwegian Trust Fund provided resources for overall study as well as support to each individual country case, many other bilateral and multilateral partners in various country case studies n Papers for each case study list individuals involved

9. Poverty Targeting

10. Questions to be Answered n Are SFs successful at geographic targeting? n Given difficulties in reaching poorest communities and difficulties of poor communities to organize and access resources, are social funds reaching the poorest communities and households? n Are poor households choosing to use SF facilities? n Does the demand-driven nature of SFs affect targeting outcomes, either by type of sub-project or according to SF institutional targeting efforts? n Does targeting improve over time?

11. Mechanisms in Targeting SFs n Geographical targeting - poverty maps –regional allocations –weighted allocations according to poverty-related indicators (poverty, basic needs indices) n Self-targeting through menu choice –type of projects; open vs. more restricted menu n Regional offices for outreach n Ex-ante social assessment n Regional offices and/or proactive promotion efforts for outreach

12. Methodology n Geographic –Compare SF resource allocation by district to district poverty levels n Household ( first time able to look at this systematically across countries ) –If large SF coverage/incidence - utilize existing household survey or modify survey to be able to identify SF beneficiary households (Peru) –If smaller SF coverage/incidence - oversample SF beneficiaries, modify national household survey as needed, then compare SF households to national households (ideally same timeframe, survey, surveyors)

13. Initial Geographical Targeting Results Social Funds generally reach poor areas/districts n In Nicaragua between 1991-98, 76% of SF resources went to municipalities with either extreme or high poverty levels (accounting for 53% of the population). n In Peru over 80% of resources went to poorest 40% of municipalities n In Honduras, between 1994-97 (2nd SF), 51% of SF resources went to the 2 poorest categories of municipalities (labeled very poor or poor and accounting for 41% of the population) n In Zambia, geographic allocation of resources among districts has been fairly neutral. Targeting has improved over time in all cases

14. Initial Geographical Targeting Results Social Funds are successfully reaching the poorest districts and municipalities –In Nicaragua the extremely poor municipalities (that have 17% of the country’s population) received 23% of SF resources between 1991-98. This has increased from 11% in 1991 to 34% of resources in 1998. –In Honduras 27% of resources went to the poorest category of municipalities (where 20% of the population lives). –In Peru about 50% of SF resources go to poorest 20%, and about 30% to poorest 10%. Not surprisingly poorer districts received more SF resources in per capita terms than better-off districts.

15. Peru Targeting 1998 (Schady 1999)

16. Initial Household Targeting Results But do SFs reach poor households within districts? YES n In Honduras, 53% of SF resources went to beneficiaries in the poorest 40% of the population n In Nicaragua, don’t have the aggregation across projects, but find a slight pro-poor distribution of resources with 41% of education investments, 58% of health investments, and 74% of latrine investments going to the poorest 40%. These project types account for almost 3/4 of SF investments and over 90% of projects. n In Zambia SF beneficiaries are poorer than non-SF households. –Rural/urban differences found in Zambia. At an aggregate level SF resources reach poorer households in rural areas but better-off households in urban areas.

17. Initial Household Targeting Results SF expenditures also often reach the poorest n In Honduras, 36% of SF resources go to poorest 20% of households and 24% to the poorest 10%. Related, 32% of SF beneficiaries were in the poorest quintile and 19% in poorest decile. n In Nicaragua where extreme poverty line is 17%, the poorest 20% of households receive 23% of education, 23% of health, and 33% of latrine investments. However, latrines are the only investments reaching the poorest 10% in a progressive manner. n In Zambia beneficiaries of rural education investments are 30% poorer than non-SF beneficiaries

18. Targeting by Type of Project Education Projects Typically Pro-poor n In Nicaragua, 18% of SF investments went to the extremely poor and 54% to poor households, showing pro-poor bias with the extreme poor receiving slightly more than their population share. n In Peru geographic targeting of education projects is quite progressive (poorest 20% receiving over 40% of resources; and poorest 10% receiving over 20% of resources). Is also noticeably more pro-poor than other large public sector education programs. n In Zambia, education investments pro-poor overall. In rural areas SF beneficiary households are 30% poorer than in non- SF communities. In urban areas, however, regressive. n In Honduras 36% of education investments went to beneficiaries in lowest income decile and 46% of resources to the poorest 20%

19. Targeting by Type of Project Health interventions typically pro-poor n In Nicaragua, as in Honduras, health interventions more pro-poor than education interventions. In Nicaragua, 23% of health investments are received by poorest 20% and 58% to poorest 40%. In Honduras, 74% of health investments go to poorest 40% and 53% to poorest 20%. n In Zambia, health interventions are overall neutral, but pro-poor in urban areas (where SF beneficiaries have ‘income’ 18% lower than non-SF folks), but quite regressive in rural areas (where health SF beneficiaries ‘incomes’ 23% more). n In Peru, geographic targeting of health interventions are quite pro-poor.

20. Targeting by Type of Project Latrine interventions are very pro-poor n In Nicaragua most pro-poor distribution of projects with 64% of resources going to poorest 40% of the population (and 14% of resources to poorest 10% of population). n In Honduras also most progressive with 66% of resources going to poorest 40% (and 22% of resources to poorest 10%) Water targeting neutral or pro-poor n In Nicaragua distributed fairly evenly across population with neither pro-rich nor pro-poor bias, although gone to comparatively poorer households than in 1993. n In Honduras, 57% of resources to poorest 40% of beneficiaries and 17% to poorest 10%.

21. Targeting by Type of Project Sewerage not targeted to poor n In Nicaragua sewerage has pro-rich distribution with richest 30% of population receiving 60% of benefits. n In Honduras is also pro-rich with only 23% of resources going to beneficiaries (households with connections) in lowest 4 deciles and 42.5% of resources going to households in the 8th decile.

22. SF Targeting - Conclusion Social Funds Tend to Reach the Poor n Geographic targeting usually good n Household targeting to the poor is even better n In many cases, SF are reaching the extreme poor and even the poorest decile, although not for all project types n Targeting has consistently improved over time n Type of project seems to make a difference to targeting outcome in majority of cases

23. Impact Evaluation of Social Funds: Initial Results Outline of Presentation Objective Methodology Initial Results - Education Initial Results - Health Initial Results - Water and Sanitation Conclusion

24. Impact Evaluation Objective n What is the condition of social fund beneficiaries compared to what would have been observed in the absence of the social fund intervention? –Change in quality of facilities and services –Changes in coverage/utilization –Impact of interventions on recipients’ education and health status

25. Impact Evaluation Methodology Impact evaluations assess causality through the construction of counterfactuals (what the status of the beneficiaries would have been without the program) Counterfactuals are usually constructed through the use of control/comparison groups In the country cases control/comparison groups created using different techniques depending on timing and the availability of survey instruments…. General evaluation design is a matched comparison between SF and non-SF beneficiaries Use of a variety of analytical techniques (propensity score matching, instrumental variables, difference in differences) General limitation - lack of baseline (pre-SF) data Creative approach to constructing comparison groups Living Standard Measurement Survey (Armenia, Peru, Nicaragua, Zambia) ‘Pipeline’ projects (Honduras and Zambia)

26. Impact Evaluation Methodology n Focus on sub-projects that received the majority of social fund financing: education, health, water and sanitation –Focus on facilitity (case studies of schools, health posts, etc.) and household level results (representative samples of SF and non-SF) using a hierachy of indicators (intervention, utilization, health and education outcomes) n Mixed- Method approach: household surveys, facilities surveys and qualitative studies to provide robustness –Household Surveys (all cases) - measure outcomes in health and education resulting from social fund investments –Facilities Surveys (all but Peru) - evaluate the quality and sustainability of social fund infrastructure, utilization and community participation –Qualitative Studies (Honduras, Nicaragua, Zambia) - evaluate whether social funds meet community investment priorities, levels of community satisfaction, community participation in project development and maintenance

27. Initial Results: Education Contributions to School Infrastructure In all country cases, social funds are major national contributors to expanding access to school infrastructure : Nicaragua: SF provided 49% of public education investments (‘91- ’99) Honduras: SF provided an estimated 58% of new schools and 61% of all new classrooms (‘94-’97) Peru: SF financed over 10,000 education projects worth US$12.2 million (‘92-’98) Zambia: SF school rehabilitation reached 16% of the existing stock of schools (‘91-’98) education projects generally account for the largest share of SF subproject investments

28. Initial Results: Education School Quality n Social fund investments have a positive impact on the quality of school infrastructure –In Nicaragua and Zambia, SF schools have better access to piped water and electricity. –In Bolivia and Nicaragua, SF schools have better access to sanitation services n Access to desks improved in SF schools n Students in SF schools had equal or better access to textbooks than students in non-SF schools

29. Initial Results: Education School Quality (Cont.) n SF schools have staffing patterns similar or better than non-SF schools –In Nicaragua, numbers of teachers increased by 20% in SF schools (matching a similar increase in enrollment), while the numbers of teachers remained constant in non-SF schools. –In Zambia, SF schools have on average more teachers (30 vs. 16 in non-SF schools) and similar student/teacher ratios as non- SF schools –In Bolivia, evidence suggests that the student to teacher ratio fell as a result of SF investments, while slightly increasing in non-SF schools –In Honduras, student-teacher ratios in SF schools in line with national norms (about 34 students per teacher)

30. Initial Results: Education Utilization (School Data) n Schools receiving SF investments have experienced a rise in enrollment –In Nicaragua, SF schools experienced a 20% increase in enrollment, while non-SF schools experienced a slight decline –In Zambia, SF schools experienced a 17% increase in enrollment, compared to a 2% decline in non-SF schools

31. n In half the cases, social fund investments resulted in higher enrollment/attendance rates: –In Nicaragua, enrollment rates were 5-10% higher in the SF communities, depending on the comparison group used Also, female enrollment rates were 4% higher than male enrollment rates in SF schools, a difference not noted in the comparison group schools –In Peru, increased attendance was more likely in districts that received more SIF resources than in districts that received less SIF resources –In urban areas in Zambia, attendance is higher in SF than in non-SF communities (86% vs. 82%) ….. but not in Honduras, Bolivia or other parts of Zambia –In Honduras, Bolivia there was no evidence of higher enrollment/attendance rates in SF communities than in non-SF communities Initial Results: Education Utilization (Household Data)

32. Initial Results: Education Other Household Impacts n Besides enrollment/attendance, education investments have clear, positive impacts on a variety of measures: –Age for grade improved in Nicaragua, Honduras and rural areas in Zambia as a result of SF investments (by 15% in Honduras, 1-5% in Nicaragua, and 4% in Zambia) –Drop-out rates show a possible improvement in Bolivia …..but no impact on achievement as measured by math and language tests in Bolivia, the only case where this was measured

33. Initial Results: Health Quality of Health Facilities n SF health facilities’ physical infrastructure is either similar to or better than non-SF infrastructure –In Bolivia, SF health facilities had better access to sanitation services, and averaged.7 more patient rooms than non-SF –In Zambia, SF health posts have more medical rooms (10 vs. 7) and bathroom facilities, better access to electricity (81% vs. 50%), and better access to water (protected well or better - 88% vs. 70%) than non-SF ….however, in some countries infrastructure problems persist –In Nicaragua, a similar percentage of SF and non-SF health posts lack access to electricity (~35%) and piped water (about 52%). –In Honduras, a similar number of SF and non-SF posts lacked piped water (~ 37%) and sanitation services (40%).

34. Initial Results: Health Facilities Quality of Health Facilities (Cont.) n Social fund health facilities are similar to or better than non- SF with respect to inputs such as staffing, availability of medicines and other non-SF provided inputs –Better SF access to furniture and basic medical equipment in Bolivia, Nicaragua and Zambia –Better SF access to doctors and nurses In Nicaragua, numbers of doctors and nurses increased to 1.4 per SF health post, while remaining at.5 for non-SF In Zambia, doctors in SF posts increased 30% while declining 33% in non-SF posts. Similar trend with public health nurses (35% increase vs. 89% fall in non-SF) …. but medicines and supplies are usually lacking across all facilities (SF and non-SF alike) in Honduras less than half of the medicines that should be regularly available are not, in SF and non-SF rural health posts alike in Nicaragua, 25% (in SF health posts) and 31% (in non-SF) of the medicines that should be regularly available according to Ministry of Health norms are never available

35. Initial Results: Health Utilization (Health Facilities Data) Social funds have had a significant impact on increasing the utilization of health facilities receiving the social fund interventions In Nicaragua, health post utilization increased universally for SF and non-SF However, utilization increased more for SF posts overall utilization rates were higher for SF posts (17 visits/day vs. 14 visits/day for non-SF posts) In Honduras, SF health posts have higher daily utilization rates than health posts overall ( urban - 54 consultations/day vs. 34 overall, rural - 9.4 consultations vs. 9.2 overall) In Zambia, child attendance, deliveries and total hours open increased in SF health posts, while declining in non-SF health posts

36. Initial Results: Health Utilization (Household Data) n At the household level, utilization of health services has increased as a result of social fund investments In Honduras, 51% of sick people had sought medical attention where the SF had invested, compared with 41% for the populations in similar communities without SF support. SF investments in Bolivia result in significant increases in pre- natal control and numbers of attended births SF investments in Nicaragua had a positive impact on health post utilization for children<6 with diarrhea In Zambia, households in communities with SF supported health posts were more likely to seek medical treatment at their health post, and less likely to go to the hospital, than households in communities without SF health posts. Also, in communities with SF health posts, households have a higher # of vaccinations per child (for diptheria and polio)

37. Initial Results: Health Health Outcomes n Only in Bolivia is the increase in utilization clearly associated with an impact on health outcomes –< 5 child mortality significantly reduced as a result of SF health facility investments –Results are robust to different impact estimates (propensity score matching, difference in differences, and hazard function) and controlling for other interventions (water investments) n In Nicaragua, Honduras and Zambia there is no clear measured impact of SF health investments on health outcomes

38. Initial Results: Water Water Project Quality n Water supply is variable across social fund systems, despite good connection rates In Nicaragua, mixed record –water was rationed in half of the SF water systems (supplying on average 3 days/week) –water availability 25 days out of the month vs. 18 for non-SF In Honduras, supply problems –half of SF water systems report low production, compared to 21% of non-SF systems In Bolivia, increased availability –SF investment resulted in a 4-10% increase in frequency of water supply n Water quality improved in Bolivia In Bolivia, the one case where water quality was tested, 55% of the water sources had significant levels of fecal contamination before the social fund investments, compared to only 13% of the new sources after the SF investment... but improvements occurred only after training was provided

39. Initial Results: Water Utilization (Household Data) n Social fund water investments have resulted in a net increase in access to water in SF vs. non-SF areas –General increase in coverage Honduras: 5% rise in coverage Bolivia: 10% (Chaco) and 18% (Resto Rural) rise in coverage Nicaragua: 25% rise in coverage –General decrease in time/distance to water source Honduras: time collecting water reduced by 42 minutes/month Bolivia: distance to source reduced by 154 meter (Chaco) and 55 meters (Resto Rural) Nicaragua: distance to the nearest water source reduced by 600 meters

40. Initial Results: Water Household Health Impact n Good impact of social fund water investments on health outcomes in most cases Bolivia: impact on duration of diarrhea and impact on under 5 child mortality of water and sanitation investments Nicaragua: impact on stunting (height for age) in children under 6 from 25% to 14% Honduras: no observable impact of SF investments on the incidence of diarrhea

41. Initial Results: Sanitation (Sewerage and Latrines) Quality of Investments n Sewerage systems in Nicaragua and Honduras generally operate without serious difficulties However, both experience problems in periods of rain n Connection rates are low for SF sewerage systems Nicaragua, connection rates are 61% overall Honduras, connection rates are 44% overall –but we lack comparator projects... high connection costs are cited as the main obstacle to sewerage connection In Honduras, private costs of purchasing/ installing a WC averages US$150.00

42. Initial Results: Sanitation Utilization (Household Data) n Social Fund sanitation investments have resulted in a net increase in access to sanitation facilities in SF vs. non-SF areas –Increase in access to latrines Nicaragua: 19% increase in households with latrines Honduras: 65% increase in households with latrines –Increase in access to sewerage Nicaragua: 32% increase in households with sewerage connections Honduras: 34% increase in households with sewerage connections Bolivia: water investments in Resto Rural area resutlted in a 44% increase in households with sewerage (3% increase in Chaco region)

43. Initial Results: Sanitation Health Outcomes n Social fund sanitation projects have positive impacts on health outcomes, but results vary by country and type of investment –Sewerage investments Nicaragua: SF sewerage investments have significant impact on the incidence of diarrhea for children <6 (3% in SF communities vs. 21% in non-SF communities) Honduras: no impact of SF sewerage investments on health outcomes –Latrine investments Nicaragua: no impact of SF investments on health outcomes Honduras: SF latrine investments have significantly reduced the incidence of diarrhea from 10.1% to 6.6%

44. Summary Education Social fund investments in education have ….. n a clear, positive impact on infrastructure n usually accompanied by an increase in the availability of non- infrastructure inputs (teachers, textbooks) n resulting in positive, significant changes in enrollment/attendance (Nicaragua, Peru, urban Zambia) and/or other outcome indicators (age for grade - Nicaragua, Honduras, rural Zambia; dropout - Bolivia) Health Social fund investments in health have ….. n a clear, positive impact on infrastructure n usually accompanied by an increase in the availability of key non- infrastructure inputs (medical equipment, staff) n although levels of key inputs remain low in SF and non-SF facilities n the generally positive, significant impact on utilization is not accompanied by clear, significant changes in health outcomes, except in Bolivia

45. Summary Water and Sanitation Social fund investments in water have ….. n increased access to water, raising coverage and reducing time and distance to the nearest water source n resulting in positive, significant changes in health outcomes in two of the three cases studied (Bolivia and Nicaragua, not Honduras) Social fund investments in sewerage have… n increased access to sanitation, although connection rates remain low n resulting in positive impacts on health outcomes in one of the two cases studied (Nicaragua, not Honduras) Social fund investments in latrines have… n increased access to sanitation n resutling in positive impacts on health outcomes in one of the two cases studied (Honduras, not Nicaragua)

46. Issues for Discussion n What is the best way to address the specific operational issues raised by this study? n How to organize social fund projects to achieve desired outcomes? –Especially in health, where social fund investments have often led to changes in utilization, but rarely health outcomes n How to increase evaluation capacity and knowledge beyond the first steps represented in this study? –Building baseline data into social fund evaluations –Sharing knowledge, innovations and successful ventures

47. Sustainability: The Role of Participation and Social Capital A Look at Some Cross-Cutting Issues

48. What Furthers Project Sustainability? ¶ Investments truly reflect community needs and priorities Ë Community participation in the identification, execution and maintenance of the project Ì Level of social capital and ‘community capacity for collective action’

49. Community Preferences and Priorities n Previous beneficiary assessments found that social funds investments overwhelmingly reflected felt needs of poor communities n New evidence explored the degree to which the project chosen was the one most wanted by the community

50. Findings on Preferences and Priorities n Nicaragua –SF projects consistently deemed to be the most beneficial one received by the community - 71% of interviewees believed that the SF project was the most beneficial; only 7 percent benefits of project chosen were less than other alternatives –Honduras: In communities that implemented water, health and education projects, these project types received far more ‘votes’ than any other type of project. But, sewerage and latrines often did not correspond to community priorities - in sewerage projects, only 4% of respondents ranked it the highest need in the community

51. Participation: Identification Phase n Participation was usually higher in identification than during subsequent design and execution phases - natural to the process n Community participation in identification varies by sectors and countries. –Nicaragua: In terms of school infrastructure, 66% of SF community members were consulted at the design stage compared with 90 % of non-SF community members –In Honduras: Households were consulted more often on SF projects (73%) than other agency projects (50%), with consultation highest on latrines and sewerage (86% and 85%) and lowest on water projects (66%)

52. Participation: Execution Phase n Levels vary between social funds, but usually higher than comparators : –In Zambia, participation stronger in rural areas - 64% of SF households participated during execution versus only 57% in control households –In Nicaragua, overall only 31 percent of respondents participated in construction, but this varies by sector: Community participation was either low or non- existent in water and sewerage projects, but was higher in SF than non-SF health posts. –In Honduras, 58% of households participated in some way in SF projects versus 30% in other agencies’ projects (mainly donations of labor, money, materials)

53. Participation: Operations and Maintenance Phase n Social fund projects are generally better maintained than control group facilities n Social fund-financed infrastructure is more active in community fundraising and support –In Zambia, SF schools report higher payment of PTA fees than non-SF schools; SF households spend more on education than non-SF; SF health centers more likely to have a maintenance committee (50% vs.14%) –In Honduras, 92% of schools that received SF support had a committee or board in charge of maintenance –In Nicaragua, SF schools more active in fundraising. SF rural health posts had over twice the level of community volunteers (19) than non-SF posts (8).

54. Operations and Maintenance (cont.) n Even though more fees are collected in SF projects, the absolute level is often insufficient to cover costs, particularly in water and sanitation. –83% of communities in Honduras and 50% of communities in Nicaragua with SF water systems report that fees collected are not sufficient to pay operations and maintenance costs

55. Does Participation Matter? n Participation increases facility utilization –Bolivia: participation significantly increases household’s use of health care services, including women’s pre-natal care. –Honduras: for all types of projects, beneficiary consultation increased the usage rate (latrines from 55% to 81%; water systems from 78% to 99%; health centers from 83% to 96%) n Community participation raises the probability of project success

56. Findings on Peru n Analyzed data from two rounds of ex-post evaluations of completed SF projects (FONCODES) with sample sizes of 350 and 220 projects - surveyed beneficiaries and project committee members n Created index of variables that measured some aspects of ‘success’ –Project functions adequately –Project addresses principal necessities of the community –Project benefits the neediest portion of the community –Project benefits the entire community –There were no problems during project implementation –There were no delays in construction

57. Findings on Peru (cont.) n Explored which factors were responsible for higher levels of project success (participation, education and poverty levels, training provided, supervision efforts, and some community organizational characteristics) n Participation variables were (1) participation in the local project committee, (2) participation in project construction, and (3) election of project committee by a majority of the population n Results: Community participation raises the probability of project success - findings robust n Participation in water and sanitation projects is more beneficial than participation in other types of projects

58. Peru Findings (cont.) n Does it matter who participates? Participation by community members with higher education levels had a larger impact on project success n Participation has a larger effect on increasing project success levels in less poor communities n However, other factors may be able to address this weakness in the poorer communities: –Both training and indicators of SF management (perception of beneficial supervision, frequency of inspection visits) had significant and positive effects of the probabilities of project success.

59. Social Capital n Social capital: the ability of individuals to secure benefits as a result of membership in social networks or other social structures n Different types: “Bonding” social capital: the strong ties connecting family members, neighbors, and close friends; “bridging social capital: the ties connecting individuals to work colleagues, fellow members of religious or civic organizations, and business associates; and “linking” social capital: vertical ties between the poor and people in positions of influence in formal organizations (such as the state)

60. Role of Social Capital n In the Peru study, the level of ‘bonding’ and bridging’ social capital did not appear to effect project success : –A greater number of informal community groups (mothers’ association, youth groups, etc.) had no significant effect on project success n ‘ Linking’ social capital appeared to have a strong effect of the probability of project success: –A greater number of local institutions (police department, judge, mayor, church, etc.) had a strong and positive effect on project success, however this effect displayed diminishing returns above certain levels n Since there were no control groups, cannot say whether SF communities had more community groups and/or institutions than non-SF communities

61. Do Social Funds Create Social Capital? n Extremely difficult to measure accurately n Mostly ‘subjective’ evidence –In Zambia, households asked whether there was an increased a sense of ‘community togetherness’. In rural areas, SF communities responded more positively than non- SF communities. However in urban areas, the results were the opposite. n Further research is needed

62. Cost-Efficiency of Investments How Do Social Fund Costs Compare to Other Programs?

63. Background n Countries use many delivery mechanisms to finance small-scale infrastructure (social funds, line ministries, NGOs, local governments) n Programs use different technical designs, procurement and contracting systems, quality of materials, levels of community participation

64. Areas of Analysis n At sub-project level: –cost efficiency of social fund investments compared with other delivery mechanisms in terms of unit costs (per m2 construction, per classroom, per litre of water delivered, etc.), cost per beneficiary n At program level: –administrative efficiency - expenditures on overhead (personnel, equipment, other operating costs, etc.) versus investments

65. General Approach of the Analysis n Engineers hired to review unit costs and technical parameters of a sample of social fund projects and ‘similar’ investments made by other agencies n Some analysis of process efficiency and administrative costs n Data sources: Field visits, social fund MIS systems, program files

66. Issues Faced in the Analysis n Finding valid comparisons n Lack of systematic information or restricted access to information n Hidden or implicit costs n Interpretation of findings - is a lower unit cost ‘better’? Is a faster process ‘better’? Tension between cost and quality

67. Preliminary Findings: Armenia n The social fund tended to have lower unit costs of construction than other central government agencies, especially in schools n Compared with NGOs, the social fund tended to construct schools at a lower unit cost, had lower costs per beneficiary in irrigation and slightly higher costs in water

68. Preliminary Results: Nicaragua n Unit costs of construction by social fund found to be higher than comparators in all sectors (education, health, water) except sewerage. n Some of the cost difference can be explained by FISE building more complex and ’complete’ works which complicates comparability

69. Preliminary Results: Honduras n Social fund unit costs on average were higher for school classrooms than those built by local governments but lower for school sanitary facilities n Social fund unit costs for water, sewerage and latrines were higher than those of local government and national water agencies (however, SF builds larger, more complex water systems)

70. Preliminary Findings: Peru n Social fund unit costs lower in school construction than the other central school construction agency, but influence of urban focus of latter n SF cost per capita of health posts is about 1/4 that of other central government program n SF cost per beneficiary in water projects slightly lower than NGO comparator n Irrigation: inconclusive comparison with other national program (depends on year and region)

71. (Very) Preliminary Findings: Zambia n Assessed community-based approaches (including SF) versus contractor-based approaches in 9 school infrastructure programs n Unit costs quite similar among the various community-based programs n Contractor approach more than twice as expensive –but delivers infrastructure faster –can be appropriate where communities cannot make necessary contribution and/or where rapid, large- scale impact required

72. Preliminary Findings on Administrative Efficiency n Scale of programs: Social funds operate significantly larger programs than comparators n Among social funds, ‘overhead’ costs ranged from 6% -13% (Peru 6%, Honduras 13%, Armenia 10%, Nicaragua 12%), but no standard accounting n Difficult to get information on other programs, (especially NGOs) though they tend to have higher administrative costs (10-46%)

73. Insights into Findings n In cases of social funds with higher unit costs (especially Nicaragua and Honduras), influence of cost escalation during construction was important n Some indication that either where community counterparts were higher or community directly managed contracts, unit costs were lower (Peru, Armenia, Zambia) but some difficulty accurately accounting for local contributions

74. Insights into Findings n Large variations in unit costs and cost per beneficiary between programs and even between projects financed by the same program n High variability in technical quality, but few major concerns over outright poor quality in any programs n In general, social funds tended to build more ‘complete’ infrastructure, including utilities, external works, sanitation facilities

75. Recommendation: n Need for improved transparency on costs of programs and standardized accounting methodologies n At the national level, better sharing of technical and cost information between programs n General indication that closer community involvement and greater community responsibility reduces cost

76. Next Steps for “Evaluating Social Fund Performance Across Countries ” n Incorporate household survey data from Armenia - currently being processed n In addition to the various country level reports (some currently available), the cross-country analysis will be published by the end of the year –Will send out notification to conference participants n Dissemination to debunk ‘myths’ - both positive and negative n Toolkits and training will be developed on how to implement impact evaluations of social funds