Pitfalls of Participatory Programs: Evidence from a randomized evaluation in education in India Abhijit Banerjee (MIT) Rukmini Banerji (Pratham) Esther Duflo (MIT) Rachel Glennerster (MIT) Stuti Khemani (The World Bank)

Motivation  Community participation is being pursued in policies and programs to improve the quality of public services  We evaluate the impact of mobilizing communities (using information & advocacy) to participate in publicly provided education services

The Study Location  In Uttar Pradesh (UP)—a “typical” district, Jaunpur

Institutional Context in UP  Village-level education committees (VECs), representing parents and village leaders, created by state education policy in India since the 1990s  VECs have specific powers to monitor quality, and manage public resources to improve quality  Can recruit community-based additional teachers

Findings from Baseline Survey  Baseline Survey (March – June 2005), 280 villages: Learning outcomes  30 households randomly selected in each village  all children between the ages of 7 and 14 tested on basic reading, writing, and math skills Community participation and local governance  All VEC members interviewed  All government school head teachers interviewed  10 households (from among the above 30) interviewed School resources and school functioning  All government schools surveyed through interviews and direct observation

Parents don’t know that a VEC exists Have you heard of the VEC? 1.5% 1.1% 5.0% 7.6% 92.4% Villagers Who Don't Know of a Village Education Committee Villagers Who Think there is a VEC Villagers Who Believe there is a VEC, But Can't Name Any VEC Members Villagers Who Can Name Only One or Two VEC Members (the Pradhan and/or Headmaster) Villagers Who Can Name More VEC Members than Just the Pradhan and Headmaster * Based on 2,803 household surveys in 4 random blocks in the District of Jaunpur, UP. Each household is weighted by total number of households in village divided by number households surveyed in village.

VEC members don’t know their roles Percent of VEC Members Who: Don’t know that they are members of the VEC Don’t know that funds are provided to VECs to improve schools Don’t list hiring of community- based additional teachers as a VEC instrument Head Teachers 4.2% 86% Other VEC Members 22.7%73.4%95%

Parents of children at low levels of learning tend not to know this…

Interventions to Encourage Participation: (1) Mobilizing communities through advocacy, and information about VECs  Small, informal meetings in each hamlet during 2 days in a village  Village-wide meeting on 2 nd or 3 rd day, with participation of key VEC members—Village Head and School Teacher  Distribution of pamphlets to VEC members listing and explaining their roles

Interventions to Encourage Participation: (2) ( 1) + Testing: Reading test for outcomes information, building monitoring capacity  In hamlet meetings, facilitators begin testing children; community invited to test children themselves and prepare hamlet-level “report cards”  In village-wide meeting, hamlet volunteers invited to present testing tools and “report cards”

Interventions to Strengthen Community Participation: (3) (2) + Teaching : Capacity Building to Directly Improve Learning  In village meetings, facilitators present Pratham’s program for improving reading  Offer to train any volunteers to hold reading classes

Experience with Interventions  65 villages each received Interventions 1, 2, and 3 between September and December 2005  Soon after a new village government was elected, and could constitute new VECs  Repeat visits in Jan-Feb 2006 to hand-out and explain pamphlets to VEC members  85 villages served as controls

Experience with Teaching Intervention Local youth volunteered: 405 reading classes held No. of Reading Classes in Teaching Intervention Villages No. of Reading Camps No. of Villages

Participation of 7453 children in reading camps in 55 villages: 135 children per village

Impact Evaluation End-line survey took place over March-June 2006 Specification (for each of K “families” of outcomes):  X: baseline levels of every outcome in the family  Difference-in-Difference estimates  White standard errors clustered by village  Additional controls in some specifications (parent’s caste, education, occupation, literacy; child’s age, gender, school status)

 No impact on VEC activity  No impact on VEC awareness of additional teacher program

 Only small impact on parent knowledge (only 5-10% of parents know about the VEC even at endline)  No impact on parent activism

Non impact on parent activism is confirmed by school teacher responses

No positive impact on school resources

No positive impact on teacher attendance or effort

No positive impact on student enrollment and attendance in government school

 In “teaching” intervention villages, where reading classes were offered, small increase in children moving out of school, possibly because they were moving into reading camps

Improvement over time among illiterate children

Improvement over time among children who could recognize letters

Improvement over time among children who could read words or paragraphs

Summary of Impact  No impact at all on public schools, or on VEC activism  Learning improved because an offer to the community of a teaching program was taken up by local youth volunteers— reading classes were held  Children who enrolled in these classes made significant improvements in reading within a couple of months  No evidence, anecdotal or otherwise, of VECs, or Village Heads, or school teachers, supporting these volunteer-led reading classes, or mainstreaming the reading program in public schools

Why was there no impact on public schools?  Not because of lack of interest in education, or willingness to participate, or because it’s just too difficult to get children to learn Contrast between impact outside versus within the public school system suggests that the real challenge lies in improving public provision

Conclusions  Significant barriers to participation, for social accountability of public providers  Instituting participation from outside, in lagging or disadvantaged areas, likely requires credible support from above— from government or influential NGOs  “Short route” to accountability (WDR 2004) likely requires going a long distance