Presentation on theme: "The SPADA/MCLE impact evaluation: work in progress Markus Goldstein AFTPM/DECRG."— Presentation transcript:
The SPADA/MCLE impact evaluation: work in progress Markus Goldstein AFTPM/DECRG
The acronyms This is an impact evaluation of the Mediation and Community Legal Empowerment of the Support for Poor and Disadvantaged Areas Project in Indonesia It’s a big project, and the impact evaluation will only be for one component (other impact evaluations for other components)
The big idea We have (almost) no rigorous evidence of the effects of governance reform Why? –It’s hard to measure –It’s hard to set up Setting up control groups Getting sufficient sample size –It’s political science & IE has tended to come from economists We know very little about poverty-governance- justice nexus
MCLE: project objectives 1.Strengthen the capacity of institutions and individuals to resolve disputes in an open, independent, and just manner 2.Assist and facilitate the resolution of disputes involving community wide interests 3.Increase public trust in the formal legal system by enhancing community legal awareness and access to the system
MCLE: project activities Community legal education Capacity building for informal justice actors Legal aid and complaint handling network Awareness raising of community legal needs and challenges NGO implemented under government contract
MCLE activities - 2 Support network for providing legal aid and empowerment activities Focus at community level Sub-district: establish posko (community legal aid posts), staffed by facilitator and providing legal information, mediation and complaints handling Village level: paralegals (one man/one woman) – work with local leaders and community figures District level: community lawyer – part time technical legal support
Why evaluate this project? Global –What does access to law get you? –What works to get people access to justice? –Top down vs bottom up access to legal system & formal vs informal dispute resolution Local –Gov’t is preparing first ever access to justice strategy for inclusion in the medium-term development plan –Inform nation-wide poverty and empowerment program –Scale up or not?
Evaluation questions: primary Does MCLE increase beneficiaries ability to claim rights, enforce rights, and resolve disputes through non-violent means? Does MCLE reduce violent conflict and increase safety and security? Does MCLE improve HH welfare in the villages that receive paralegals?
Note on analyzing the causal chain… Idea here is to look at different points of the causal chain: Knowledge of rights (ask about legal K) Enforcement of rights (e.g. get a land cert.) Better dispute resolution (e.g. mode used, hypothetical dispute resolution) Investment in property (e.g. in land improvements) Improvements in household consumption
Evaluation questions: secondary Does MCLE increase the reporting of corruption and bribery and decrease these events in the long term? Does MCLE improve individuals’ capacity to engage with the state and the state capacity to respond, particularly for dispute resolution? Does MCLE improve gender relations at the household and community level?
Evaluation design Two provinces: selected for level of conflict (Aceh and Maluku) Districts chosen by gov’t based on need and demand: issues of generalizability Sub-districts: random selection agreed by gov’t Villages: random selection Randomization done in Washington with evaluation team
Evaluation design: quantitative core Method is a randomized difference in difference Baseline quant survey to run right as contracting for implementation goes out Mid-point to run 2 years after implementation (early results for policy) End-line survey to run 4 years after implementation
Evaluation design: survey structure Innovative household survey (“best of” other governance/justice surveys) –Pretest and adjust –Links with SUSENAS –Rotating panel community survey (capture major disputes, village level variables) Key informant survey (capture village/district authority attributes, perceptions of safety, corruption, legal claims) Program facilitator survey (get at heterogenity of treatment – ed levels, experience, social connections)
Evaluation design: other data sources Newspaper monitoring: conflict levels, security and safety, reported corruption Project MIS – implementation and also dispute tracking and other case handling information
Evaluation design: Qualitative core Quant answers: does MCLE have an impact while Qual answers: How does it have an impact Village mapping (understanding village dynamics) –Village profile (demographics, livelihoods, etc) –Dispute timeline (types, sequence, actors,etc) –Claims and collection timeline (indiv vs community) –Power relations and networks – map these
Evaluation design: Qualitative core Comparative case studies –In-depth studies in treatment and comparison locations – similar to quant in structure Community ethnographies –Two per province –How does program work, how does it interact w/local power structures, experiences of key individuals Cooperation across qual-quant data collection, not just iterative data collection
Complication: CRT Add a conflict resolution training component in a select number of villages Creates sampling headache Power issues: the effect of treatment A vs treatment B requires (in general) a larger sample than treatment A vs control Solution: lay it on top of MCLE (still have avg effect, control for CRT), separate CRT only villages. CRT contributes to surveys Danger here is that things get too complicated and we end up with nothing done well…
Evaluation team East Asia SDV: project design team, evaluation/survey expertise PRMPR/DECRG: evaluation design/analysis team (quant/qual) Gov’t: project design/M&E staff Legal: eval design, guidance, dissemination, funding DECRG: survey design specialist Consultants: sampling, local qualitative work Survey design firm
Dissemination plan Core team builds design Disseminate evaluation design w/ wider gov’t audience, implementing NGOs Baseline analysis – informs program design, gov’t agencies IE results: –Workshop w/ gov’t early – compare treatment arms –Disseminate more widely – other relevant programs (national pov/empowerment program), NGOs, Bank, donors, academics, regional J4P program
Budget and funding Survey costs: still unknown, problem of Maluku islands Qual contract out Initial seed funding from BNPP grant, Aceh TFs, project $. More grant applications on the way Donated time helps a lot (save on BB) Missions at end of FY…
A couple of points of reflection The process was important, esp initial workshop with gov’t & project team Project & IE team really, really have to work together (lots of early am VCs) Be prepared for transitions in all teams Doing something new? Early survey pretest helps for sample size calculations Timing of “end-line” survey depends on policy cycle – but also time for effects to manifest Keep it as simple as you can