1 What is the Evidence of the Impact of Microfinance on the Well-being of Poor People? Maren Duvendack 1, 3 Richard Palmer-Jones 1 With J. Copestake 2, L. Hooper 1, Y. Loke 1, N. Rao 1 3ie/LIDC, 29 June 2011 Funded by 1. UEA, 2. University of Bath 3. IFPRI

2 Introduction Objectives:  Assess impact of microfinance on social and economic well-being of people living in developing countries and are poor, excluded or marginalised within their own society. Methods:  Adapted to social science studies from Cochrane and Campbell Collaboration and EPPI-centre guidelines Results:  2 RCTs, 9 pipeline studies, remainder w/wo studies.  No robust evidence on most economic, social and empowerment outcomes.  Negative as well as positive impacts.  Many studies based on weak research designs and problematic analysis.

3 Inclusion Criteria Participants:  in poor, lower and upper-middle income countries Intervention:  credit, credit plus and credit plus plus Comparison group:  control group w/o microcredit Outcomes:  economic, social and empowerment outcomes Cut-off point:  studies published since 1970 Methodologies:  RCTs, pipelines, before/after & with/without studies, >100 cases Publication status:  formal and informal  74 papers examining approx 20 broader economic, social and empowerment outcomes

4 Search Strategy Search strategy Records identified through database (11) searching n=3,620 Additional record identified through other sources (12)n= 115 Records after duplicates removedn=2,643 Screening Records screened (abstracts and titles)n=2,643 Records excludedn=2,442 Eligibility Full-text articles assessed for eligibility n = 201 Full-text articles excluded, with reasonsn = 127 Included Papers included in synthesisn = 74 Further screening -> n = 58/29 studies

5 Data Extraction & Validity Assessment  Assessment of validity focused on: Assessing the intervention & measurement of outcomes Contextual factors affecting outcome heterogeneity (sub-group analysis) Research design & analytical method  Assessment took a long time because Abstracts not well structured or methodologically informative  Few studies used rigorous research designs RCTs, pipeline, with/without, natural experiments  Most/all papers suffered weak external and/or internal validity  Reliance on sophisticated statistical analysis to obtain impact estimates Need for replication because of  Data processing or computational errors,  Alternative analytical methods and/or assumptions etc.

6 Therefore select studies based on scoring Selection based on Hierarchy of methods Weak research design requires more sophisticated methods of analysis to attain similar levels of validity High validity Low validity RCTs, pipelines; with and without; natural experiments 2SLS, instrumental variables, PSM, DiD, control function Combined score into an index with fuzzy cut-off point Arbitrary but transparent approach reduced 74 papers 58 papers Strict inclusion criteria  too few studies meet orthodox validity criteria but relaxing inclusion criteria  too many included!

Results – Research Designs 2 RCTs – several problems – not gold standard  Weak external and internal validity Banerjee et al., 2009/10; Karlan and Zinman, 2010 Pipelines  Are present and future participants good matches?  Drop-outs, graduates, contextual factors, small samples With and without – panel and cross section  High risk of bias Requires complex statistics of questionable validity Doubts about data-mining, and replicability of analyses Need to compare with alternative assumptions about causation 7 You cannot substitute weak research design (and poor data) with complex statistics (Meyer and Fienberg, 1992)

8 Results – Number of Outcomes Most outcomes tested are early in the causal chain Huge number of tests – 58 papers 2869 impact estimates Cells = Number of estimatesType of lending Outcome variable ProductIndividual Group & individual Group Economic Credit only Credit & savings Credit plus Social Credit only Credit & savings Credit plus0026 Empower ment Credit only Credit & savings Credit plus0020

Results – Impact Signs and Significances More estimates non-significant than significant  Majority of +ive impacts, but most not statistically significant  Many negative impacts (significant and non- significant) Multiple testing – many estimates using same data  Failure to adjust test statistics (many use 10% level) Ethical research, analysis and dissemination  No evidence on missing studies, or analyses 9

10 Conclusion Common belief is that microfinance is pro-poor and pro-women  BUT: little convincing evidence Almost all MF IEs have high vulnerability to bias  Most worryingly RCTs which are the best regarded/widely quoted studies  Suffer from weak methodologies and inadequate data which adversely affects impact estimates Unclear under what circumstances and for whom MF works Recommendations  Focus on need for more and better research  Conduct well designed quasi-experimental and observational studies including longitudinal studies, earlier in fashion cycle  Replication of highly regarded studies of whatever research design  Capacity building in multi-disciplinary & mixed methods research

11 THANK YOU!

Causal Pathways

13 Results – Distribution of Methodologies Statistical Methods of Analysis Research Design IV,PSM,2SLS/LIML, DIDMultivariateTabulation Scores123 RCT121 Pipeline2900 Panel or b/a & w/wo31460 Either b/a or w/wo Natural Experiment5200 Observation/ survey6000 Legend Higher validity50Low validity16 Medium validity6Excluded * 2 papers (Chen and Snodgrass, 1999 and Dunn, 1999) are included in our analysis but are missing from this table since they had a high score (2 and above). We included them in our synthesis because they were part of a group of papers that used the same data set, i.e. the USAID data on India and Peru.