What is the evidence of the of the impact of microfinance on the well-being of poor people? a systematic review by: Maren Duvendack Richard Palmer-Jones James G Copestake Lee Hooper Yoon Loke Nitya Rao August 2011

presentation by Paula Stockman Allerstorfer Duvendack M., Palmer-Jones R., Copestake J.G., Hooper L., Loke Y., Rao N. (2011) What is the evidence of the impact of microfinance on the well-being of poor people? London: EPPI-Centre, Social Science Research Unit, Institute of Education, University of London. presentation by Paula Stockman Allerstorfer

Plain Language Summary Microfinace is popular, but there is no clear evidence that it has any impact Following Cochrane & Campbell methods, evaluations of microfinace examined

Plain Language Summary 11 academic databases, four microfinance aggregator and eight non-governmental (NGO) or aid organization websites. We also consult bibliographies of reviewed books, journal articles, PhDs, and grey literature research designs classified into five broad categories; in descending order of internal validity - randomized control trials (RCTs), pipeline designs, with/without comparisons (in panel or cross- section form), natural experiments and general purpose surveys. These five categories are cross-classified with three categories of statistical methods of analysis, which in descending order of internal validity are two-stage instrumental variables methods (IV) and propensity score matching (PSM), multivariate (control function) and tabulation methods.

Plain Language Summary no robust evidence of positive impacts on women’’s status, or girl’’s enrollments - this may be partly due to these topics not being addressed in valid studies (RCTs and pipelines). Well-known studies which claim to have found positive impacts on females are based on weak research designs and problematic IV analyses which may not have survived replication or re-analysis using other methods, i.e. PSM.

Plain Language Summary The report shows that almost all impact evaluations of microfinance suffer from weak methodologies and inadequate data (as already argued by Adams and von Pischke 1992), thus the reliability of impact estimates are adversely affected.

objective The original objective of this SR was to assess the impact of microcredit. This study reviews not only ‘‘credit’’, but also ‘‘credit plus’’ and ‘‘credit plus plus’’ interventions on the social and economic well-being of people living in developing countries who are poor, excluded or marginalized within their own society. Excluded studies that solely look at microsavings and have no geographical focus; only works reported in English. As set out in the protocol, adjusted the original review question from:‘‘What is the evidence of the impact of micro-credit on the incomes of poor people?’’, to ‘‘What is the evidence of the impact of microfinance on the well-being of poor people?’

Sub Questions 1) What is the evidence of the impact of microfinance on other money metric indicators such as microenterprise profits and revenues, expenditure (food and non-food), assets (agricultural, non-agricultural, transport and other assets) and housing improvements? 2) What is the evidence of the impact of microfinance on other human development indicators such as education (enrollment and achievements for adults and children), health and health behavior as well as nutrition? 3) What is the evidence of the impact of microfinance on women’’s empowerment?

Theory Microfinance theoretical literature can be divided into two strands The first addresses problems that poor people have in gaining access to financial services at an affordable cost as a result of their lack of collateral. Lenders are deterred by high costs of collecting reliable information about the incomes that borrowers might be able to lend against The second strand of literature explores impact pathways of microfinance on enterprises, households, and individuals. The ways communities assign access to livelihood opportunities & how problems of access to credit, other income and consumption smoothing opportunities can at least partially be overcome by engagements with MFIs.

gender If credit is successfully targeted to women, it may benefit women by enhancing their status and empowering them It may also beneficially affect the pattern of household resource allocation, particularly benefitting children, especially females

Interventions MFI interventions differ vastly; it is difficult to know if comparing like with like

Outcomes Outcomes can be classified into three groups: economic, social and empowerment.

Research designs RCTs, pipelines, with/without, panels, and natural experiments

This review

methods In order to conduct an unbiased stakeholder relevant review, an (unpaid) advisory group was set up to support the SR and contacted a balanced team of reviewers by approaching representatives of major stakeholders and microfinance adepts. The main objective for this advisory group was to comment on the final outcome of our searches and to evaluate whether the relevant microfinance impact evaluation studies are included. However, we received a response from only one member of the advisory group and proceeded with the SR without this potentially valuable feedback.

inclusion criteria

methodologies Controlled trials, before/after studies, action and observational studies and impact evaluations, and social survey datasets with pertinent indicators. Qualitative studies are assessed for inclusion but set aside and used to scope the literature in this area. Intervention studies including randomized controlled trials, controlled trials, before/after studies and action research assesses the impact of introduction of microfinance to some participants compared to the lack of such introduction in other participants. Observational studies assess outcomes in populations served by microfinance and compare them to outcomes in areas not served by microfinance Publication status: Studies may be formally published or available in abstract, web- based, PhD thesis or organizational report form. Published after 1970

search strategy

Keywords #1 Topic=(evaluat* OR impact* OR benefit* OR poverty* OR empower* OR income* OR profit* OR revenue* OR employ* OR ‘‘labour supply’’ OR job* OR expenditure* OR consume OR consumes OR consumed OR consumption OR asset* OR housing OR education* OR health* OR nutrition*) OR Title=(evaluat* OR impact* OR benefit* OR poverty* OR empower* OR income* OR profit* OR revenue* OR employ* OR ‘‘labour supply’’ OR job* OR expenditure* OR consume OR consumes OR consumed OR consumption OR asset* OR housing OR education* OR health* OR nutrition*) #2 Topic=(microfinanc* OR microcredit* OR micro-credit* OR micro- financ* OR microenterprise* OR micro- enterprise* OR ‘‘group lending’’ OR ‘‘credit program*’’ OR ‘‘credit plus*’’ OR credit-plus*) OR Title=(microfinanc* OR microcredit* OR micro-credit* OR micro-financ* OR microenterprise* O micro-enterprise* OR ‘‘group lending’’ OR ‘‘credit program*’’ OR ‘‘credit plus*’’ OR credit-plus*) #3 #1 AND #2 The draft search for Medline, EMBASE, AMED and Psychinfo (all on OvidSP) and the Cochrane Library (without the ‘‘mp’’ at the end) was: (microfinanc* or microcredit* or micro-credit* or micro-financ* or microenterprise* or micro-enterprise* or ‘‘group lending’’ or ‘‘credit program*’’ or ‘‘credit plus* or credit-plus*’’).mp. This did not need limiting by outcome as few studies were located. *The reference lists of included quantitative studies and relevant reviews were checked for further relevant studies.

Searching & screening The searches were initially screened by one researcher who retrieved full text publications, etc. These were then assessed independently in duplicate by the two lead reviewers using inclusion forms developed for the review. There was a lack of clarity in abstracts (along with a lack of methodological indexing terms) making it was difficult to run specific and sensitive searches, & assessing whether titles and abstracts may relate to relevant studies was extremely difficult. Running the searches and assessing titles and abstracts for collection of full text articles was much more time-consuming than it otherwise would have been.

Assessing validity

Followed the established medical and educational experience embodied in Cochrane and Campbell Collaborations. Assessment of validity initially focused on: checking the delivery and adequacy of the intervention (e.g. provision of microfinance), reliability of the outcome measures (e.g. income, expenditure, assets, and so on), contextual factors affecting heterogeneity of outcomes (including other microfinance services), and potential existence and significance of confounding factors. In the context of microfinance evaluations there were few studies which met the rigorous standards of research design that this approach is based on. Hence, much of the process involved trying to assess validity of analyses of observational data which had further problems in their design.

Many of these problems emerged during the review of papers, which did not have well-structured and methodologically informative abstracts; nor, on closer examination of the text was it easy to extract important details of the methodology (combination of design and analysis) Initial selection criteria evolved during the course of the study, based on further consideration of validity of different designs and analyses In many cases the only way to be clear about the validity of results would have been to replicate19 the studies in order to get a clearer understanding of the ways in which variables had been constructed and analyses undertaken

Synthesis & Selection

Stage one Consists of searches that produced a list with 3,735 publications. 1,092 duplicates were found and removed leaving a final list of 2,643 publications.

Stage two Screen titles and abstracts of 2,643 publications, applying inclusion criteria Shortlist 201 publications & obtain the full text to decide on inclusion or exclusion

Stage three Carefully screen shortlisted publications; this includes 74 papers to analyze in depth; very few papers met rigorous selection criteria Only two RCT microfinance IEs & all papers based on observational data have research design and analytical problems Included many papers based on observational data In selecting among these papers- research design & data analysis approaches scored. Weight attached to the quality of research design and statistical methods of analysis. These scores are then weighted and aggregated and a cut-off value specified to include papers judged to warrant further investigation. Scoring, weighting and aggregation were performed using Excel. The spreadsheet is available from the authors.

Our original plan was to scan papers by validity criteria; this proved burdensome, if not impossible Too many papers failed to provide precise information to answer the questions; subjective judgements were required to complete the forms. Could not be confident about some assignments, few papers could achieve a ‘‘low threat to validity’’ status. A different approach than originally intended was taken, scoring papers by their self-proclaimed research design, and by the analytical procedures The general principle adopted is that weak research design requires more sophisticated methods of analysis in order to reach similar levels of validity. While this may be true, it is also the case that more sophisticated analytical methods may not (fully) compensate for weak research design. The final paper selection was based on a simplified ranking compiled from scores for design and analysis into a single index which varied from 0.0 (low threat to validity) to 2.78 (high threat to validity)23; we used a cut-off at 2 which excluded a significant number of studies with scores clustered just above 2 (see Figure 4). A few papers which were marginally excluded by this approach were included based on our judgement, resulting in a final count of 58 papers

The Studies

Some Findings

RCT findings (2 studies) Few significant impacts were found apart from borrowing amounts and sources –– i.e. on the direct and intended effects of the intervention. (Increased MFI borrowing is not an indicator of well-being) A few coefficients implying impacts on business activities (inventories, profits, etc.), were seen, but these have only indirect effects on well-being, even if some consider increased business activity is (or portends) a good itself. Very few impacts on direct (health, education, subjective well-being) or indirect (income or consumption expenditures) indicators of well-being were found. Do these studies deliver strong evidence that (short- term) impacts on well-being of MFIs are not present? as has been the popular interpretation of these studies- we can accept the null hypothesis of no impact with a high level of confidence. Prominent academics involved in microfinance do not reject the alternate hypothesis. They imply that studies do not provide evidence leading to rejection of the hypothesis that MFIs have beneficent impacts.

Gender Empowerment Some included studies address issues of female empowerment Study authors highlight the short study period during which effects could occur may be insufficient for such impacts to appear & limitations to the study methods suggest there is not enough statistical power Women’’s empowerment is notoriously difficult to measure and the few quantitative studies included in this report that do examine empowerment issues lack credibility due to unresolved issues of measuring womens’ empowerment Qualitative studies may provide more understanding of empowerment, but were excluded from this SR

Conclusion The majority of microfinance IEs included investigate group lending and credit only interventions which do not reflect the diversity of the sector, hence this does not allow us to reach a conclusion as to the impact of the microfinance sector as a whole; individual lending is a more recent phenomenon that has not yet been evaluated widely. Paired with doubts about research designs and analytical methods used by various microfinance IEs, we can neither support nor deny the notion that microfinance is pro-poor and pro-women

Design provided, but changed based on studies Two different data extractors, but way to handle disagreements not clear Comprehensive literature search Discussion of gray literature limited; only English language List of studies provided Characteristics of studies included Scientific quality of studies assessed, but more observational studies with weak design & analysis included Rigor considered, but studies limited and conclusions difficult to draw Difficult to combine studies because of lack of heterogenetity No discussion of publication bias Conflict of interest (none) discussed

thank you