Impact Assessments in Finance and Private Sector Development: What have we learned and where do we stand? David McKenzie, World Bank.

Background World Bank and National Governments carry out many finance and private sector projects every year, but to date few have been rigorously evaluated. This makes it difficult for us to know what works, why, and what could we do better… Purpose of this workshop is to take a big step towards rectifying this status quo. => But first let’s take stock of what has been possible, and in the process illustrate different methods of evaluation you will be learning in the workshop, as well as issues to keep in mind this week.

What is an impact evaluation? Differs from how “M&E” typically done for Bank/Government projects. – E.g. project disbursed $7 million in loans, trained 2,000 firms, etc. This tells us nothing about effectiveness. Rigorous impact evaluation compares the outcomes of a program or policy against an explicit counterfactual of what would have happened without the program or policy, and takes seriously the issue of why some firms, regions etc. participate in the program and others do not.

Why has evaluation not taken place to date? Perception that many finance and private sector policies lend themselves less to formal evaluations: – Changes in laws or regulations may occur at economy- wide level – Large loan may be given only to one or two banks Lack of incentives for both Bank operational staff and Governments to do Lack of knowledge as to what is possible => Hopefully this week we can overcome these barriers.

What have we learned? A Few examples: How to raise the incomes of the self- employed? Rethinking the design of microfinance Providing insurance to the poor Learning from regulatory reform

What have we learned? How to raise the incomes of the self-employed? Policy Question: Self-employment accounts for important share of employment. But do these enterprises have scope to grow, or are they just stuck in subsistence waiting for wage jobs? – Randomized Experiment in Sri Lanka took random sample of 600 microenterprises and surveyed them. – Then randomly chose half of them to give grants of $100 and $200 to. – Then followed these enterprises for the next three years, surveying them every 3 months.

– Firms randomly chosen to receive these grants can be compared to firms which were randomly chosen not to receive them (Male Results)

(Female results)

What have we learned? Rethinking central precepts of microfinance? – Low returns to female businesses in Sri Lanka provides reason to question exclusive focus on women – Experiment in Philippines found converting group to individual liability led to faster client growth and no change in default (among existing clients) – Experiment in South Africa found marginal loan recipients taking high interest consumer loans were better off 6-12 months later – along with high returns to capital, rejects argument that poor can’t pay high rates. – Large randomized trial of microfinance in India and Philippines finds fairly modest results after 1 year, with little impact on poverty reduction.

What have we learned? Insuring poor farmers – Does offering rainfall insurance increase use of credit by risk-averse farmers Experiment in Malawi finds evidence it does not. – General experience has been low take-up: e.g. 4.6% for one product in India, 17% in Malawi. – Research suggests trust and financial literacy both play roles Implications for designing products – should be simple, and pay out with reasonable frequency.

What have we learned? Learning from regulatory reform Setting up one-stop shops to improve process of starting a business is a popular reform – Often reforms like this are phased in at a municipal or regional level, offering scope for a difference-in-differences evaluation – Example: Mexico simplified the process of business registration – Due to staffing constraints, the federal agency could not implement the reform in all priority municipalities at once, but instead staggered the reforms, introducing them first in some municipalities and then later in others. Among the municipalities identified as priorities for implementation, there was no specification of which should go first. – A Difference-in-differences estimation can then be used to estimate the effect of the reform by comparing changes in business start-ups in the municipalities which got the reforms earlier to the changes in business start-ups in the similar municipalities which were to receive this reform later. – Results show increase in registration and employment, but coming from new entry, not conversion of existing firms.

What have we learned? Learning from regulatory reform – Often reform introduced into entire country at once, but only applies to subset of population. Regression Discontinuity or Diff-in-Diff may be possible. E.g. evaluation of bankruptcy reform in Colombia which reduced costs of re-organizing a bankrupt firm. Active, non-bankrupt firms not affected by law, so can use difference-in-differences comparing active to bankrupt. – Find reform led to improvement in efficiency of bankruptcy procedure.

Lessons 1)Why have these topics been the ones where progress has been made? -Close ties to theoretical work, and in many cases, theoretical effects unclear. -Have been feasible for researchers to do with existing data sets or implementing with an NGO => Big opportunity is to do this with larger projects.

Lessons 2) Variety of tools means evaluation of many finance and private sector programs is possible.

Lessons 3) Ex-ante design beats ex-post - Huge issues with data availability, understanding what program did, how it selected people, etc. ex post. => Need to get researchers involved from initial design of new projects, rather than coming ex post.

Where should we go from here? Scope for much more work in the areas where existing efforts have occurred: microfinance, microenterprises, insurance, regulatory reform. Still only a handful of rigorous studies. Big gains to be had from looking at impacts of policies and projects designed to benefit large numbers of consumers and firms: – Financial literacy and consumer protection – SME sector policies –training, loans, enterprise support – Housing finance => Major area is big policies where too expensive for NGOs or researchers to run themselves – should be in comparative advantage of the World Bank to work with National Governments to do.