Presentation to Funder Consultation, Luxembourg Sukhwinder Arora Oxford Policy Management The Evolving Role of Donors and DFIs in Advancing Financial Inclusion How do we measure progress in market development? Wednesday 14 November, 2012

Presentation Structure The environment of the market system Challenges in measuring progress for market development projects Difficult context Measuring progress can promote market development! Measurement example - OPMs VFM work for FSD Kenya Wednesday 14 November, 2012© 2011 Oxford Policy Management Ltd2

The environment of the market system Some interventions miss the target entirely THE FINANCIAL SERVICES MARKET DONORS Note the change in the direction of the arrows as they enter the market – a refraction that represents unintended consequences ! OTHER MARKETS e.g. Agriculture, Industry Trade, Infrastructure, Land, Government services For example, demand for financial services includes demand from enterprises involved in every sector. Similar externalities apply to almost every element in the market Source: Robert Stone Wednesday 14 November, 2012 © 2011 Oxford Policy Management Ltd3

Challenges in measuring progress for market development projects A market development approach does not result in any obvious hard outputs that can be measured necessarily involves influencing other players (but attributing success to any set of players would be judgemental/ difficult) involves complex impact pathways – operating directly and indirectly and at every level of the financial sector (Rules, Supporting functions and Core market) – not easily amenable to impact assessment (with or without; time frames etc.) esp. tracking changes in rules of the game/ market support function) facilitates a process of change (but cannot really supply any detailed ex ante statement of the precise results that will eventually emerge from that process) Wednesday 14 November, 2012© 2011 Oxford Policy Management Ltd4

Difficult context Increased scrutiny of aid budgets Limited relevant market data in developing markets Market development is not a science Impacts of M4P programmes not easily quantifiable Logframe indicators can not be aggregated Mixed signals from donors to market actors Getting the balance right between – Short term market development vs systemic change (aligning incentives for market actors, policy makers and donors) – donors taking credit for contribution/ successes and national stakeholders driving the market development process Independent Commission for Aid Effectiveness summed up the dilemma What matters is not always easy to quantify: we recognise that evaluating the achievement of many objectives within the UK aid programme – such as increasing accountability and reducing conflict and fragility – may rely more heavily on qualitative than quantitative analysis. We do not believe it is right to shy away from the difficult-to-measure impacts, as these are often the ones that are most effective in the long term. Wednesday 14 November, 2012© 2011 Oxford Policy Management Ltd5

Measuring progress can promote market development! Why measure progress? Prove Impact Improve Practice Assess Value for Money The process of prioritisation, generation and dissemination of market data and insights is also a critical part of market facilitation and development For Example: Facilitate informed debates amongst key stakeholders Establish common vision/ road map and how to monitor progress Signal what is important Create evidence for setting and tracking policy priorities Identify market opportunities Wednesday 14 November, 2012© 2011 Oxford Policy Management Ltd6

FSD Kenya – value for money assessment Part of knowledge management by FSD Kenya (one of the impact assessment tools) Recognise two types of outcomes/ outputs a) Can be quantified with – plausible assumptions and – reasonable costs for data gathering and analysis b) Difficult to quantify Measurement of four key attributable outcomes Increased number and penetration of bank accounts Money transfers – lower costs and greater volumes Increased availability of bank and non-bank financial outlets The lower risks of SACCO failure Wednesday 14 November, 2012© 2011 Oxford Policy Management Ltd7

FSD Kenya – value for money assessment Key takeaways Not measuring everything can work in well conceived projects where some of the benefits are large enough on their own to justify the full costs of that project The whole result can be transformational - much greater than sum of its parts - but dis-aggregation is not easy e.g. – TA to MFIs plus more enabling rules of the game; – SACCO regulation and increasing competition Outcomes not always correlated with direct expenditure – FSDK advice sometime more valuable then the much larger investments – Right advice on demand can deliver magnified results Success always has many parents (failure is often the orphan) Key difference between ex-post (looking back) and ex-ante (looking ahead) Time frame for measuring results Measurement time and costs proportionate to the programme/ project costs (drive towards VFM of the measurement itself) Explicitly share data sources and assumptions for scrutiny by other stakeholders Wednesday 14 November, 2012© 2011 Oxford Policy Management Ltd8

Distinguishing between Direct and Final Impacts Wednesday 14 November, 2012© 2011 Oxford Policy Management Ltd9

Distinguishing between Direct and Final Impacts Define the theory of change Measuring Direct Impact: start with the project/portfolio data from FSD and its implementing partners – capture FSDs own impacts (attribution is relatively easier but limited tracking of final impacts) Measuring Final Impact: Better at measuring WHAT has changed but weaker on its own to clarify WHY and HOW Need for triangulation of evidence (to confirm attribution) Gathering the evidence for multiple purposes – proving impact, – improving practice, – assessing VFM and – facilitating market development Tracking short term trends and not just long term post mortems Ensuring VFM of the measurement itself Wednesday 14 November, 2012© 2011 Oxford Policy Management Ltd10

References Robert Stone, Susan Johnson and Janet Hayes, FSD Kenya: Impact Assessment, Final Report, January 2010 (OPM and University of Bath). Sukhwinder Arora, Alan Roe and Robert Stone, Assessing Value for Money: The Case of Donor Support to FSD Kenya, OPM Working Paper, June 2012 Sukhwinder Arora, Alan Roe and Robert Stone, Valuing the unquantifiable? How VFM can be applied to complex projects, OPM Briefing Paper, October 2012 Sukhwinder Arora, Alan Roe and Robert Stone, FSD Impact Research and Value for Money Assessment: A Suggested Approach, Report to FSD Kenya, August 2012 Available at Wednesday 14 November, 2012© 2011 Oxford Policy Management Ltd11

Thank you