Presentation is loading. Please wait.

Presentation is loading. Please wait.

Issues and Challenges in M&E of Rural Livelihoods Programs: Designing Feedback Loops Rohini Pande Mohammad Kamal Professor of Public Policy 12 August,

Similar presentations


Presentation on theme: "Issues and Challenges in M&E of Rural Livelihoods Programs: Designing Feedback Loops Rohini Pande Mohammad Kamal Professor of Public Policy 12 August,"— Presentation transcript:

1 Issues and Challenges in M&E of Rural Livelihoods Programs: Designing Feedback Loops Rohini Pande Mohammad Kamal Professor of Public Policy 12 August, Evidence for Policy Design Harvard University

2 2 The NRLM Framework Proposal: Increase household income of rural poor via sustainable livelihood enhancement and improved access to financial and non-financial services Mission components: Social, financial, and economic inclusion Mission features: Decentralized implementation Prospective design of M&E framework

3 3 NRLM M&E: What we want to know? 1.Why do we expect NRLM to achieve its goals? 2.How do we know that, in practice, it is achieving these goals? 3.If it isnt, how can we modify it so that it does achieve the goals?

4 4 Talk Outline A Holistic view of M&E: Smart Policy Design Policy development Monitoring and Feedback loops Working with smart policy design and feedback loops Testing design features for microfinance MIS systems as feedback mechanisms: Data visualization in MNREGA Final summing up

5 5 Talk Outline A Holistic view of M&E: Smart Policy Design Policy development Monitoring and Feedback loops Working with smart policy design and feedback loops Testing design features for microfinance MIS systems as feedback mechanisms: Data visualization in MNREGA Final summing up

6 6 Smart Policy Design: Policy Development What is the policy goal Is existing evidence consistent with the theory? What is your theory for how to achieve the goal? Refine objectives, theory Policy design

7 7 Smart Policy Design: Monitoring and Feedback Loops Policy design and pilot rollout Is policy achieving its goals Refine objectives, theory Large scale rollout but keep monitoring monitor

8 8 Talk Outline A Holistic view of M&E: Smart Policy Design Policy development Monitoring and Feedback loops Working with smart policy design and feedback loops Testing design features for microfinance MIS systems as feedback mechanisms: Data visualization in MNREGA Final summing up

9 9 Objective: Promotion and growth of micro- enterprises Smart Policy Design: Testing Design Features in Microfinance

10 10 From Objective to Policy Objective: Support growth of micro-enterprises. Hypothesis: Micro-enterprises have high growth potential but are limited by lack of financing. Policy: Access to Microfinance

11 11 Smart Policy Design: Policy Development What is the policy goal Is existing evidence consistent with the theory? What is your theory for how to achieve the goal? Refine objectives, theory Policy design

12 12 Is existing evidence consistent with theory? The typical evidence provided to examine microfinance impacts: 1.High returns to providing enterprises cash 4-5% monthly returns 2.High rate of growth of microfinance use through India 3.High repayment rates With most MFIs posting over 90% repayment rates Concludes: Microloans are a sustainable way of providing the poor finance

13 13 Is existing evidence consistent with hypothesis? But is that the correct evidence support the hypothesis? – Directly test impact of microfinance on enterprize growth – Rigorous impact evaluations show small average impact Triangulate evidence to reformulate design 1.Why could this be given that micro-entrepreneurs have high growth potential and site credit constraints as a predominant obstacle to the growth of their businesses? 2.How can we design an evaluation to test where the answer lies? For redesign: Pilot new policies

14 14 How should policy be redesigned?or what feedback is needed? One possibility: Replace private sector providers by community organizations But why will this solve the problem? Another possibility (and one we test) Maybe the microfinance contract is too rigid and this implies entrepreneurs dont undertake high risk high return activities.

15 15 Refine policy: What limits returns to microfinance? Policy design Theory: Entrepreneurs limited by lack of credit Empirics: High returns to capital, but impact of microfinance limited Conceptualframework Policy goal : Encourage entrepreneurship From Objective to Policy Goal

16 16 Smart Policy Design: Monitoring and Feedback Loops Does the evidence enable better policy design? How will we monitor? What is the pilot policy

17 17 Identify pilot policy What do we know about microenterprises? Subject to frequent household and demand shocks High rate of business closings Lack of insurance to overcome shocks Irregular income streams with businesses highly subject to seasonality Conclude: Microentrepreneurs unable to make profitable investments that require a longer return horizon Need flexibility in loan contract to make such investments

18 18 What do we know about microfinance contracts? Standard Grameen microcredit loan contract – Immediate repayment obligation: One week between loan disbursement and first repayment – Frequent loan repayment: Weekly repayment Rigid contract Conclude: Test impact of increasing repayment flexibility to microentrepreneurs. – Provide a grace period between loan disbursal and repayment

19 19 Competing theories! 1. Our theory: Presented above 2. Competing theory: Contractual flexibility can have adverse effects Rigid repayment schedule is necessary for self-discipline? Removing repayment rigidity can actually encourage poor business decision-making Reduce business profits and rate of repayment Recall need for evaluation

20 20 Smart Policy Design Method What is the pilot policy? How to monitor

21 21 Isevidence consistent with the theory? The question: Does a grace period before loan repayment starts enable enterprise growth? Method to test the question 1 : Work with large MFI provider in Kolkata By lottery select some MFI groups that receivethe grace period contract Other groups remain on standard contract 1 Field, Pande, Papp, and Rigol (2013)

22 22 Pilot design Half of study sample receives standard contract Immediate repayment obligation: Two weeks between loan disbursement and first repayment Loan repayment:Fortnightly repayment Half of study sample receives contract with a grace period Immediate repayment obligation: Two months between loan disbursement and first repayment Loan repayment: Fortnightly repayment

23 23 Results

24 24 Smart Policy Design: Monitoring and Feedback Loops Policy design and pilot rollout Is policy achieving its goals Refine objectives, theory Large scale rollout but keep monitoring monitor

25 25 Using this evidence to enable smart policy design for NRLM 1.Under NRLMs SHGschoose loan repayment terms for group members. 2.IS Trade-offs faced by SHG similar to MFI providers? If SHG has incentives to ensure high repayment but the group doesnt benefit from single members success then they may go for conservative rigid repayment schedule Policy question: How can SHG incentives be aligned with those of individual member? What is the hypothesis? What evaluation mechanism exists for feedback?

26 26 Talk Outline A Holistic view of M&E: Smart Policy Design Policy development Monitoring and Feedback loops Working with smart policy design and feedback loops Testing design features for microfinance MIS systems as feedback mechanisms: Data visualization in MNREGA Final summing up

27 27 MIS and Smart Policy Design: Example of NREGA Smart policy design is about building feedback loops into the policy process. The NREGA data visualization project seeks to do this by: Identifying key program performance indicators Presenting descriptive statistics with an eye toward digestibility and aesthetics Facilitating basic hypothesis testing – investigate correlations between key outcomes and administrative inputs Making program data easier to access for the public and academics (encourage external feedback)

28 28 Smart Policy Design Framework: First causal hypotheses Use simple descriptive statistics to identify problem areas and confirm/reject initial hypotheses Develop and investigate explanations for observed phenomena Start with simple correlations between policy inputs and outcomes; move on to more complex regression analysis, field evaluations; update hypotheses at each iteration Idea to build increasingly defensible causal explanations

29 29 NREGA implementation: Need for focus The NREGA MIS collects dozens of indicators at multiple levels of granularity –Even highly motivated implementers may not have capacity to sift through information and get a clear picture of program performance These need to be distilled into a few key measures of program performance such as female and SC/ST participation or average time to payment.

30 30 Example: female participation in NREGA Getting data on female participation currently takes 4+ clicks (if you know where to look). District-level data only available state-wise, block-level data only available district wise – hard to get a broad view of performance without resorting to data scraping Increasing data accessibility is low-hanging fruit

31 31 Reusable charts can increase data accessibility Web technology exists to pull data from MIS and display it as interactive graphics Charts are reusable – we can use the same chart format to display many user-selected indicators from the MIS Technology (JavaScript and the d3.js library) is open source with a large support community

32 32 Reusable charts can increase data accessibility

33 33 Reusable charts can increase data accessibility

34 34 Ongoing work: Data visualization as a feedback loop How should administrators use MIS/data visualization to identify potential policy levers? Possible hypotheses: 1.Demand Side: Absolute supply of work affects female participation through male-first work allocation 2.Supply side: Higher female empowerment implies greater female participation First step (work in progress!): Develop software that combines different data to generate user-defined scatter Such initial diagnostics can help kickstart smart policy design.

35 35 Demand side: Examining correlations

36 36 Supply side: Examining correlations

37 37 Talk Outline A Holistic view of M&E: Smart Policy Design Policy development Monitoring and Feedback loops Working with smart policy design and feedback loops Testing design features for microfinance MIS systems as feedback mechanisms: Data visualization in MNREGA Final summing up

38 Smart Policy Design: The process 38 Identify your objective Develop a theory that explains why a specific policy will achieve the objective –What is the mechanism by which the policy causesthe objective to be achieved? –What is evidence that these causal links will work? How similar is problem to other contexts from which we have evidence on similar policies? What more do you need to know? Use monitoring and feedback loops to generate this evidence

39 Takeaways for Designing NRLM M&E 39 M&E should not stand apart from policy design and implementation process. Rather its design should ensure feedback loops with policy process. Shouldhavelittle to no distinction between how to think about monitoring versus evaluation. Monitoring mechanisms should enable concurrent evaluation. In all cases, iterate between theory, existing evidence and pilots.


Download ppt "Issues and Challenges in M&E of Rural Livelihoods Programs: Designing Feedback Loops Rohini Pande Mohammad Kamal Professor of Public Policy 12 August,"

Similar presentations


Ads by Google