Presentation is loading. Please wait.

Presentation is loading. Please wait.

Evaluation Designs in the IMATCHINE project: Regression Discontinuity Cluster Randomized Trial Presentation by Manoj Mohanan, Duke University 1 COHESIVE-INDIA.

Similar presentations


Presentation on theme: "Evaluation Designs in the IMATCHINE project: Regression Discontinuity Cluster Randomized Trial Presentation by Manoj Mohanan, Duke University 1 COHESIVE-INDIA."— Presentation transcript:

1 Evaluation Designs in the IMATCHINE project: Regression Discontinuity Cluster Randomized Trial Presentation by Manoj Mohanan, Duke University 1 COHESIVE-INDIA

2 Commercial Break! A word about COHESIVE-India COHESIVE-India: Collaboration for Health Systems Impact Evaluation in India. Jerry La Forgia (World Bank) Grant Miller (Stanford U. & NBER) Manoj Mohanan (Duke U.) Marcos Vera-Hernandez (U. College London & IFS) Focus on evaluation in health sector using a combination of quasi-experimental and experimental methods Provide critical input into the design of policies and interventions, to provide rigorous evidence on how to improve performance as part of evaluation. Collaborating with SAMBODHI and DFID-India on IMATCHINE project 2 COHESIVE-INDIA

3 Key Question in Impact Evaluation What is the counterfactual? What would have happened if this program did not exist? Identifying CAUSAL effects is the central to impact evaluation Two common fallacies in making causal inference: Cum hoc ergo propter hoc Post hoc ergo propter hoc

4 Cum hoc ergo propter hoc Source:

5 The post hoc fallacy Observe people on the 9AM to predict weather? In medicine, there is a disease progression, hence you can use temporal changes to make causal claims. In social sciences, less so.

6 Visualizing problems in evaluation Participants Counter factual This is why the Before-After method is sometimes called The “Counterfeit Counterfactual” method!! TIME  OUTCOMES 

7 Outline of Today’s Presentation Projects being evaluated in the IMATCHINE project Gujarat: (Regression Discontinuity)  Chiranjeevi Yojana (CY) Karnataka: (Experimental Evaluation)  Thayi Bhagya Yojana (TBY) 7 COHESIVE-INDIA

8 Gujarat: Chiranjeevi Yojana (CY) Introduced in 2005 Response to acute shortage of OBGYNS in public sector Leveraging presence of private providers in rural areas Pays approx Rs to accredited provider per delivery Eligibility: BPL card holder or BPL eligible (~23% of population; total pop 55 million) 8 COHESIVE-INDIA

9 More on CY First introduced in 5 backward districts and then rolled out across the state Jan onwards B/w 2005 – Feb 2008, CY had covered over 165,000 deliveries provided by 852 providers Claims: (in 2009)  Has increased institutional births from a national average of 57% to over 80%  Has reduced MMR & IMR  Won WSJ Innovations Award & is now widely looked upon as the “model” 9 COHESIVE-INDIA

10 Previous ‘Evaluations’ of CY Source: Mavalankar, D. et al Saving mothers and newborns through an innovative partnership with private sector obstetricians: Chiranjeevi scheme of Gujarat, India. International Journal of Gynecology and Obstetrics 107: 271–276. Have typically used data from CY facilities to extrapolate estimated utilization and health benefits in the population 10 COHESIVE-INDIA

11 Gujarat: Regression Discontinuity Evaluation Design The program uses the BPL line as the eligibility criteria, which allows a quasi- experimental RD design  Since the BPL score is continuous, households immediately next to each other across the BPL line are comparably similar to each other  Discontinuity in program eligibility across the BPL line allow us to test for differences in outcomes that can be attributed to the program 11 COHESIVE-INDIA

12 Gujarat: RD Design …contd… Two main challenges with BPL criteria:  Lots of other state run programs use BPL eligibility (such as JSY, food subsidies),: −SOLUTION: We use a “Difference-in-Difference” framework by relying on timing of introduction and expansion of the program to 5 districts in 2005 and all over the state in 2007 to try to identify program effects of CY.  Manipulation of BPL criteria, resulting in misclassification −SOLUTION: We rely on a “Fuzzy” regression discontinuity strategy, where we calculate the “true” eligibility and then instrument for CY participation using simulated eligibility 12 COHESIVE-INDIA YearPilot DistrictsExpanded Districts 2005BPL + CYBPL 2008BPL + CY

13 Gujarat: RD Analysis First Stage: (instrumenting for CY Participation) Second Stage: (Change in avg. outcomes) Key outcomes:  Rates of institutional delivery  Study is not powered to detect effects on IMR or MMR, but we will collect data on these measures anyway in addition to measures of morbidity that are more common. Also will collect data on HH characteristics 13 COHESIVE-INDIA

14 Gujarat: RD Analysis using the Geographic Spread Accounting for CY roll out in time and space (2 nd difference estimate) Second Stage: (change on avg. outcomes : same equation accounting for roll out) 14 COHESIVE-INDIA

15 BREAK B/W RD & CRT 15 COHESIVE-INDIA

16 Experimental Methods The Randomized Controlled Trial Clearly, the gold standard An obvious solution to the Prog Eval problem, although not always practical: Assigns treatment in a manner that is unrelated to outcomes Two important steps in randomized evaluation 1.Randomly selecting potential participants from population 2.Randomly assign treatment to the group The RCT in IMATCHINE Project…

17 Karnataka: Thayi Bhagya Yojana (TBY) New Conditional Cash Transfer program of Rs for women who prefer to give birth in the private sector Two components (1) Prospective evaluation Since the program uses a BPL eligibility like the CY program, we will use a RD based method, combined with a difference-in-difference  We are working with the Govt. of KN to implement a baseline survey 17 COHESIVE-INDIA

18 KN : Experimental Evaluation of Provider Incentives (2 nd component) Cluster randomized trial of incentives for providers to estimate effect of incentives for improvements in process measures v/s outcome measures  Arm 1: Provider incentives evaluated based on performance on quality of care indicators  Arm 2: Provider incentives evaluated based on improvement of MCH outcomes in catchment area population  Arm 3: Control group, with no incentives 18 COHESIVE-INDIA

19 KN Incentives Experiment: Design Issues 180 rural clusters, in 3 groups of 60 each Each cluster is approximately at the level of the HOBLI (called kasba in N. India) On average: 3 OBGYN providers in each cluster Sample for the study: approx 550 doctors and 18,000 households (100 women who have a baby in catchment area of each cluster) 19 COHESIVE-INDIA

20 Key challenge from the RCT perspective Recall the two important steps in randomized evaluation 1.Randomly selecting potential participants from population 2.Randomly assign treatment to the group The second one is relatively easy – just write a STATA code The first one is the big challenge Need to identify providers whom we can include in the study Both a conceptual and logistic challenge Need to define eligibility based on objective of the experiment 20 COHESIVE-INDIA

21 KN Incentives Experiment: Design Issues …contd… There are three key issues related to measurement in this study:  Definition of clusters and catchment area  Measurement of process measures of quality and health outcomes in a population  Identification of women who have had a baby and interviewing them in time 21 COHESIVE-INDIA

22 Measurement Issues – contd.. Some things to keep in mind: Sample Sizes in Cluster Randomized Trials … the devil called Intra-Cluster Correlation Careful attention to defining eligible population – mapping is VERY effort and resource intensive, but has HUGE pay offs How practical is it to do RCTs? “Politically Robust Randomization” – Gary King et al. Policy Relevance…. Very important topic, but we don’t have time for this today. Working in conjunction w state governments to ensure buy- in and policy impact. Findings from our research in COHESIVE-INDIA

23 Thanks 3ie Government of Gujarat Government of Karnataka DFID World Bank My colleagues at Sambodhi For further details on project contact:  Ms. Manveen Kohli, Project Manager, IMATCHINE, 23 COHESIVE-INDIA


Download ppt "Evaluation Designs in the IMATCHINE project: Regression Discontinuity Cluster Randomized Trial Presentation by Manoj Mohanan, Duke University 1 COHESIVE-INDIA."

Similar presentations


Ads by Google