Heterogeneous impact of the social program Oportunidades on contraceptive methods use in young adult women living in rural areas: limitations of the regression discontinuity approach Héctor Lamadrid-Figueroa 1, Gustavo Ángeles 2, Tom Mroz 3, José Urquieta 1, Aurelio Cruz 1 and MM Téllez-Rojo 1 1 National Institute of Public Health, MEXICO. 2 University of North Carolina at Chapel Hill, 3 Clemson University.

Overview The regression discontinuity design (RDD) The Oportunidades program Impact of Oportunidades on the use of contraception –RDD results vs. Intent to treat results –Why doesn’t RDD work in this setting? Graphical methods Conclusions

Regression Discontinuity Design (RDD) Needs a continuous eligibility index Those below a certain cutoff point are enrolled or offered the program Provides an unbiased estimate of the treatment effect at the vicinity of the cutoff point (LATE) Eligibility score

Oportunidades Formerly Progresa, intends to “break the intergenerational cycle of poverty”. Cash transfers conditional on.. –Keeeping kids at school –Attendance to “health talks” Information on family planning (reduction of fertility is one of the program’s goals) For eligibility to the program the household has to be below a threshold in a poverty index. For evaluation purposes, rural communities were randomized to receive or not receive the program for a certain period.

Oportunidades The cutoff was designed at 752 points However... –Eligibility was not defined solely on the basis of the poverty score... For evaluation purposes, rural communities were randomized to receive or not receive the program for a certain period.

What we did... Take advantage of the experimental design of the Oportunidades evaluation effort... –Compare several analytical strategies, including RDD, to estimate the impact of Oportunidades in the use of FP methods in year old women. –The sample was comprised of 2239 young adult women in 395 communities.

Descriptive statistics for study subjects at baseline (1997). † Dichotomous variable, 1=yes, 0=no.‡ Arithmetic mean for numeric variables, proportion for dichotomous variables.*Mann-Whitney test for numeric variables,  2 for dichotmous. ** p value from multiple logistic regression model where treatment assignment is the outcome, robust standard errors with clustering at locality level were calculated. Global test of significance: p=0.17.

Descriptive statistics for study subjects in the year † Dichotomous variable, 1=yes, 0=no. ‡ Arithmetic mean for numeric variables, proportion for dichotomous variables. * Mann-Whitney test for numeric variables, c2 for dichotomous.

Intent-to-treat effect estimates. All models are OLS. Impact estimate

OLS and 2SLS regression models of contraceptive methods use among 20 to 24 year old women in treatment areas, with varying poverty score window width. In 2SLS models Eligibility was instrumented by an indicator of the individual being below the poverty score cutoff point used to determine eligibility for most households.

LOWESS of the use of contraception by treatment assignment and eligibility

LOWESS of the use of contraception by treatment assignment, 90% confidence bands obtained by a 1000 repetition bootstrap performed on clusters (communities).

LOWESS of the IMPACT of the program on contraception, 90% confidence bands obtained by a 1000 repetition bootstrap performed on clusters (communities).

Conclusions This is a reminder that usual methods only provide an average estimate of effect, and do not tell us anything about the distribution of effects. RDD in particular only provides a LOCAL average treatment effect at the vicinity of the cutoff. Doesn’t tell us much about what happens far from the cutoff. In this case RDD greatly underestimates the program impact in the poorest. Good news: The program has a large impact on the poorest; estimated impact is at least 5 pp increase for 60% of the eligibles! Those poorest and receiving the treatment actually have the highest contraceptive use.

Bad news: The program appears to have a negative effect on those near the threshold. So far this is unexplained... Conclusions