Presentation on theme: "N ON -E XPERIMENTAL M ETHODS Shwetlena Sabarwal (thanks to Markus Goldstein for the slides)"— Presentation transcript:
N ON -E XPERIMENTAL M ETHODS Shwetlena Sabarwal (thanks to Markus Goldstein for the slides)
O BJECTIVE Find a plausible counterfactual Every non-experimental method is associated with an assumption The stronger the assumption the weaker the estimate TEST ASSUMPTIONS Reality check
P ROGRAM TO EVALUATE Hopetown HIV/AIDS Program (2008-2012) Objectives Reduce HIV transmission Intervention: Peer education Target group: Youth 15-24 Outcome Indicator: Pregnancy rate (proxy for unprotected sex)
I. B EFORE - AFTER IDENTIFICATION STRATEGY Counterfactual: Rate of pregnancy observed before program started EFFECT = After minus Before
I. B EFORE - AFTER IDENTIFICATION STRATEGY Counterfactual: Rate of pregnancy observed before program started EFFECT = After minus Before Year Number of areas Teen pregnancy rate (per 1000) 20087062.90 20127066.37 Difference+3.47
C OUNTERFACTUAL ASSUMPTION : No change over time Effect = +3.47 Intervention Question: what else might have happened in 2008-2012 to affect teen pregnancy?
Number of areas Teen pregnancy (per 1000) 200420082012 7054.9662.9066.37 TEST ASSUMPTION with prior data REJECT counterfactual hypothesis of no change over time
II. N ON -P ARTICIPANT I DENTIFICATION S TRATEGY Counterfactual: Rate of pregnancy among non-participants Teen pregnancy rate (per 1000) in 2012 Participants66.37 Non-participants57.50 Difference+8.87
C OUNTERFACTUAL ASSUMPTION : Without intervention participants have same pregnancy rate as non-participants Effect = +8.87 Participants Non-participants Question: how might participants differ from non- participants?
T EST ASSUMPTION WITH PRE - PROGRAM DATA ? REJECT counterfactual hypothesis of same pregnancy rates
III. D IFFERENCE - IN -D IFFERENCE IDENTIFICATION STRATEGY Counterfactual: 1.Non-participant rate of pregnancy, purging pre-program differences in participants/nonparticipants 2.“Before” rate of pregnancy, purging before-after change for nonparticipants 1 and 2 are equivalent
Average rate of teen pregnancy in 20082012 Difference (2008-2012) Participants (P)62.9066.373.47 Non-participants (NP)46.3757.5011.13 Difference (P=NP)16.538.87 -7.66 III. D IFFERENCE - IN -D IFFERENCE IDENTIFICATION STRATEGY
C OUNTERFACTUAL ASSUMPTION : Question: why might participants’ trends differ from that of nonparticipants? Without intervention participants and nonparticipants’ pregnancy rates follow same trends
T EST ASSUMPTION WITH PRE - PROGRAM DATA Average rate of teen pregnancy in 20042008 Difference (2004-2008) Participants (P)54.9662.907.94 Non-participants (NP) 39.9646.376.41 Difference (P=NP)15.0016.53 +1.53 ? REJECT counterfactual hypothesis of same trends
IV. M ATCHING WITH D IFFERENCE - IN - D IFFERENCE IDENTIFICATION STRATEGY Counterfactual: Comparison group is constructed by pairing each program participant with a “similar” nonparticipant Minimize differences in the vector of observed characteristics between participant and nonparticipant Parametrically (propensity score matching) Nonparametrically
C OUNTERFACTUAL ASSUMPTION Question: how might participant differ from matched nonparticipant? Unobserved characteristics do not affect outcomes of interest
T EST ASSUMPTION WITH EXPERIMENTAL DATA REJECT counterfactual hypothesis Meta-analysis of studies using experimental data to estimate bias in impact estimates using matching: unobservables matter! direction of bias is unpredictable!
V. R EGRESSION DISCONTINUITY IDENTIFICATION STRATEGY Applicability: When strict quantitative criteria determine eligibility Counterfactual: Nonparticipants just below the eligibility cutoff are the comparison for participants just above the eligibility cutoff
C OUNTERFACTUAL ASSUMPTION Question: Is the distribution around the cutoff smooth? Then, assumption is reasonable However, can only estimate impact around the cutoff, not for the whole program Nonparticipants just below the eligibility cutoff are the same as participants just above the eligibility cutoff
E XAMPLE : E FFECT OF SCHOOL INPUTS ON TEST SCORES Target transfer to poorest schools Construct poverty index from 1 to 100 Schools with a score <=50 are in Schools with a score >50 are out Inputs transfer to poor schools Measure outcomes (i.e. test scores) before and after transfer
S UMMARY Gold standard is randomization – no assumptions needed, always precise estimates Non-experimental requires assumptions – can you live with them? We did not cover: – Encouragement design – Instrumental variables – Pipeline comparisons