Presentation is loading. Please wait.

Presentation is loading. Please wait.

N ON -E XPERIMENTAL M ETHODS Shwetlena Sabarwal (thanks to Markus Goldstein for the slides)

Similar presentations


Presentation on theme: "N ON -E XPERIMENTAL M ETHODS Shwetlena Sabarwal (thanks to Markus Goldstein for the slides)"— Presentation transcript:

1 N ON -E XPERIMENTAL M ETHODS Shwetlena Sabarwal (thanks to Markus Goldstein for the slides)

2 O BJECTIVE Find a plausible counterfactual Every non-experimental method is associated with an assumption The stronger the assumption the weaker the estimate TEST ASSUMPTIONS Reality check

3 P ROGRAM TO EVALUATE  Hopetown HIV/AIDS Program ( )  Objectives  Reduce HIV transmission  Intervention: Peer education  Target group: Youth  Outcome Indicator: Pregnancy rate (proxy for unprotected sex)

4 I. B EFORE - AFTER IDENTIFICATION STRATEGY Counterfactual: Rate of pregnancy observed before program started EFFECT = After minus Before

5 I. B EFORE - AFTER IDENTIFICATION STRATEGY Counterfactual: Rate of pregnancy observed before program started EFFECT = After minus Before Year Number of areas Teen pregnancy rate (per 1000) Difference+3.47

6 C OUNTERFACTUAL ASSUMPTION : No change over time Effect = Intervention Question: what else might have happened in to affect teen pregnancy?

7 Number of areas Teen pregnancy (per 1000) TEST ASSUMPTION with prior data REJECT counterfactual hypothesis of no change over time

8 II. N ON -P ARTICIPANT I DENTIFICATION S TRATEGY Counterfactual: Rate of pregnancy among non-participants Teen pregnancy rate (per 1000) in 2012 Participants66.37 Non-participants57.50 Difference+8.87

9 C OUNTERFACTUAL ASSUMPTION : Without intervention participants have same pregnancy rate as non-participants Effect = Participants Non-participants Question: how might participants differ from non- participants?

10 T EST ASSUMPTION WITH PRE - PROGRAM DATA ? REJECT counterfactual hypothesis of same pregnancy rates

11 III. D IFFERENCE - IN -D IFFERENCE IDENTIFICATION STRATEGY Counterfactual: 1.Non-participant rate of pregnancy, purging pre-program differences in participants/nonparticipants 2.“Before” rate of pregnancy, purging before-after change for nonparticipants 1 and 2 are equivalent

12 Average rate of teen pregnancy in Difference ( ) Participants (P) Non-participants (NP) Difference (P=NP) III. D IFFERENCE - IN -D IFFERENCE IDENTIFICATION STRATEGY

13 = – = 3.47 Non-participants Participants Effect = 3.47 – = III. D IFFERENCE - IN -D IFFERENCE (1) Nonparticipants purging before/after

14 After Before Effect = 8.87 – = – = – = III. D IFFERENCE - IN -D IFFERENCE (2) Before purging participation

15 III. D IFFERENCE - IN -D IFFERENCE

16 III. D IFFERENCE - IN -D IFFERENCE

17 C OUNTERFACTUAL ASSUMPTION : Question: why might participants’ trends differ from that of nonparticipants? Without intervention participants and nonparticipants’ pregnancy rates follow same trends

18 T EST ASSUMPTION WITH PRE - PROGRAM DATA Average rate of teen pregnancy in Difference ( ) Participants (P) Non-participants (NP) Difference (P=NP) ? REJECT counterfactual hypothesis of same trends

19 IV. M ATCHING WITH D IFFERENCE - IN - D IFFERENCE IDENTIFICATION STRATEGY Counterfactual: Comparison group is constructed by pairing each program participant with a “similar” nonparticipant Minimize differences in the vector of observed characteristics between participant and nonparticipant Parametrically (propensity score matching) Nonparametrically

20 C OUNTERFACTUAL ASSUMPTION Question: how might participant differ from matched nonparticipant? Unobserved characteristics do not affect outcomes of interest

21 Matched nonparticipant Participant Effect = C OUNTERFACTUAL ASSUMPTION

22 T EST ASSUMPTION WITH EXPERIMENTAL DATA REJECT counterfactual hypothesis Meta-analysis of studies using experimental data to estimate bias in impact estimates using matching: unobservables matter! direction of bias is unpredictable!

23 V. R EGRESSION DISCONTINUITY IDENTIFICATION STRATEGY Applicability: When strict quantitative criteria determine eligibility Counterfactual: Nonparticipants just below the eligibility cutoff are the comparison for participants just above the eligibility cutoff

24 C OUNTERFACTUAL ASSUMPTION Question: Is the distribution around the cutoff smooth? Then, assumption is reasonable However, can only estimate impact around the cutoff, not for the whole program Nonparticipants just below the eligibility cutoff are the same as participants just above the eligibility cutoff

25 E XAMPLE : E FFECT OF SCHOOL INPUTS ON TEST SCORES Target transfer to poorest schools Construct poverty index from 1 to 100 Schools with a score <=50 are in Schools with a score >50 are out Inputs transfer to poor schools Measure outcomes (i.e. test scores) before and after transfer

26

27 Non-Poor Poor

28

29 Treatment Effect

30 S UMMARY Gold standard is randomization – no assumptions needed, always precise estimates Non-experimental requires assumptions – can you live with them? We did not cover: – Encouragement design – Instrumental variables – Pipeline comparisons


Download ppt "N ON -E XPERIMENTAL M ETHODS Shwetlena Sabarwal (thanks to Markus Goldstein for the slides)"

Similar presentations


Ads by Google