1. innovations in investment climate reforms: an impact evaluation workshop November 12-16, 2012, Paris Non Experimental Methods Florence Kondylis

2. what we know so far We want to isolate the causal effect of our interventions on our outcomes of interest Randomizing the assignment to treatment is the “gold standard” methodology (simple, precise, cheap) What if we really cannot use it? there are other methods (double differences; matching; and discontinuity design) key problem is the search for a counterfactual o question its validity very seriously to ensure we get the results we want

3. innovations in investment climate reforms: an impact evaluation workshop November 12-16, 2012, Paris i. double difference + matching ii. discontinuity design

4. innovations in investment climate reforms: an impact evaluation workshop November 12-16, 2012, Paris i. double difference + matching ii. discontinuity design

5. non-experimental methods Can we find a plausible counterfactual? Is there a group of firms / commercial farmers out there that’s o on average similar to our treatment group o not receiving our intervention? If so, why are they not receiving? o We have to think about this carefully: it is the key to understanding whether this group is / is not a good counterfactual 5

6. example 1: Matching Grants Program in Kundu Principal Objective o Increase firm productivity and sales Intervention o Matching grants distribution o Non-random assignment Target group o SMEs with 1-10 employees Main result indicator o Sales 6

7. 7 (+) Impact of the program (+) Impact of external factors Illustration: Matching Grants - Randomization

8. what if we could not randomize … What’s a good counterfactual? Proposition: Similar firms in Kundu that did not receive the matching grants o If they are so similar, then why did they not receive it? – Too “impatient” to apply >> less motivated to grow? – Smaller on average >> not sufficiently staffed to meet deadline – Did not meet the deadline >> less organized? – Applied but were not selected >> less talented? – Live further from the center of the district where the call was made >> more remote, less profitable? » why did we roll out the program where we did? o Does that tell us these firms would be a good source of comparison for the firms that received the grants? – are these characteristics likely to drive differences in sales, even in the absence of the matching grant scheme?

9. 9 « After » difference btwn participants and non-participants Illustration: if we could not randomize « Before» difference btwn participants and nonparticipants >> What’s the impact of our intervention?

10. Difference-in-Differences Identification Strategy Underlying assumption: In the absence of the matching grants, firms that received the grants and those that did not would have experienced the same volume of sales Is that credible? Need to think critically about >> self-selection into a program >> geographic placement of a program

11. example 2: cellphones and fish market in India Idea: If an intervention affects outcomes immediately, and if you can measure these outcomes >> non-experimental evidence may be extremely convincing Example: Jensen 2007: Cellphone infrastructure, new information access and fish markets in India Theory: with perfect information about markets across space, prices for identical good should be identical (spatial arbitrage); >> cheap information makes markets work better Assume cellphones reduce search cost of getting info about distant fish markets empirically show price impacts of new access to cellphone towers in a specific market

12. key facts: fishing industry in Kerala Huge industry i.t.o. employment, consumers Market frictions: no storage, no transport, only time to sell in 1 market, no way to gather information (before) Before: Lots of price dispersion Lots of waste

13. data and context FIVE YEARS….1996-2001: weekly fish sales (prices and quantities) across 3 fish markets in Kerala, India Unit of observation = the fishing unit Before and after setting for 3 major markets, 15 beach markets Phone TOWERS introduced in 3 waves combine treatment and comparison areas in each week of price data At each landing: 10 large, 10 small units surveyed Total catch, total sales, sale price, fishing costs, fishing location, weather conditions and cell phone use Context: rollout was DEMAND DRIVEN those markets that received coverage first were probably different but rapid response  look for immediate response in price variables

15. how does this get at causal effect? If Y=main outcome (variance of prices) And T1=After, T0=Before Before-After = change in variance of fish prices before to after cellphone introduction, or: Impact  =Y T1 -Y T0 Assume: no contemporaneous region-specific shocks correlated with outcomes (fish prices and quantities) Since outcomes are captured weekly and daily  less chance of this happening Plus, there are three similar experiments results should be similar across all

16. Cellphone adoption rates by week

17. Impact on daily fish prices

18. Main results Cell phones arrived 1997  60% uptake by 2001 Variation in fish prices across markets  0 in all three markets, in about 10 days after tower is switched on Waste (5-8% of catch before) eliminated Profits rise by 6% Prices fall 4% Consumer surplus rises by 6% (small effects)

19. conclusion Basic lesson: With the right context – natural experiment with fast-reacting outcomes – you can get at causal effects of rollout You still need to check “threats to identification”, which may require more data o E.g. did the phones cause other aspects of the industry to change immediately? – Fishermen entered/left the industry? (NO) – Catch size changed bc info about weather is cheaper too (NO)

20. Matching Method + Differences Match participants with non-participants on the basis of observable characteristics Counterfactual: Matched comparison group » Each program participant is paired with one or more similar non-participant(s) based on observable characteristics >> On average, matched participants and non- participants share the same observable characteristics (by construction) » Estimate the effect of our intervention by using difference-in-differences 20

21. Matching Method (2) Underlying counterfactual assumptions After matching, there are no differences between participants and nonparticipants in terms of unobservable characteristics AND/OR Unobservable characteristics do not affect the assignment to the treatment, nor the outcomes of interest

22. How do we do it? Design a control group by establishing close matches in terms of observable characteristics Carefully select variables along which to match participants to their control group So that we only retain o Treatment Group: Participants that could find a match o Comparison Group: Non-participants similar enough to the participants >> We trim out a portion of our treatment group!

23. implications I How does this help us with our original quest for a counterfactual Why did the comparison group (other firms in Kundu) not receive the matching grants intervention? We had agreed that this was because they are – Too “impatient” to apply >> less motivated to grow? – Smaller on average >> not sufficiently staffed to meet deadline – Did not meet the deadline >> less organized? – Applied but were not selected >> less talented? – Live further from the center of the district where the call was made >> more remote, less profitable? Are these observable characteristics?

24. are these observable ? Patience Size of staff Organizational abilities Talent Remoteness

25. implications II In most cases, we cannot match everyone The larger the sample, the higher the probability of finding matches o costly Need to understand which firms are left out Example Score Nonparticipants Participants Matched firms Size / Profit margin Portion of treatment group trimmed out

26. Conclusion Advantage of the matching method Can help the search of a counterfactual where observable characteristics are o determine the impact of the program o can explain participation in the program Yet Often hard to ignore the role played by unobservable characteristics We can only measure the impact for those participants that could be matched to similar non-participants o This requires a lot of data o The outcome from the matching exercise is hard to predict ex ante 26

27. innovations in investment climate reforms: an impact evaluation workshop November 12-16, 2012, Paris i. double difference + matching ii. discontinuity design

28. Regression Discontinuity Designs RDD is closer cousin of randomized experiments than other competitors RDD is based on the selection process When in presence of an official/bureaucratic, clear and reasonably enforced eligibility rule A simple, quantifiable score Assignment to treatment is based on this score A threshold is established o Ex: target firms with sales above a certain amount – Those above receive, those below do not o compare firms just above the threshold to firms just below the threshold 28

29. RDD in Practice Policy: US drinking age, minimum legal age is 21  under 21, alcohol consumption is illegal Outcomes: alcohol consumption and mortality rate Observation: The policy implies that individuals aged 20 years, 11 months and 29 days cannot drink individuals ages 21 years, 0 month and 1 day can drink however, do we think that these individuals are inherently different? wisdom, preferences for alcohol and driving, party-going behavior, etc People born “few days apart” are treated differently, because of the arbitrary age cut off established by the law a few days or a month age difference could is unlikely to yield variations in behavior and attitude towards alcohol  The legal status is the only difference between the treatment group (just above 21) and comparison group (just below 21) 29

30. RDD in Practice In practice, making alcohol consumption illegal lowers consumption and, therefore, the incidence of drunk-driving Idea: use the following groups to measure the impact of a minimum drinking age on mortality rate of young adults Treatment group: individuals 20 years and 11 months to 21 years old Comparison group: individuals 21 years to 21 years and a month old Around the threshold, we can safely assume that individuals are randomly assigned to the treatment the only difference is the application of the law >> We can then measure the causal impact of the policy on mortality rates around the threshold 30

31. RDD in practice 31 MLDA (Treatment) reduces alcohol consumption

32. RDD in practice 32 Total number of Deaths Higher alcohol consumption increases death rate around age 21 Total number of accidental deaths related to alcohol and drug consumption Total number of other deaths

33. RDD Logic Assignment to the treatment depends, either completely or partly, on a continuous “score”, ranking (age in the previous case): potential beneficiaries are ordered by looking at the score there is a cut-off point for “eligibility” – clearly defined criterion determined ex-ante cut-off determines the assignment to the treatment or no-treatment groups These de facto assignments usually result from administrative decisions resource constraints limit coverage very targeted intervention with expected heterogeneous impact transparent rules rather than discretion used 33

34. Example: matching grants (fuzzy design) Government gives matching grants to firms Eligibility rule based on annual sales: If annual sales < $5,000 then firm receives grants If annual sales >= $5,000 then no matching grants A firm with sales=$5,001 would not be treated (be eligible) but would be very similar to a firm with sales=$5,000 Need to measure annual sales before the scheme is announced to prevent manipulation of the figure RDD would compare firms just above and just below the $5,000 threshold 34

35. *subtle* point … Question: How to address incomplete compliance to the treatment Ex: Low take-up of a matching grant scheme There are two types of discontinuity Sharp (near full compliance, e.g. a law) Fuzzy (incomplete compliance, e.g. a subsidy) Going back to our examples … 35

36. Sharp and Fuzzy Discontinuities (1) Ideal setting: Sharp discontinuity, from 0 to 100% the discontinuity precisely determines treatment status o e.g. ONLY people 21 and older drink alcohol, and ALL drink it! o Only small firms receive grants o Progressive taxation rate 36

37. Sharp and Fuzzy Discontinuities (2) Fuzzy discontinuity the percentage of participants changes discontinuously at cut-off, but not from zero to 100% o e.g. rules determine eligibility but amongst the small firms there is only partial compliance / take-up o Some people younger than 21 end up consuming alcohol and some older than 21 don’t consume at all 37

38. Example: matching grant (fuzzy design) Now suppose that not all the eligible firms receive the grants. Why? limited knowledge of the program voluntary participation  these reasons signal a selection bias into the program: decision to enter the program is correlated with other firm characteristics Yet, the percentage of participants still changes discontinuously at cut-off from zero to less than 100% this is called a fuzzy discontinuity (vs. sharp) 38

39. Probability of Participation under Alternative Designs 39 100% 0% 75% 0% Sharp Design for Grant receiptFuzzy Design for Grant receipt Variations above the threshold

40. Internal Validity General idea the arbitrary cut off implies that individuals to the immediate left and right of the cut-off are similar  therefore, differences in outcomes can be directly attributed to the policy. Assumption Nothing else is happening: in the absence of the policy, we would not observe a discontinuity in the outcomes around the cut off. o there is nothing else going on around the same cut off that impacts our outcome of interest would not hold if, for instance: o 21 year olds can start drinking however the moment they turn 21 they have to enroll in a “drinking responsibly” type seminar o Grants: there is another policy that gives grants to firms with sales bigger than $5,000. 40

41. testing the assumption if something else is happening at threshold 41 Different shape

42. external validity How general are the results obtained through RDD? Counterfactual: individuals “marginally excluded from benefits” just under 21 sales just under $5,000  get results for these neighborhoods only  Causal conclusions are limited to individuals, households, villages and firms at the cut-off The effect estimated is for individuals “marginally eligible for benefits” extrapolation beyond this point needs additional, often unwarranted, assumptions (or multiple cut-offs) [Fuzzy designs exacerbate the problem] 42

43. Graphical Analysis 43

44. The “nuts and bolts” of implementing RDDs Major advantages of RDD transparency graphical, intuitive presentation Major shortcomings requires many observations around cut-off o (down-weight observations away from the cut-off) Why? only near the cut-off can we assume that people find themselves by chance to the left and to the right of the cut-off think about firms with $1M sales vs. firms with $1,000 or compare a 16 vs a 25 year-old 44

45. wrap UP Can be used to design a prospective evaluation when randomization is not feasible The design applies to all means tested programs Multiple cut-offs to enhance external validity o Menu of subsidies targeting various types of firms Can be used to evaluate ex-post interventions using discontinuities as “natural experiments”. 45

46. Summary Randomized-Controlled-Trials require minimal assumptions and procure intuitive estimates (sample means!) Non-experimental methods require assumptions that must be carefully tested » More data-intensive » Not always testable Get creative: Mix-and-match types of methods! Address relevant questions with relevant techniques 46