Treatment Evaluation

Identification Graduate and professional economics mainly concerned with identification in empirical work. Concept of understanding what is the causal relationship behind empirical results.

Selection Bias Example 1: Do hospitals make people healthier?

Selection Bias National Health Interview Survey (NHIS) “During the past 12 months, was the respondent a patient in a hospital overnight?” “Would you say your health in general is excellent, very good, good, fair, poor?” (1 is excellent; 5 is poor)

Selection Bias

Going to the hospital makes people sicker It’s not impossible: hospitals are full of other sick people who might infect us, and dangerous machines and chemicals that might hurt us.

Selection Bias People who go to the hospital are probably less healthy to begin with. Even after hospitalization, people who have sought medical care are not as healthy, on average, as those who never get hospitalized. Though they may well be better after hospitalization than they otherwise would have been.

Selection Bias Example 2: Does college education increase wage?

Selection Bias College graduates earn 84% more than high school graduates.

Selection Bias Selection into college: higher ability, smarter, work harder, etc. College graduates would have earned more even without college education. Simple comparison can not identify the causal impact of college education on wage.

Solution 1: Randomization Random assignment makes the treatment independent of potential outcomes. It eliminates selection bias and reveals true treatment effect. Treatment effect: compare post-treatment outcome between those who get the treatment and those who don’t.

Solution 1: Randomization Example 1: hormone replacement therapy (HRT) Recommended for middle-aged women to reduce menopausal symptoms.

Solution 1: Randomization Nurses Health Study (non-experimental survey of nurses): better health among the HRT users. Randomized trial: few benefits; serious side effects (see, e.g., Women’s Health Initiative [WHI], Hsia, et al., 2006).

Solution 1: Randomization Example 2: government-subsidized training programs. Provide a combination of classroom instruction and on-the-job training for groups of disadvantaged workers such as the long-term unemployed, drug addicts, and ex-offenders. Aim: increase employment and earning.

Solution 1: Randomization Non-experimental studies: trainees earn less than comparison groups (see, e.g., Ashenfelter, 1978; Ashenfelter and Card, 1985; Lalonde 1995). Evidence from randomized evaluations of training programs generate mostly positive effects (see, e.g., Lalonde, 1986; Orr, et al, 1996).

Solution 1: Randomization Problems of randomization: Randomly offers, but people don’t want to be part of the game High costs Small sample size

Solution 2: Difference-in-difference Panel data available

Solution 2: Difference-in-difference New problem: time trend Compare change in outcomes between treatment group and control group Impact is the difference in the change in outcome Impact = (Y t 1 -Y t 0 ) - (Y c 1 -Y c 0 )

Solution 2: Difference-in-difference PrePost

Solution 2: Difference-in-difference Effect of program using only pre- & post- data from T group (ignoring general time trend). PrePost

Solution 2: Difference-in-difference Effect of program using only T & C comparison from post-intervention (ignoring pre-existing differences between T & C groups). PrePost

Solution 2: Difference-in-difference Whatever happened to the control group over time is what would have happened to the treatment group in the absence of the program. PrePost Effect of program difference-in-difference (taking into account pre- existing differences between T & C and general time trend).

Solution 2: Difference-in-difference Example: Schooling and labor market consequences of school construction in Indonesia: evidence from an unusual policy experiment Esther Duflo, MIT American Economic Review, Sept 2001

Solution 2: Difference-in-difference School infrastructure Educational achievement Educational achievement? Salary level?

Solution 2: Difference-in-difference : The Indonesian government built 61,000 schools equivalent to one school per 500 children between 5 and 14 years old The enrollment rate increased from 69% to 85% between 1973 and 1978 The number of schools built in each region depended on the number of children out of school in those regions in 1972, before the start of the program.

Solution 2: Difference-in-difference 2 sources of variations in the intensity of the program for a given individual By region: simplify the intensity of the program: high or low By age: Young cohort of children who benefitted Older cohort of children who did not benefit

Solution 2: Difference-in-difference Intensity of the Building Program Age in 1974HighLow 2-6 (young cohort) (older cohort) Difference DD (0.089)

Solution 2: Difference-in-difference Fundamental assumption that trends (slopes) are the same in treatments and controls (sometimes true, sometimes not)

Time Treatment Outcome EstimatedAv erage Treatment Effect Average Treatment Effect Treatment Group Control Group

Solution 2: Difference-in-difference Need a minimum of three points in time (age of cohort in the example) to verify this and estimate treatment (two pre-intervention)

Time Treatment Outcome Treatment Group Control Group Average Treatment Effect First observation Second observation Third observation

Solution 2: Difference-in-difference Intensity of the Building Program Age in 1974HighLow Difference DD (0.098)

Solution 3: Matching Panel data NOT available Controls: non-participants with same characteristics as participants The matches are selected on the basis of similarities in observed characteristics

Solution 3: Matching Instead of aiming to ensure that the matched control for each participant has exactly the same value of X, same result can be achieved by matching on the probability of participation

Solution 3: Matching For each participant find a sample of non- participants that have similar propensity scores (prob. of treatment) Compare the outcome

Solution 3: Matching Common support

Solution 3: Matching Assumes no selection bias based on unobserved characteristics