Matching Estimators Methods of Economic Investigation Lecture 11

Last Time  General Theme: If you don’t have an experiment, how do you get a ‘control group’  Difference in Differences How it works: compare before-after between two comparable entities Assumptions: Fixed differences over time Tests to improve credibility of assumption  Pre-treatment trends  Ashenfelter Dip

Today’s Class  Another way to get a control group: Matching Assumptions for identification Specific form of matching called “propensity score matching” Is it better than just a plain old regression?

The Counterfactual Framework  Counterfactual: what would have happened to the treated subjects, had they not received treatment?  Idea: individuals selected into treatment and nontreatment groups have potential outcomes in both states: the one in which they are observed the one in which they are not observed.

Reminder of Terms  For the treated group, we have observed mean outcome under the condition of treatment E(Y 1 |T=1) and unobserved mean outcome under the condition of nontreatment E(Y 0 |T=1).  For the control group we have both observed mean E(Y 0 |T=0) and unobserved mean E(Y 1 |T=0)

What is “matching”?  Pairing treatment and comparison units that are similar in terms of observable characteristics  Can do this in regressions (with covariates) or prior to regression to define your treatment and control samples

Matching Assumption  Conditioning on observables (X) we can take assignment to treatment ‘as if’ random, i.e.  What is the implicit statement: unobservables (stuff not in X) plays no role in treatment assignment (T)

A matched estimator  E(Y 1 – Y 0 | T=1) = E[Y 1 | X, T=1] – E[Y 0 | X, T=0] - E[Y 0 | X, T=1] – E[Y 0 | X, T=0] Key idea: all selection occurs only through observed X Matched treatment effect Assumed to be zero

Just do a regression…  Regression are flexible if you only put in a “main effect” the regression will estimate a purely linear specification Interactions and fixed effects allow different slopes and intercepts for any combination of variables Can include quadratic and higher order polynomial terms if necessary But fundamentally specify additively separable terms

Sometimes regression not feasible…  The issue is largely related to dimentionality  Each time you add an observable characteristics, you partition your data into bins.  Imagine all variables are zero-one variables Then if you have k X’s, you have 2 k potential different values Need enough observations in each value to estimate that precisely

Reducing the Dimensionality  Use of propensity score: Probability of receiving treatment, conditional on covariates  Key assumption: if and defining  If this is true, can interpret estimate of differences in outcomes conditional on X as causal effect

Why not control for X  Matching is flexible in a different way Avoid specifying a particular for the outcome equation, decision process or unobservable term Just need the “right” observables Flexible in the form of how X’s affect treatment probability but inflexible in how treatment probability affects outcome

Participation decision  Remember our 3 groups: Always takers: take the treatment if offered AND take the treatment if not offered We observe them if T=0 but R=1 Never takers: don’t take the treatment if not offered AND don’t take it even if it is offered We observe them if T=1 but R=0 Compliers: just do what they’re assigned to do T=1 & R=1 OR T=0 & R=0

Conditions for Matching to Work Take 1-sided non-compliance for ease…if not offered, can’t take it, but some people don’t take it even if offered Error term for never takers Error term for compliers If it’s zero  Perfect compliance: so conditioning on X replicates experimental setting On avg, conditional on X unobservable are the same

Common Support  Can only exist if there is a region of “common support” People with the same X values are in both the treatment and the control groups Let S be the set of all observables X, then 0<Pr(T=1 | X)<0 for some S * subset of S  Intuition: Someone in control close enough to match to treatment unit OR enough overlap in the distribution of treated and untreated individuals

Lots of common support Between red and blue line is area of common support

Not so much common support

Trimming  Define Min and Max values of X for region of overlap—drop all units not in that region  Remove Regions which do not have strictly positive propensity score in both treatment and control distributions (Petra and Todd, 2005)  Both are quite similar when used in practice but if missing sections in middle of distribution can use the second option

How do we match on p(X)  Taken literally, should match on exactly p(X i )  In practice hard to do so strategy is to match treated units to comparison units whose p-scores are sufficiently close to consider  Issues: How many times can 1 unit be a “match” How many to match to treatment unit How to “match” if using more than 1 control unit per treatment unit

Replacement  Issue: once control group person Z is a match for individual A, can she also be a match for individual B  Trade-off between bias and precision: With replacement minimizes the propensity score distance between the matched and the comparison unit Without replacement

Are we doing a one-to-one match?  If 1-to-1 match: units closely related but may not be very precise estimates  More you include in match, the more the p-score of the control group will differ from the treatment group  Trade-off between bias and precision Typically use 1-to-many match because 1-to-1 is extremely data intensive if X is multi- dimensional

Different matching algorithms-1  Can use nearest neighbor which chooses m closest comparison units implicitly weights these all the same Get fixed m but may end up with different pscores  Can use ‘caliper’—radius around a point Again implicitly weights these the same Fixed difference in p-scores, but may not be many units in radius  Stratify Break sample up into intervals Estimate treatment effect separately in each region

Different Matching Algorithms-2  Can also use some type of distribution: Kernel estimator puts some type of distribution (e.g. normal) around the each treatment unit and weights closer control units more and farther control units less Explicit weighting function can be used if you have some knowledge of how related units of certain distances are to each other

How close is close enough?  No “right” answer in these choices—will depend heavily on sample issues How deep is the common support (i.e. are there lots of people in both control and treatment group at all the p-score values  Should all be the same asymptotically but in finite samples (which is everything) may differ

Tradeoffs in different methods Source: Caliendo and Kopeinig, 2005

How to estimate a p-score  Typically use a logit Specific, useful functional form for estimating “discrete choice” models You haven’t learned these yet but you will  For now, think of running a regular OLS regression where the outcome is 1 if you got the treatment and zero if you didn’t  Take the E[T | X] and that’s your propensity score

The Treatment Effect  CIA holds and sufficient region of of common support  Difference in outcome between treated individual i and weighted comparison group J, with weight generated by the p-score distribution in the common support region N is the treatment group and |N| is the size of the treatment group J is comparison group with |J| is the number of comparison group units matched to i

1-to-n Match  Nearest neighbor matching  Caliper matching  Nonparametric/kernel matching Run Regression: Dependent variable: T=1, if participate; T = 0, otherwise. Choose appropriate conditioning variables, X Obtain propensity score: predicted probability (p) General Procedure Multivariate analysis based on new sample 1-to-1 match  estimate difference in outcomes for each pair  Take average difference as treatment effect

Standard Errors  Problem: Estimated variance of treatment effect should include additional variance from estimating p Typically people “bootstrap” which is a non- parametric form of estimating your coefficients over and over until you get a distribution of those coefficients—use the variance from that Will do this in a few weeks

Some concerns about Matching  Data intensive in propensity score estimation May reduce dimensionality of treatment effect estimation but still need enough of a sample to estimate propensity score over common support Need LOTS of X’s for this to be believable  Inflexible in how p-score is related to treatment Worry about heterogeneity Bias terms much more difficult to sign (non- linear p-score bias)

Matching + Diff-in-Diff  Worry that unobservables causing selection because matching on X not sufficient  Can combine this with difference and difference estimates Take control group J for each individual i Estimate difference before treatment If the groups are truly ‘as if’ random should be zero If it’s not zero: can assume fixed differences over time and take before after difference in treatment and control groups

Next Time  Comparing Non-Experimental Methods to the experiments they are trying to replicate  Goal: See how well these techniques work to get the estimated experimental effect