1. Making the most of RCTs: reassessing ERA (Sianesi & Lise) The Employment, Retention and Advancement demonstration ( ) –first large-scale RCT in social policy in UK (over 16,000 people) –has been evaluated experimentally (Hendra et al., 2011) Aim: maximise the value of the ERA experiment –Improve the design of non-experimental evaluations –Improve way such evaluations add to the knowledge base “Gold standard” randomisation is still rare –costly, impractical or politically infeasible → Project 1A –lack of external validity and ex ante analysis → Project 1B
3. Control functions in policy evaluation (Blundell, Costa Dias, Rosen, Chesher, Kitagawa) Choice among alternative evaluation methods is driven by three concerns –Question to be answered –Type and quality of data available –Assignment rule (the mechanism that allocates individuals to the programme) This project focuses on the last Idea –The ideal assignment rule comes from an RCT –But if we know something about the assignment rule, then the control function approach allows us to account for/correct for the endogenous selection into treatment
3. The control function approach: example Interested in the impact of university education on subsequent labour market earnings (the “returns to university education”) Unobservable determinants of earnings, e.g. underlying ability, will be correlated with the decision to attend university, so a simple regression will provide a biased view of the returns to university By modelling key features of the decision to attend university – the “assignment rule” to university – the control function approach can correctly recover the average return to university among those who took up a place
3. The control function approach: example (continued) These key features will ideally be factors that determine assignment to university but do not determine directly final earnings in the labour market –Family socio-economic background, level of university fees, distance to university, availability of university places (if rationed) If can write down an equation modelling the way these factors determine university attendance, we can construct an index (or ‘control function’) that can then be included in the earnings regression along with the indicator for attending university. –Extension of the ‘Heckman’ selection approach that controls for the endogenous selection into treatment
3. The control function approach: our research Research questions: –Under what circumstances does the use of a control function compare favourably to matching and instrumental variables? What are the key trade-offs? –How does a control function approach map into a behavioural model? What can a control function approach tell us about structural parameters of interest? –Can we weaken the control function approach by incorporating partial knowledge of the assignment rule to produce bounds? –Will study various education and labour market policies
4. Dynamic behavioural models for policy evaluation (Low, Dias, Shaw, Meghir, Pistaferri) Classical ex post empirical evaluation methods often fail to explain the nature of the estimated effect –Cannot disentangle impact of programme on incentives from how incentives affect individual decisions –Cannot account for dynamic responses (anticipation or changes now affect decisions in future) –Studies often rely on different sets of behavioural assumptions Difficult to understand, as not explicitly stated Complicates task of synthetising information from different studies –Cannot be used for counterfactual analysis Results are specific to the policy, time and environment
5. Social networks and program evaluation: example of Progresa Progresa is village-level intervention in rural Mexico. Previous research has shown that: –1 in 5 households are “isolated” (none of their extended family resides within the same village) –On some margins, only non-isolated households responded to Progresa Was it because poor families needed assistance and encouragement to join the programme? Or was it because of nature of Progresa intervention, part of which was to encourage teenage girls to stay in school?
5. Social networks and program evaluation Substantive research questions –How are the benefits of program interventions dissipated within communities once social networks are accounted for? –How do such spillovers (from beneficiary to non-beneficiary households) affect the cost-benefit analysis of programs, and how we think about targeting? –Why and how are social networks formed (can investigate this by studying particular interventions) Methodological research questions –How best to measuring whether and how households are socially tied (blood ties, resource flows)?