1. Randomisation Bias and Post-Randomisation Selection Bias in RCTs: Barbara Sianesi Institute for Fiscal Studies September 14, 2006 The role of non-experimental methods in the ERA demonstration Randomised Controlled Trials in the Social Sciences: Challenges and Prospects The University of York

2. Talk Outline RCTs are the gold standard in evaluation BUT not immune from limitations 1. parameter retrieved 2. outcomes that can be looked at  Judicious combination with non-experimental methods can enhance (under suitable assumptions!) what can be learnt from a RCT Excellent example to illustrate this: ERA

3. What is ERA a new package of support financial incentives (job retention, training) to assist ND25+ and NDLP customers obtain, retain and advance in work evaluated via RA (~14,000 in 6 districts)

4. Issue #1 Some eligibles in the ERA Districts did not reach the decision stage or refused to take part in research scheme Experimental contrast  unbiased estimate of ERA impact for those who have reached the RA stage & have agreed to participate.

6. ERA impact on ERA participants

8. Staff discretion. choice of marketing strategy Customer finding job unlikely finding job likely but no desire to stay in touch with JCP antipathy to government, systems of support, mandatory programmes resistant to change, …

10. Staff Some may not think customer would benefit / be interested. Some non-ERA advisers may think customers close to job entry may provide quick win.

11. ?

12. ? Non-participants

13. Why do non-participants pose a potential issue? Would have liked experimental estimate of impact of ERA for the full eligible population in the ERA Districts. Benchmark: pilot/control area evaluation In ideal scenario staff would offer ERA to any eligible (no discretion) all eligibles would participate (no need for consent)

14. Why do non-participants pose a potential issue? Would have liked experimental estimate of impact of ERA for the full eligible population in the ERA Districts. But: ERA tested only on a subset of ERA eligibles in ERA Districts – the participants.

15. How to view this a)Interested in impact on eligibles but only get impact on participants. “Randomization Bias occurs when random assignment causes the type of persons participating in a program to differ from the type that would participate in the program as it normally operates.” (Heckman and Smith, 95, p.99)

16. How to view this b)Focus on what the RCT consistently estimates (impact on participants) and interested in how well it generalizes to wider population (impact on eligibles). Issue of External Validity. How representative of the full eligible population?

17. When are non-participants a problem? E(impact | eligibles) What we want

18. When are non-participants a problem? E(impact | eligibles) = E(impact | eligible partic.) what we want what we get

19. When are non-participants a problem? E(impact | eligibles) = E(impact | eligible partic.) Prob(eligible partic.) what we want observed E(impact | eligibles) = E(impact | eligible partic.) Prob(eligible partic.) what we get

20. When are non-participants a problem? E(impact | eligibles) = E(impact | eligible partic.) Prob(eligible partic.) + E(impact | eligible non-part.) what we want what we get observed ?

21. When are non-participants a problem? E(impact | eligibles) = E(impact | eligible partic.) Prob(eligible partic.) + E(impact | eligible non-part.) Prob(eligible non-part.) what we want what we get ? observed

22. When are non-participants a problem?  eligibles –  partic = (  nonpartic –  partic )  Prob nonpart bias=  p It depends on: 1.relative size of eligible non-participant group 2.whether it is very different from RA group

23. I.Descriptive Analysis I.Extent of non-participation II.How different a.observable characteristics b.outcomes (non-participants vs. controls) II.Non-Experimental Analysis

24. Issue #2 Does ERA enhance hourly wages (productivity) wage growth (advancement) employment duration (retention) Cannot use experimental contrast due to post-randomisation selection bias into employment. ?

29. To conclude … “Since experiments can answer only a subset of the questions of interest to the evaluator, it remains important to build up the stock of basic social science knowledge required to successfully utilize non-experimental methods, both in themselves and as tools for more extensive analyses of experimental data.” Heckman and Smith (1995, p.95)