1 Government Social Research Unit Randomised Controlled Trials Conference University of York, September 2006 Why Governments Need RCTs Philip Davies PhD Government Social Research Unit HM Treasury London SW1A 2HQ

2 Answer Because RCTs can help get policy initiatives right more often than wrong

3 Caveats Governments have got to want to get things right ‘Getting things right’ is more than achieving an effect size Governments do things other than achieving impacts RCTs are not a panacea RCTs are only as good as their design and implementation

4 Governments Have Got to Want to Get Things Right “Social science should be at the heart of policy making. We need a revolution in the relationship between government and the social research community - we need social scientists to help determine what works and why, and what type of policy initiatives are likely to be most effective …” (David Blunkett, 2000)

5 Governments Have Got to Want to Get Things Right Government policy must be: Evidence-based Properly evaluated Based on best practice

6 Governments Have Got to Want to Get Things Right Policy making must be soundly based on evidence of what works Improve departments’ capacity to make best use of evidence Improve the accessibility of the evidence available to policy makers

7 Governments Have Got to Want to Get Things Right Professional Skills for Government (2005) Analysis and Use of Evidence Anticipate and secure appropriate evidence Test for deliverability of policy/practice – and evaluate Use evidence to challenge decision making Identify ways to improve policy/practice Champion a variety of tools for collecting/using evidence Ensure use of evidence is consistent with wider government requirements Work in partnership with a wide range of experts/analysts

8 What ‘Getting It Right’ Means “…a society that would use social science methods and evaluation techniques to vigorously try out possible solutions to recurrent problems and would make hard- headed, multidimensional evaluations of outcomes, and when the evaluation of one reform showed it to have been ineffective or harmful, would move on and try other alternatives” The ‘Experimenting Society’ (Donald T. Campbell)

9 What ‘Getting It Right’ Means “A social experiment benefits society by providing better information on which to base public policy. Such information can improve policy in one of two ways: it can lead policymakers to adopt a program or policy that is found to have net social benefits, or it can lead to the termination of an existing program that is found to have a net social cost” (Orr, L., 1999).

10 Why RCTs? (Recap) Randomised Controlled Trials: Provide most precise estimates of the likely effects of policy interventions. Against a strong counterfactual – i.e. the net effect/impact Reduce/account for various biases. Establish cause and effect of policy interventions and outcomes. Raise the quality of evidence to support policy making.

11 So, Why Don’t Governments Use RCTS More Often? RCTs not on most policy makers’ radar RCTs not on many government researchers’ radar RCTS not on many academic researchers’ radar For some (many?) RCTs are anathema:  Don’t like ‘random’ (too ‘chancey’)  Don’t like ‘control’ (no ‘wiggle room’)  Don’t like ‘trials’ (sounds inhumane / 1940s Germany) There are things other than ‘impact’ There are things other than ‘evidence’

12 Things Other Than Impact Evidence Research Evidence Impact Evidence Implementation Evidence Descriptive Analytical Evidence Economic and Econometric Evidence Ethical Evidence Experimental Quasi-Experimental Surveys Admin Data Comparative Qualitative Cost-Benefit Cost-Effectiveness Cost-Utility Experimental Qualitative Theories of Change Social Ethics Public Consultation Statistical Modelling Multivariate Analysis

13 Things Other Than Evidence “ Research of any kind is seldom the determining factor in shaping public policy. Experimentation is no exception to this rule ” (Orr, L., 1999)

14 Things Other Than Evidence Evidence Experience & Expertise Judgement Resources Values and Policy Context Habits & Tradition Lobbyists & Pressure Groups Pragmatics & Contingencies

15 Policy Makers’ Evidence Colloquial (Contextual) Anything that seems reasonable Policy relevant Timely Clear Message Researchers’ Evidence ‘Scientific’ (Context free) Proven empirically Theoretically driven As long as it takes Caveats and qualifications Different Notions of Evidence

16 Where Do Policy Makers Go For Evidence?

17 But Not All Is Lost Employment Demonstration Project (ERA) Job Retraining and Rehabilitation Project (JRRP) Restart evaluation (1990) BA Visiting Officer Pilot Evaluation of Employment Zones New Deal 25+ Pilots Evaluation of the New Deal for Young People Intensive Gateway Evaluation of the Restorative Justice programme Examples of Recent UK Government RCTs

18 RCTs Are Only As Good As Their Design and Implementation Common Design and Implementation Faults: Unclear policy objectives, populations, outcomes Insufficient time  (lack of strategic policy making) Test condition often pre-determined Sample sizes too small – no statistical power Problems of randomisation, concealment, cross-over High attrition (especially in control groups) Intention to Treat analysis not used Limited capacity (in the UK) to do high quality RCTs Poor counterfactual too often the result

19 RCTs Are Only As Good As Their Design and Implementation These faults can usually be overcome! Better strategic policy making Better planning and protocol development Involvement of researchers, policy makers, front-line staff High quality piloting / pre-fieldwork testing High quality research/project management Careful real time data management Use of experienced technical experts throughout Policy Trials Service Professional development of researchers and policy makers

20 Take Home Message RCTs have enormous potential for governments RCTs can help governments get things right more often RCTs are currently used considerably to UK Govnt. But there are still non-believers And those who remain to be convinced RCTs, like other types of research, must compete with other notions of evidence And with factors other than evidence There is a need to improve the capacity to do high quality RCTs in the UK

21 Contact Government Social Research Unit HM Treasury 1 Horse Guards Road London SW1A HQ England Tel: +44 (0)