‘What Works’ The role of evidence based policy and research in Britain’s welfare to work policies Professor Dan Finn University of Portsmouth

“.. Social science should be at the heart of policy making. We need a revolution in the relationship between government and the social research community – we need social scientists to help determine what works and why, and what type of policy initiatives are likely to be most effective” (David Blunkett, Secretary of State for Education and Employment, 2000)

Context Relationship between applied research and British social policy develops from late 1890s Relationship between applied research and British social policy develops from late 1890s Took considerable time to establish that research was of use in both formulating social policies and administering them Took considerable time to establish that research was of use in both formulating social policies and administering them Research into, and evaluation, of government policies now systematic & routine – interest in impacts, efficiency and value for money (post New Public Management) Research into, and evaluation, of government policies now systematic & routine – interest in impacts, efficiency and value for money (post New Public Management) Increase in organisations producing ‘evidence’ for Government (growth in size and diversity of research community) Increase in organisations producing ‘evidence’ for Government (growth in size and diversity of research community)

New Labour and Evidence Based Policy Making (EBPM) New Labour regard EBPM as central to the modernisation of government New Labour regard EBPM as central to the modernisation of government Multiple influences on policy but in context of increase in public expenditure and major social and economic development programmes, ambition that evidence should form a key part of the reform process Multiple influences on policy but in context of increase in public expenditure and major social and economic development programmes, ambition that evidence should form a key part of the reform process

Review of evidence use by Government Little of the research commissioned by Government Departments or other academic research appears to be used by policy makers Little of the research commissioned by Government Departments or other academic research appears to be used by policy makers Prime Minister’s Cabinet Office develops strategy: Prime Minister’s Cabinet Office develops strategy:  Policy making must be soundly based on evidence of what works  Improve the accessibility of the evidence available to policy makers  Improve departments’ capacity to make best use of evidence (training officials in how to interpret, use and apply evidence)

EBP – Government Strategy Cabinet Office – ‘Policy Hub’ Cabinet Office – ‘Policy Hub’ Evidence Based Policy Fund – strengthening links between universities, research institutes and government (e.g., EPPI – ‘Evidence for Policy and Practice Information and Coordination Centre’) Evidence Based Policy Fund – strengthening links between universities, research institutes and government (e.g., EPPI – ‘Evidence for Policy and Practice Information and Coordination Centre’) Economic and Social Research Council programme (see ‘Evidence Network’) Economic and Social Research Council programme (see ‘Evidence Network’) ‘Campbell Collaboration’ – new collaboration for systematic research synthesis in the social science and policy fields, modelled on the ‘Cochrane Collaboration’. ‘Campbell Collaboration’ – new collaboration for systematic research synthesis in the social science and policy fields, modelled on the ‘Cochrane Collaboration’. Government Social Research Office oversees use of social research – purpose “to provide government with objective, reliable, relevant and timely research to inform policy making and delivery” Government Social Research Office oversees use of social research – purpose “to provide government with objective, reliable, relevant and timely research to inform policy making and delivery”

Evidence of ‘what works’ Rules of evidence differ greatly between research cultures (healthcare, education, criminal justice, social care, welfare, housing, transport and urban renewal) Rules of evidence differ greatly between research cultures (healthcare, education, criminal justice, social care, welfare, housing, transport and urban renewal) Balance between quantitative and qualitative research in assessing what works and what use to make of secondary research Balance between quantitative and qualitative research in assessing what works and what use to make of secondary research Systematic evidence reviews replacing traditional literature reviews Systematic evidence reviews replacing traditional literature reviews Primacy given to ‘hard’ quantitative data, esp. randomised control trials, meta analysis. Primacy given to ‘hard’ quantitative data, esp. randomised control trials, meta analysis.

UK policy now more informed by evidence (‘Better Policy Making’, Cab. Office, 2001) In UK, evidence based government/policy making well established In UK, evidence based government/policy making well established Reviews of existing & proposed policies (international policy learning) Reviews of existing & proposed policies (international policy learning) Commissioning new research/evaluating policy design and impacts by commissioning/undertaking research. Typically: Commissioning new research/evaluating policy design and impacts by commissioning/undertaking research. Typically:  Expert evidence and systematic evidence reviews (internet)  Pilots and case studies  Rapid/Interim evidence assessments  Some Random Control Trials but more typically statistical/econometric ‘net impact’ studies Emphasis also on learning from practice – gathering evidence from delivery agencies and frontline workers Emphasis also on learning from practice – gathering evidence from delivery agencies and frontline workers

Evidence Based Policy Making and Welfare to Work Reforms Department for Work and Pensions (about 80 social researchers work with other analysts - economists, statisticians): Department for Work and Pensions (about 80 social researchers work with other analysts - economists, statisticians):   internal research capacity   contracted out research programmes (about £18m, 2007) DWP commissions research using a variety of methods – large scale surveys, in-depth interviews, experiments, ethnographic methods and analysis of administrative data DWP commissions research using a variety of methods – large scale surveys, in-depth interviews, experiments, ethnographic methods and analysis of administrative data Not directly engaged with social theory - the approach tends to be inductive rather deductive – work on building up ideas, theories, concepts from the data rather than using data to verify theories. Not directly engaged with social theory - the approach tends to be inductive rather deductive – work on building up ideas, theories, concepts from the data rather than using data to verify theories. Development of Welfare to Work policies has been informed by continuous process of research and evaluation – major influence of US welfare experiments and evaluation culture Development of Welfare to Work policies has been informed by continuous process of research and evaluation – major influence of US welfare experiments and evaluation culture

Pathways to Work New way of engaging with people on disability benefits: New way of engaging with people on disability benefits:  Compulsory Work Focused Interviews with Case Managers  ‘Choices’ - innovative package of employment and ‘condition management’ measures Methodology Methodology  Review of Existing Evidence  Integrated Evaluation with Policy Development and Implementation  Qualitative Interviews (clients, officials, case managers and employers)  Administrative Data and Surveys – measuring impacts of pilot areas with matched statistical control groups  Local Context Analysis (Qualitative and Quantitative)  Economic Appraisal (Cost Benefit Analysis)

UK Employment, Retention & Advancement Demonstration Project What is the most effective and efficient way of: What is the most effective and efficient way of:  Retaining people in work  Advancing low paid people in the labour market Methodology Methodology  Review of Existing Evidence  Integrated Evaluation with Policy Development and Implementation  Qualitative Interviews (clients, officials, case managers and employers)  Experimental Method (Randomised Control Trial)  Administrative Data and Surveys  Local Context Analysis (Qualitative and Quantitative)  Economic Appraisal (Cost Benefit Analysis)

Realities Limitations of ‘evidence’ produced from a Government managed research process (not all research is of sufficient quality) Limitations of ‘evidence’ produced from a Government managed research process (not all research is of sufficient quality) Time scale of evaluations (esp. randomised trials) – often overtaken by events Time scale of evaluations (esp. randomised trials) – often overtaken by events Absence of impact evidence is not necessarily evidence of absence of effect Absence of impact evidence is not necessarily evidence of absence of effect Evidence requires analysis and interpretation! Evidence requires analysis and interpretation!

Conclusion Reality of policy making – contingent and unpredictable, evidence just one factor that policy makers take into account. Reality of policy making – contingent and unpredictable, evidence just one factor that policy makers take into account. Other (maybe more important) influences in policy making – rresources; lobbyists and pressure groups; pragmatism; expediency and contingencies. Other (maybe more important) influences in policy making – rresources; lobbyists and pressure groups; pragmatism; expediency and contingencies. Should ask also why politicians are showing an increased interest in evaluation and EBP - danger that legitimising functions are being used in a purely symbolic (political) fashion. Should ask also why politicians are showing an increased interest in evaluation and EBP - danger that legitimising functions are being used in a purely symbolic (political) fashion.

Conclusion Evidence Based Policy Making is a contested concept – closely tied to the politics of a strong central state. Evidence Based Policy Making is a contested concept – closely tied to the politics of a strong central state. Best assessment is there is more ‘evidence informed policy making’ Best assessment is there is more ‘evidence informed policy making’ UK political realities – ‘speaking truth to power’, academic freedom, and importance of critical social research UK political realities – ‘speaking truth to power’, academic freedom, and importance of critical social research Role of independent foundations – Joseph Rowntree Trust, Leverhulme Trust, Cadbory Trusts, etc. Role of independent foundations – Joseph Rowntree Trust, Leverhulme Trust, Cadbory Trusts, etc.

Some useful sources Britain Britain  Policy Hub and ‘Magenta Book: Guidance Notes on Policy Evaluation’ at  Evidence Based Policy Network (  Evidence for Policy and Practice Information and Coordination Centre (http//eppi.ioe.ac.uk)  Social Care Institute for Excellence (  Evidence & Policy: A journal of research, debate and practice (  DWP Research (  Office for National Statistics ( International: International:  Cochrane Collaboration (medical:  Campbell Collaboration (social science and public policy: