VIII Evaluation Conference ‘Methodological Developments and Challenges in UK Policy Evaluation’ Daniel Fujiwara Senior Economist Cabinet Office & London School of Economics Warsaw, November 2012.

Policy evaluation in the UK  Analysts in government have a long history and tradition of policy evaluation.

The Magenta Book  Assessing whether the policy had a causal impact on the outcomes of interest.

The Green Book  Cost-benefit analysis is the key tool for policy assessment. CBA enters at the Appraisal stage  Appraise policy interventions in terms of their social costs and benefits.  Further supported through the Social Value Act (2012)

Challenges to policy evaluation ●Some constraints on use of impact evaluation findings: i.Are the impact and valuation estimates robust? ii.Have we acknowledged and included all possible impacts? iii.Are there some things that are considered separately – eg, merit goods, distribution (of income, health etc).

Challenges to policy evaluation: Impact measurement (1)  Identifying and measuring the counterfactual  Many public sector organisations moving to the Maryland Evaluation scale.  ‘What Works’ institutes like Washington State Institute for Public Policy.  Scale ranks how well counterfactuals have been measured in the analysis. Level DesignStatistical method 5Randomised trials Evaluations with well implemented random assignment of treatment to subjects in treatment and control groups. 4Quasi- Experiments Evaluations that use a naturally occurring event (that makes the treatment assignment as good as random) 3Matching techniques; Regression analysis Non-experimental evaluations where treatment and comparison groups are matched on observable characteristics 2Simple comparisons Studies with a treated and comparison group, but with no attempt made to control for differences among the groups. 1Pre- and post analysis Studies where no comparison group is used. Outcomes are measured pre and post-treatment.

Challenges to policy evaluation: Impact measurement (2)  Level 5 study designs are being increasingly used for public policy interventions.  RCTs used in tax policy, environment policy and labour market interventions.

Challenges to policy evaluation: Valuation  Supplementary Green Book guidance provides guidelines on how to value social impacts  Preference methods or Wellbeing Valuation ?  Different methods are increasingly allowing us to value and include a broad range of impacts

Challenges to policy evaluation: Summary  These developments and new guidelines for policy evaluation help us to understand the impacts of a policy better and to continuously improve the success of policy interventions.  Policy interventions are enhanced during this process through the use of evidence from behavioural science.