Designing a Random Assignment Social Experiment In the U.K.; The Employment Retention and Advancement Demonstration (ERA)

My Role in ERA A member of the 6-person team in the British Cabinet Office that: –Designed the ERA program –Designed the evaluation of ERA to determine how effective it is A member of the team evaluating ERA –I’m in charge of the cost-benefit analysis

My Talk Today How it was designed What the ERA program is The ERA evaluation and how experimental methods are being used to evaluated it Some early results Some lessons from designing ERA (if time permits)

Employment Retention and Advancement Demonstration (ERA) Design Work in Run as pilot program in in 6 sites Analysis conducted:

Why Was ERA Undertaken? To test a program that tries to keep low-wage workers employed after they find jobs and help them advance To promote the use of random assignment experiments in the U.K. Because it was expected the value of the information obtained about the program would exceed the cost of the obtaining the information (“the rational paradigm) –This requires that the information is actually used

Program Features Continued contact with Jobcentre Plus advisors (ASAs) after obtaining employment Retention bonus of £400 every 17 weeks if works full-time for 13 weeks Training bonus of £8 per hour of training Must contact ASA to receive bonus ASAs advise participants on job advancement

Unique Features of ERA Design of Program and Evaluation done simultaneously Design work done at British Cabinet Office Expectation that decision on national implementation of pilot would be based on evaluation findings Pilot run as large-scale random assignment experiment Focus is on what happens after a job is obtained

Evaluation Components Process or implementation study Impact analysis Cost study Cost-benefit analysis

Project Characteristics Long planning period –Absence of political pressure Developing the program and evaluation designs in tandem –Allowed for a good evaluation design Designing the project in the U.K. Cabinet Office –Permitted project team to focus on the project –Need to transfer project to DWP once designed Random assignment

How Does Random Assignment Work? Estimate the impact of a policy, a change in a program, or a intervention Provide evidence of whether the policy has led to (or caused) the change it was designed to - a causal link! The overall objective - to provide policymakers with evidence of whether their policy works

Measuring Impact Experiments measure the impact of policies/interventions in terms of their impact on outcomes; E.g. Does ERA increase earnings? The outcome measure is the earnings of the program group

Establishing Causality To establish that a policy or intervention has caused change to occur rather than some other factor –E.g. many factors will affect individuals earnings in addition to participation in ERA If we find that earnings have increased among the program group, how do we rule-out the influence of other factors? –E.g. most persons who entered ERA were not employed, but some would inevitably have found jobs without ERA. We do this by estimating what we call the ‘counterfactual’—what would have happened without the program

The Counterfactual Is what would have occurred in the absence of the policy or intervention –E.g. what would have happened to earnings over the same period of time for the same individuals had ERA not existed? This is unobservable or missing information We have to estimate the counterfactual - that is determine what would have happened in the absence of ERA

Estimating the Counterfactual Wide range of ways to do this These vary in complexity, rigor and the degree of control required by those planning the evaluation We will look at only one method: The simple two group random assignment experiment When feasible, random assignment is the best method for estimating a counterfactual

Random Assignment In theory the most rigorous way to assess the impact of a policy or intervention Provides unbiased estimates of intervention impacts How does it work? –You identify individuals or groups who are eligible for a new intervention or policy –You create two groups at random - intervention and control groups (essentially a computer flips a coin) –Intervention group receives the new service or intervention, control group does not.

Simple Randomised Experiment Intervention Eligible population R Intervention group Control group Outcome = O 1 Outcome = O 2 the counterfactual is simply ‘O 2 ’ policy impact is ‘O 1 ’- ‘O 2 ’

Random Assignment In ERA, half of those willing to be randomly assigned were randomly assigned to the program group and half to a control group Those assigned to the control group could not receive the services and financial incentives provided The baseline information collected for ERA at the point of random assignment indicates that the two groups are very similar in terms of all observable measures –randomisation worked

Random Assignment Continued After entering ERA, employment, earnings, and other outcomes of the program group are compared for several years to those of the control group during the same years Randomised evaluations of social programs used frequently in US (over 200 times) It is increasingly used in other countries, especially developing countries; but so-far it has been used far less often than in the U.S.

An Illustration At random assignment about 25% of one of the ERA program groups worked 24 months after random assignment, about 55% of the program group worked Was this increase due to ERA?

Illustration (continued) 24 months after random assignment, about 52% of the control group worked Thus, only about a 3 percentage point increase in employment (55% - 52%) is attributable to ERA

Advantages of Random Assignment As intervention and control groups were created randomly, they are statistically equivalent Equivalent in both what we can observed about them and what we cannot At follow-up, when me measure our outcome variables, the only difference between the two groups is the impact of the intervention In theory provides unambiguous results No need for complex statistics as with other methods - ease of interpretation Baseline information not essential, but helpful

Disadvantages of Random Assignment On its own, only provides a measure of average impact - policy makers may have other questions about the policy Can be expensive and complicated to implement Sometimes impractical to implement Possible lack of generalizability Can create political problems by denying services to controls (but if resources are limited some method must be used to deny some) In many cases, can take time for results to emerge Other evaluation designs may also be subject to these limitations-- depends on what is being evaluated

Conclusion Trade off -- rigour against difficulty in implementation Random assignment can be expensive - policy budget! Experimental methods require lots of data Only answers certain types of questions Still, when it is feasible, random assignment provides the most accurate estimates of impacts

Some Early Findings (* = statistically significant impact) NDDPWTC Program Group Control Group ImpactProgram Group Control Group Impact Ever worked Year 165.3%59.7%5.7*97.6%95.9%1.7* Year 267.5% *95.8%94.6%1.2 Months Worked Full time * * Part time * Earnings (£) Year 13,6122,764849*8,2557, Year 24,7814,108673*8,9628,458503*

Lessons from the ERA Design Work Developing ownership among those fielding and running a program important Randomized experiments are feasible, but circumstances must warrant their use Use of multi-disciplinary teams should be encouraged Designing programs and evaluations in tandem should be done whenever possible