Evaluation of Labour Market Policies: The Use of Data-Driven Analyses in Ireland Elish Kelly Economic and Social Research Institute National Development Agency: 4 th International Evaluation Conference (26-27 September 2013)

Outline Overview: - Why conduct evaluations? - Barriers to conducting effective evaluations - Most common forms of labour market evaluations How to evaluate a programme’s effectiveness, and issues that need to be considered during this process Practical example: An evaluation of Ireland’s activation strategy – the National Employment Action Plan (NEAP) Conclusions: Implications for labour market policy in Ireland from the findings of the evaluation

Why is Evaluation Necessary? 1.It assesses the extent to which policy initiatives are achieving their expected targets and objectives Drawing from this, the evaluator, and consequently policy-makers, will identify the nature of any shortfalls in either programme delivery or the stated objectives. 2.Effective use of public resources, which is particularly important in the current economic environment. 3.Overall, evaluations help to ensure that policy is evidence-based and that ineffective programmes are modified or closed.

Barriers to Effective Evaluations Lack of an evaluation culture among policy-makers: Why? - Policy-makers may view evaluation as a threat and actively seek a less rigorous form of assessment - Lack of complex evaluation expertise and the competencies required to use large administrative datasets. Lack of Independence: the organisation being evaluated has the power to set the terms of reference and is involved in choosing the evaluating body. Often little consideration is given to programme evaluation at the programme design and implementation stages (consequently, lack of a viable control group to assess the counterfactual). Data constraints: Lack of available and “linkable” administrative datasets make proper evaluation difficult.

Most Common Forms of Labour Market Evaluations Generally, labour economists tend to focus on impact evaluation i.e., is the programme achieving its desired impacts e.g. training programme for the unemployed leading to employment? Process evaluation - i.e., is the programme being delivered as intended?, is less common. However, in practice most impact evaluations will also consider the efficiency of programme delivery and implementation. Overall, the bulk of impact evaluations focus on labour market programmes that are designed to improve outcomes related to employment, earnings and labour market participation.

How do we Evaluate a Labour Market Programme’s Impact? Not straightforward, but it is possible if the correct steps are taken at the i) design stage of a programme, ii) its implementation and iii) the utilisation of the correct mechanisms (e.g. data and methodologies) at the evaluation stage of the process. With labour market programme evaluations, we want to know what would happen to individuals had the programme not been in place (e.g. unemployed person did not receive training) i.e. we attempt to measure the counterfactual. There are various methods used for estimating the counterfactual, however, they all generally rely on measuring the difference in outcomes between people participating in the programme (the treatment group) and those eligible for participation but did not (the control group).

How do we ensure we have a Counterfactual to Evaluate a Programme’s Effectiveness? In other words, how do we ensure that evaluators have a control group? This needs to be considered at the programme design stage and built in during the implementation stage. One way is to pilot the programme i.e roll out the programme to different areas at difference times. - Evaluators then need to have access to administrative data on the targeted population (e.g. unemployment register data). - At the same time, records need to be kept of unsuccessful applicants to the programme in instances where the demand for programme places exceeds supply (these individuals are the counterfactual).

Issues: The Selection Problem Comparison of a treatment and control group is not straightforward: - Substantial differences may exist between the two groups that must be factored out as assignment to either is rarely random - Such differences can also arise as a consequence of ineffective control group construction. Non-random selection refers to the possibility that: i) programme administrators engaged in “picking winners” in order to ensure the programmes success, or ii) more capable individuals are more likely to put themselves forward for intervention. Failure to account for the selection problem will result in a biased estimate of the programme’s effectiveness.

Other Issues to Consider Dynamic Bias: How do we ensure that control group members will not have been activated at some point in the future (or are expecting to be activated and behave accordingly)? Unobserved Heterogeneity: Are there unobserved differences between the control and treatment group (such as ability) that have the potential to bias our estimates? Some of these issues need to be considered at the design stage of a programme and its subsequent roll-out, while others have to be addressed at the evaluation stage.

Activation in Ireland: An Evaluation of thr National Employment Action Plan (NEAP) Commissioned by the Department of Social Protection Research conducted by the Economic and Social Research Institute (2011)

Overview of Ireland’s Activation Strategy The NEAP is Ireland’s principal tool for activating unemployed individuals back into the labour market. The NEAP is currently being revamped but at the time of the evaluation the activation strategy operated as follows: 1. Individuals registering for unemployment benefit were “automatically” referred by the Department of Social Protection (DSP) to FÁS, formally Ireland’s national employment and training authority, for an activation interview after 3 months on the UE benefit system. 2. During the activation interview, clients could have been provided with Job Search Assistance (JSA) and/or referred to employment or training opportunities. Individuals with previous exposure to the NEAP – i.e. those with a previous history of unemployment, are excluded and will not be referred to FÁS for a second time. At the time of the evaluation, the NEAP was quite distinct in an international sense in that it was characterised by an almost complete absence of monitoring and sanctions, and it did not apply the ‘mutual obligation’ principal.

The NEAP Evaluation Objectives The study examined the effectiveness of two key components of the NEAP Strategy using data for the period 2006 to 2008: 1. The impact of the NEAP referral and interview process (i.e. JSA) on NEAP programme participants (the treatment group) likelihood of exiting unemployment to employment relative to non-NEAP participants (the control group) 2. To assess the extent to which individuals in receipt of both a referral interview and training had enhanced employment prospects relative to those in receipt of an interview only (i.e. assess the impact of training). Today’s presentation will focus on how we went about evaluating the effectiveness of the referral and interview component of the NEAP (1).

First Issue Encountered: No Control Group? Selection under the NEAP is automated and universal: if all claimants are automatically sent for interview at 3 months of their claim, how can we construct a counterfactual? - Remember, the counterfactual assesses what happens to individuals in the absence of the programme. The only eligible people not exposed to the programme are those already in employment by the 3 month time point. This problem illustrates that evaluation of the NEAP was not considered in the programme’s design or implementation stages.

What Did We Do? Only option was to utilise the fact that individuals with previous exposure to NEAP cannot access it again (as an aside, this could be viewed as counter-intuitive rule as those most in need of support are excluded from receiving assistance again). We took an initial control group of individuals who had previous exposure to NEAP more than two years prior to the study whose contact was limited to a FAS interview. - Given the time lapse, and changing macroeconomic conditions, any advice received by the control group should have declined in relevance; therefore, allowing for some assessment of the impact of the JSA component of the NEAP. - However, even if the above were true we were still left with a selection problem as prior to the study all of the control group would have had a previous unemployment spell of at least 13 weeks, whereas none of the treatment group did. This difference cannot be eradicated by matching, and consequently our estimates of the programme’s effectiveness were unlikely to be free of bias.

Next Step: Construction of the Evaluation Dataset Weekly Population of Live Register Claimants Weekly Population of Live Register Claimant Closure Files Profiling Questionnaire InformationforClaimant Population Issued June to September 2006 Live Register Claimant Population (September 2006– June 2008) Dataset for NEAP Evaluation FAS Events Histories Constructed using a combination of i) administrative data from the Live Register, ii) survey data from the DSP’s Profiling Database and iii) FAS’s client history administrative data

After Dataset Construction New Control Group Found…. On linking the data, we found that approximately 25% of new claimants had not been referred by the DSP to FAS after 3 months unemployment duration, despite these individuals having no previous exposure to the NEAP. Before using this group as a counterfactual, we needed to establish what was going on: i) were we missing something in terms of the referral process? ii) if not, what factors drove the omission of this group of individuals, and are they random? A list containing the PPS numbers of our potential new control group was sent to DSP for validation.

Validation Checks The DSP confirmed these individuals had fallen through the net. No concrete explanation found: most likely that these individuals were not referred when the number of referrals in DSP offices exceeded slots in local FAS offices and they were subsequently overlooked when slots became available. Even before we had begun our evaluation of the NEAP, we had uncovered two major problems with the programme’s processes: i) 25 % of potential claimants excluded and ii) a further 25% missed. This is a clear example of how ‘process evaluation’ can become a component of an ‘impact evaluation’.

The Final Treatment and Control Groups

Two control groups for the evaluation, but how random are they i.e., is there a selection problem? An initial step in addressing this issue is to compare the characteristics of the treatment and control groups – you want their characteristics to be well matched

But ultimately evaluators need to utilise econometric techniques to deal with the Selection Bias issue In this evaluation, we employed matching estimators (PSM): - Duration models and difference-in-difference estimates are other techniques that can be used. Various sensitivity tests were conducted to address the dynamic bias issue: - Changed the unemployment duration threshold from minimum of 20 weeks to 25 and 30; - Also estimated the models for various exit points – 12, 15 and 18 months.

Given this, what were the findings on the effectiveness of the NEAP? Comparing the employment prospects of those who received JSA under the NEAP (treatment group) with those who were not referred (Control Group I), this component of the NEAP was found to have a negative impact: based on the table above, their chances of entering employment were reducted by about 15 per cent; When the treatment group’s employment prospects were compared with those who had participated in a NEAP interview in the past (Control Group II), the current NEAP treatment group did no better than this control group; Thus, the JSA component of the NEAP was found to be an ineffective route to employment – Why? Results held after various sensitivity checks. Results held after various sensitiv

How Reliable are out Results? We controlled for a wide-range of observables implying that unobserved factors should be less of a factor; Sensitivity tests seemed to confirm this. We had a highly representative control group. Still, while our matching estimator framework allows us to test the sensitivity of our estimates to unobserved bias, it does not eradicate it completely. In this regard, we are seeing the increased use of combined PSM and difference-in-difference methods to ensure that evaluation estimates are free from both selection bias (on observables) and unobserved bias (picking winners etc).

Implications of Findings Findings suggested the need for an overhaul of the NEAP eligibility and administration as it existed at the time of the evaluation – the system is currently being revamped. Also, provision of more intensive job search assistance – not feasible at present due to budget, competency and resource constraints within the DSP. Findings also suggested the need for Ireland to follow international best practice by developing a fully compulsory activation system with effective monitoring and sanction mechanisms – principal of mutual obligation with sanctions is now being applied, but again resource constraints are preventing the full implementation of regular and effective monitoring of clients job search intensity.

For further information: Report and Papers available at:

Thank you