ESF EVALUATION PARTNERSHIP MEETING Bernhard Boockmann / Helmut Apel 18 March, 2011 Evaluation of the Federal Programme Kommunal-Kombi: Methodology and First Results Bernhard Boockmann / Helmut Apel Institute for Applied Economic Research (IAW), Tübingen ISG Institute for Social Research and Policy, Köln

Structure of the presentation Kommunal-Kombi: basic facts Methodology for estimating causal effects Evaluation design for Kommunal-Kombi Data requirements First results

Kommunal-Kombi: basic facts Programme objectives: Creation of additional subsidised jobs in regions with high and persistent long-term unemployment (limited to contracts with 3 yrs duration). Improvement of local public services. Reduction in welfare dependence / transition to employment of participants. Improvement in employability and social stabilisation. Target group: Long-term unemployed (> 2 yrs of unemployment). Welfare recipients (> 1 yr benefit duration). 101 local districts with unemployment rate > 10%.

Kommunal-Kombi: basic facts Number of jobs planned: 100,000 (all programmes within indicative instrument) Number of jobs created: 15,825 (job creation phase completed). Financial volume planned (Kommunal-Kombi only): 886m EUR at federal level, of which 306m EUR from ESF. Financial volume actual: 410m EUR at federal level, of which 137m EUR from ESF (as of 31-12-2009). Co-financing by Länder and municipalities: 31.7%. Co-financing by employers: 9.2%. Low demand due to co-financing requirement.

Basis facts about Kommunal-Kombi Regional distribution of jobs created:

Methodology for estimating causal effects Among outcome indicators, one may distinguish between: Gross outcomes, such as: Integration quota. Employability of beneficiaries after the programme. Net outcomes, such as: How many participants have found a job because of the programme? Change in employability due to the programme. Net effects are easier to interpret, because less information is required on context and benchmarks. They are also often a prerequisite for cost-benefit- analysis.

Methodology for estimating causal effects The ESF Operational Programme for Germany contains measures of gross effects: The OP also requires that the effectiveness of the programme should be evaluated.   Priority Axis C.1 Priority Axis C.2 OP total Output Indicator: Participants 61.500 38.500 100.000 Outcome Indicators: - Participants having improved employability + 33 % - Integration in employment 6 months after programme Baseline (in brackets)  15.4 % (15.4 %) 20.0 %  (15.4 %) 17.0 % Source: ESF-OP des Bundes 2007-2013, p. 182ff. Numbers refer to all programmes within indicative instrument 7.1

Methodology for estimating causal effects Drawback: net effects cannot simply be measured. They involve a comparison with a counterfactual. What would have been beneficiaries‘ job finding rates if they had not participated? This outcome is not observable. The unobservable outcome may be replaced by the outcome realised in a control group. Various ways of obtaining a control group: Experimental, quasi-experimental and non-experimental methods.

Methodology for estimating causal effects The methodology for estimating causal effects of labour market programme has matured over the last 10 years. Now an established part of in the evaluation of active labour market policy (and other policies). For a survey and meta-analysis of applications, see Card et al. (2010), Active Labor Market Policy Evaluations: A Meta-Analysis, Economic Journal, 120, F452-F477. The methodology is further explained e.g. in Angrist and Pischke (2009), Mostly Harmless Econometrics, Princeton UP.

Evaluation design for Kommunal-Kombi The evaluation of Kommunal-Kombi consists of different parts: Monitoring. Target achievement and effectiveness. Cost-benefit analysis. Estimation of treatment effects is one part of analysing effectiveness. Another is estimating effects at regional level, using a different methodology. Estimation of quantitative effects is supported by qualitative evidence.

Evaluation design for Kommunal-Kombi The task is to estimate the effects of the programme on transitions to employment at the individual level: Exit from welfare receipt. New employment, stable employment (duration >6 months). Definition of the control group: non-participants within in the same regions. Hence, regional labour market characteristics are the same between treatment and control groups. Problems: some evidence for selectivity (e.g., participants have better qualifications than non- participants).

Evaluation design for Kommunal-Kombi Solution: statistical matching approach: find the most similar person(s) in terms of observable socio- demographic characteristics (sex, age, qualifications, work history etc.). For each participant, we have 80 non-participants in the data (same sex, difference in birth years < 2 yrs). From this group, the matching algorithm selects the most similar persons (statistical twins).

Data requirements To avoid selectivity and identify causal effects convincingly, we need to condition on many characteristics. We also need good matching partners and, hence, many observations to choose from. Fortunately, detailed data from the Federal Employment Agency are available (Integrated Employment Biographies, IEB). The IEB contain employment data from the social insurance files, as well as process data from the public employment system (e.g., previous unemployment spells, programme participation, benefit receipt, …) and socio- demographic characteristics.

Data requirements The data allow constructing complete employment histories over >10 yrs. To identify participants, we combine this information with the ESF programme data. Still problematic, because many factors potentially influencing both participation and labour market integration are not observable  extensive checks required.

First results So far, we have no results on treatment effects. The reason is that the programm is still active and we don‘t yet observe participants‘ employment state after the programme. Very few drop-outs (<5 %). In the following, we present some descriptive results bearing on the question of selectivity.

First results More than 50% of participants are aged 50+, one third 55+. Only 18% do not have at least completed vocational training (all benefit recipients: 29%). High proportion of participants with very long unemployment spells: more than 4 yrs: 73% more than 8 yrs: 38%.

First results The evaluation uses an innovative operational concept for employability. In a previous study, we estimated the determinants of leaving unemployment, such as: Health, ability to work, social integration, job search activities. The magnitude of the influences were then used as index weights in the construction of an index of employability. Several sub-indexes for each dimension of employability were also constructed. Up to now, only one survey wave, hence cannot assess improvement.

Participants Kommunal-Kombi Unemployed SGB II welfare recipients First results   Participants Kommunal-Kombi Unemployed SGB II welfare recipients male female unweighted weighted Total indicator 0.738 0.707 0.537 0.481 0.527 0.472 Qualifications and competences 0.275 0.280 0.245 0.247 0.243 Health 0.022 0.032 -0.027 -0.030 -0.017 -0.026 Search behaviour 0.003 -0.025 0.030 0.000 0.040 Willingenss to make concessions -0.075 -0.070 -0.078 -0.060 -0.080 Individual resources 0.427 0.393 0.362 0.299 0.335 0.287 Social stability 0.081 0.087 0.001 0.018 0.004

First results Specific disadvantages (age, long-term unemployment). First results show that participants are not a random draw from the population of welfare recipients. Specific disadvantages (age, long-term unemployment). But also better employability than average. This (and more) has to be accounted for in quantitative analysis. Final results expected in 2013.