1. ESF EVALUATION PARTNERSHIP MEETING Bernhard Boockmann / Helmut Apel
18 March, 2011
Evaluation of the Federal Programme Kommunal-Kombi: Methodology and First Results
Bernhard Boockmann / Helmut Apel
Institute for Applied Economic Research (IAW), Tübingen
ISG Institute for Social Research and Policy, Köln

2. Structure of the presentation
Kommunal-Kombi: basic facts
Methodology for estimating causal effects
Evaluation design for Kommunal-Kombi
Data requirements
First results

3. Kommunal-Kombi: basic facts
Programme objectives:
Creation of additional subsidised jobs in regions with high and persistent long-term unemployment (limited to contracts with 3 yrs duration).
Improvement of local public services.
Reduction in welfare dependence / transition to employment of participants.
Improvement in employability and social stabilisation.
Target group:
Long-term unemployed (> 2 yrs of unemployment).
Welfare recipients (> 1 yr benefit duration).
101 local districts with unemployment rate > 10%.

4. Kommunal-Kombi: basic facts
Number of jobs planned: 100,000 (all programmes within indicative instrument)
Number of jobs created: 15,825 (job creation phase completed).
Financial volume planned (Kommunal-Kombi only): 886m EUR at federal level, of which 306m EUR from ESF.
Financial volume actual: 410m EUR at federal level, of which 137m EUR from ESF (as of ).
Co-financing by Länder and municipalities: 31.7%.
Co-financing by employers: 9.2%.
Low demand due to co-financing requirement.

5. Basis facts about Kommunal-Kombi
Regional distribution of jobs created:

6. Methodology for estimating causal effects
Among outcome indicators, one may distinguish between:
Gross outcomes, such as:
Integration quota.
Employability of beneficiaries after the programme.
Net outcomes, such as:
How many participants have found a job because of the programme?
Change in employability due to the programme.
Net effects are easier to interpret, because less information is required on context and benchmarks.
They are also often a prerequisite for cost-benefit- analysis.

7. Methodology for estimating causal effects
The ESF Operational Programme for Germany contains measures of gross effects:
The OP also requires that the effectiveness of the programme should be evaluated.
Priority Axis C.1
Priority Axis C.2
OP total
Output Indicator:
Participants
61.500
38.500
Outcome Indicators:
- Participants having improved employability
+ 33 %
- Integration in employment months after programme
Baseline (in brackets) 
15.4 %
(15.4 %)
20.0 %
 (15.4 %)
17.0 %
Source: ESF-OP des Bundes , p. 182ff. Numbers refer to all programmes within indicative instrument 7.1

8. Methodology for estimating causal effects
Drawback: net effects cannot simply be measured. They involve a comparison with a counterfactual.
What would have been beneficiaries‘ job finding rates if they had not participated?
This outcome is not observable.
The unobservable outcome may be replaced by the outcome realised in a control group.
Various ways of obtaining a control group:
Experimental, quasi-experimental and non-experimental methods.

9. Methodology for estimating causal effects
The methodology for estimating causal effects of labour market programme has matured over the last 10 years.
Now an established part of in the evaluation of active labour market policy (and other policies).
For a survey and meta-analysis of applications, see Card et al. (2010), Active Labor Market Policy Evaluations: A Meta-Analysis, Economic Journal, 120, F452-F477.
The methodology is further explained e.g. in Angrist and Pischke (2009), Mostly Harmless Econometrics, Princeton UP.

10. Evaluation design for Kommunal-Kombi
The evaluation of Kommunal-Kombi consists of different parts:
Monitoring.
Target achievement and effectiveness.
Cost-benefit analysis.
Estimation of treatment effects is one part of analysing effectiveness.
Another is estimating effects at regional level, using a different methodology.
Estimation of quantitative effects is supported by qualitative evidence.

11. Evaluation design for Kommunal-Kombi
The task is to estimate the effects of the programme on transitions to employment at the individual level:
Exit from welfare receipt.
New employment, stable employment (duration >6 months).
Definition of the control group: non-participants within in the same regions. Hence, regional labour market characteristics are the same between treatment and control groups.
Problems: some evidence for selectivity (e.g., participants have better qualifications than non- participants).

12. Evaluation design for Kommunal-Kombi
Solution: statistical matching approach: find the most similar person(s) in terms of observable socio- demographic characteristics (sex, age, qualifications, work history etc.).
For each participant, we have 80 non-participants in the data (same sex, difference in birth years < 2 yrs).
From this group, the matching algorithm selects the most similar persons (statistical twins).

13. Data requirements
To avoid selectivity and identify causal effects convincingly, we need to condition on many characteristics.
We also need good matching partners and, hence, many observations to choose from.
Fortunately, detailed data from the Federal Employment Agency are available (Integrated Employment Biographies, IEB).
The IEB contain employment data from the social insurance files, as well as process data from the public employment system (e.g., previous unemployment spells, programme participation, benefit receipt, …) and socio- demographic characteristics.

14. Data requirements
The data allow constructing complete employment histories over >10 yrs.
To identify participants, we combine this information with the ESF programme data.
Still problematic, because many factors potentially influencing both participation and labour market integration are not observable  extensive checks required.

15. First results
So far, we have no results on treatment effects.
The reason is that the programm is still active and we don‘t yet observe participants‘ employment state after the programme.
Very few drop-outs (<5 %).
In the following, we present some descriptive results bearing on the question of selectivity.

16. First results
More than 50% of participants are aged 50+, one third
Only 18% do not have at least completed vocational training (all benefit recipients: 29%).
High proportion of participants with very long unemployment spells:
more than 4 yrs: 73%
more than 8 yrs: 38%.

17. First results
The evaluation uses an innovative operational concept for employability.
In a previous study, we estimated the determinants of leaving unemployment, such as:
Health, ability to work, social integration, job search activities.
The magnitude of the influences were then used as index weights in the construction of an index of employability.
Several sub-indexes for each dimension of employability were also constructed.
Up to now, only one survey wave, hence cannot assess improvement.

18. Participants Kommunal-Kombi Unemployed SGB II welfare recipients
First results
Participants Kommunal-Kombi
Unemployed SGB II welfare recipients
male
female
unweighted
weighted
Total indicator
0.738
0.707
0.537
0.481
0.527
0.472
Qualifications and competences
0.275
0.280
0.245
0.247
0.243
Health
0.022
0.032
-0.027
-0.030
-0.017
-0.026
Search behaviour
0.003
-0.025
0.030
0.000
0.040
Willingenss to make concessions
-0.075
-0.070
-0.078
-0.060
-0.080
Individual resources
0.427
0.393
0.362
0.299
0.335
0.287
Social stability
0.081
0.087
0.001
0.018
0.004

19. First results Specific disadvantages (age, long-term unemployment).
First results show that participants are not a random draw from the population of welfare recipients.
Specific disadvantages (age, long-term unemployment).
But also better employability than average.
This (and more) has to be accounted for in quantitative analysis.
Final results expected in 2013.