1. Counterfactual Impact Analysis applied in the ESF-Evaluation in Austria (period 2000-2006)
Contribution to the Expert-Hearing: Member States Experiences on Using Control Groups in ESF-Evaluation
Helmut Mahringer, Austrian Institute of Economic Research (WIFO)

2. CIE in Austria Background: ESF evaluation project
Conditions for application of counterfactual impact evaluation (CIE)
Method
Some results
Lessons learnt

3. Background: ESF Evaluation
Focus on active labour market policy:
Half ot the budget of ESF Objective 3 in Austria ( ) for active labour market measures for the unemployed
Implementation along with nationally financed measures
New comprehensive administrative databases became available
Unemployment, employment, active measures, ...
Focus of monitoring on gross transition rates into employment
Unintended effects on selection
Tasks for Evaluation
“net- effects”: impact analysis
Added value of ESF

4. Conditions for application of counterfactual impact evaluation (CIE)
Requirements
Excellent information on participants and non-participants
Treatment was not obligatory for certain groups (i.e. existence of a potential control-group of relevant size)
Conditional independence assumption was plausible
Excellent data background established: administrative, individual and anonymised
PES: treatment, treated, unemployment, unemployed, benefits ( )
Social insurance records (employment, unemployment, out of labour force, wages, employer- and employee-characteristics: long pre-treatment labour market career)
Complementary information (e.g. regional labour market conditions)

5. Method Definition of treatment, participants and non-participants
Construction of treatment episodes
No other treatment shortly before
Determination of (hypothetical) entry into treatment (start of observation of the follow-up period)
Definition of the control group
Nearest neighbour propensity score matching
Testing for balancing property
Heterogeneity of effects: Separate for age-groups, gender, types of treatment
Impact measured as difference in outcome variables between treatments an controls
(Days in) regular employment in three years following the entry into treatment
Unemployment
Out of labour force

6. Some results: Impact on regular employment
Training measures:
significantly positive effects for women aged between 25 and 44 years and - for the training course fee allowance - also for men
Active job search measures:
positive impact on further employment integration for women between 25 and 44 years of age
Employment in socio-economic enterprises:
significantly positive for men and women between 45 and 54 years of age
In most cases, we found treatment effects to be higher for women than for men
Training seems to be more helpful for younger unemployed and job creation schemes in socio-economic enterprises more for the older

7. Lessons learnt
CIE is important to justify policy interventions as well as re- design (what works and for whom?)
Development of databases was crucial
In Austria more information on the type of treatment would enhance the strength of interpretation
Social experiments might be a method to overcome lack of information for specific interventions
Macroeconomic dimension of interventions is usually not identified by micro-econometric studies
Aggregation of effects on EU-level will stay difficult to achieve