1. Looking for statistical twins
Propensity Score Matching as a tool for measuring the effectiveness of Active Labour Market Policies implemented by Croatian Employment Service
Michal Kotnarowski, Ph.D.
Institute of Political Studies, Polish Academy of Sciences
Study conducted in collaboration with IPSOS Croatia and
Croatian Employment Service

2. Plan of the presentation
Research problem
Propensity Score Matching
Evaluation of ALMP implemented in Croatia
General comments

3. Research Problem
A set of Active Labour Market Policies was implemented on the Croatian labour market.
Target: unemployed people.
Aim: after the intervention unemployed should be able to get a new job.
Different kind of interventions were applied:
public works, training for the unemployed, start-up incentives, employment incentives, workplace training.

4. A person participated into ALMP
Research Problem
The research problems:
What is the effectiveness of these interventions?
Whether people subjected to intervention got a new job?
And if they got a new job, maybe they would get a new job anyway, even without being subjected to intervention. What was the unique impact of the interventions on getting a new job? It is a question about casual link:
A person participated into ALMP
A person got a new job

5. Research Problem
We can check proportions of people subjected to intervention who got a new job after some time (e.g. half of the year) after end of participation in ALMP. However:
it could be difficult to assess whether given proportion of participants who got a new job is a success or failure of the intervention:
40% of employment among young, well educated university graduates? Maybe they would get a new job anyway.
5% of employment among older, long-time unemployed people? Maybe for those 5% getting a new job would not be possible without being subjected to the ALMPs.
Our question: what would happen to people subjected to intervention if they were not subjected to it? What is the „real” impact of intervention on chances of getting a new job?

6. Experimental and quasi-experimental design
Selection bias – appears if there are differences between intervention and control group in chances of:
- participation in the intervention,
- chances of success after the intervention.
How to eliminate a selection bias:
a perfect solution is a randomisation within experimental design of the study,
when experiment is not possible, an available option is quasi-experiment.

7. Propensity Score Matching
N treated < N non-treated (reservoir of the control group)
The task: select control group out of the whole non-treated group in a way to eliminate selection bias. People in the control group should be as identical as possible to the people in the treatment group. We need to rely on observable characteristics of people in both groups.
The task: for each person in the treated group, we need to find a statistical twin from the non-treated group.
One option is to apply exact match, match cases on the basis of vector of values of set of variables. However, if we need to use many variable, this solution is impossible and impractical.
Possible solution is the Propensity Score Matching: matching is based on the basis of a scalar, one value estimated for each person in the dataset (treated and non treated) indicating probability of being treated by the intervention.
Propensity scores could be estimated using logistic regression model.

8. Propensity Score Matching
Steps in the Propensity Score Matching:
Selection of set of variables (X) predicting participation in the intervention and outcome
Estimating propensity scores for all individuals (treated and non- treated) on the basis of X
Matching each person from treated group with 1 or more persons from non-treated groups. Group of matched non-treated = control group. Possible matches: 1:1 or 1:n (in our case 1:5 was applied).
Verification of the quality of matching.
Estimation of the effect of the intervention:

9. Evaluation of Croatia’s ALMP - application of Propensity Score Matching
Our study was based on dataset of Croatian Employment Register and Croatian Pension Fund data.
We were interested in people unemployed within the period
Aim of the project was to estimate effect of the intervention for the five type of the ALMP measures.

10. Evaluation of Croatia’s ALMP - Propensity Score Matching
Measures subjected to the evaluation:
Workplace training for the unemployed,
Employment incentives,
Start-up incentives,
Public works,
Labour market oriented training.
Some measures had their sub-types. We estimated separate model for each sub-measure. Altogether 9 general models were estimated. The analysed dataset was very big, some models were estimated separately for one or half of the year time periods

11. Application of PSM
Dataset at our disposal: Croatian Employment Register and Croatian Pension Fund
Propensity scores were estimated using logistic regression analysis
Independent variables (X), predictors of participation in the intervention and outcome:
age, gender, education level, education field, disabled person, war veteran, age of the youngest child, single parent, length of the current unemployment episode, work experience, status prior current unemployment episode, sector of economy prior current unemployment episode, type of occupation prior unemployment episode, Employment Service categorisation of the person, unemployment benefits for the person, prior participation in ALMP, unemployment rate of the region.
values of variables for the time period before the intervention.
Dependent variable in the model: participation in ALMP (0 - non treated, 1 - treated).
For each person probability of being treated was estimated:

12. Application of PSM
Matching procedure: each person treated by the intervention was matched with person(s) having the most similar value of propensity score.
Various techniques of matching available:
nearest neighbour,
calliper,
kernel.
In our study:
matching 1:5 was applied (possible because of large number of non- treated individuals in the dataset).
Matching technique: nearest neighbour with calliper (0.25s, where s was mean standard deviation of propensity scores in treated and non- treated group).
In our study matching was obtained using MatchIt package (Ho, Imai, King, Stuart 2011) in R software.

13. Application of PSM Verification of the matching quality:
Comparison of distributions of propensity scores and logistic model predictors between groups: treated, non-treated and matched non-treated:
Graphical examination of differences in distributions.
Here: Example of public works. March-July 2012

14. Net effect of the intervention
Professional training for the unemployed

15. Net effect of the intervention
Employment incentives –
young people without work experience
Matched non participant - % of employed

16. Net effect of the intervention
Employment incentives – long term unemployed

17. Net effect of the intervention
Employment incentives – people over 50 y.o.

18. Net effect of the intervention
Start-up incentive

19. Net effect of the intervention
Public work

20. General comments

21. Data requirements for application of Propensity Score Matching
Data about individuals subjected to a given intervention (treated) and individuals not subjected to it (not treated).
Data should contain two kinds of respondent’s characteristics (variables):
1. should be predictor of individual’s chances of being subjected to the measure
2. should be predictor of individual’s chances of success of the intervention (i.e. having a job)
A number of individuals non-treated (reservoir of a control group) should be substantially larger than a number of treated individuals.
Possible sources of data:
Administrative data (e.g. Croatian Employment Register)
+ large dataset, huge reservoir of control group,
+ population data
+ data exist, no costs of producing data – fieldwork, but rather high costs of data processing
- possible lack of variables suitable for PSM (administrative data collected for other reasons than evaluation)
Survey data:
+ dataset contains all variables suitable for PSM analysis,
- possible problem with a low number of non-treated (reservoir of a control group),
- possible high costs of data collection (large sample needed)

22. Remarks on the interpretation of the results
When interpreting the results, one has to remember the following:
PSM enables to compare groups, not individuals;
It is always required to check the matching quality – it is always possible that quality of matching is poor and estimation of the effect of intervention is not reliable;
In case of analyses conducted using samples (not the population), it is required to calculate standard errors and confidence intervals of the effect of the interventions;
Research should consider whether all relevant predictors were included in the PSM model. In a case of administrative data, psychological variables are often missing, i.e. motivation, determination;
And numbers are not everything – each result has to be carefully interpreted and compared with results of other studies.

23. Conclusions
Despite possible problems in application and interpretation of propenstity score matching, it is still a gold standard of evaluation of the impact of different kind of interventions.

24. Evaluation report with results of our study accessible at:
ault.aspx?ID=30483

25. Thank you