1. The Application of Statistical Matching to the 2010 ESF Leavers Survey
Rhys Davies
Wales Institute for Social and Economic Research, Data and Methods, Cardiff University
Gerry Makepeace
Cardiff Business School, Cardiff University
Findings are not in the public domain and are not for further dissemination

2. Overview of 2010 ESF Survey
Aim – understand the characteristics and outcomes of those participating in ESF funded projects
Methodology - telephone survey of people who were identified as having left an ESF project during 2010
Survey conducted Jun/July 2011
Interviews achieved with 7,509 participants (50% response rate)
P2/P3 Convergence , P1/P2 Competitiveness Programme
Similar survey conducted in 2010 among 2009 Leavers
Previous and current activity, why did ESF, withdrawal from ESF, skills gained from ESF, educational attainment (pre/ESF/post), current employment, perceived additionality

3. Transitions in Main Activity: Results from the 2009 Survey
Main activity before attending ESF course
Current main activity at time of survey
Paid employment
Education and training
Unemployed
Economically inactive
Total
8.3
1.5
1.1
0.6
11.4
5.6
3.9
3.0
1.3
13.8
21.8
7.4
21.0
6.1
56.4
2.8
2.6
10.7
17.5
38.7
14.3
28.0
19.9
100
Positive Transition
No Transition
Negative Transition

4. The Labour Force Survey as a Control Group
ESF Survey provides no control group
how do these transitions in to employment compare to those observed among wider population?
what would ESF participants have done in the absence of ESF
Use the UK Labour Force Survey to provide a control group against which the effect of participating in ESF can be evaluated
Respondents to the LFS are asked what they were doing one year earlier so can look at transitions in economic activity over a period of 12 months
Questions in ESF Survey and the coding of data are designed to align with definitions used in the LFS

5. Propensity Score Matching
Use Propensity Score Matching to extract people from the Labour Force Survey who share similar characteristics to ESF participants
Matching variables include
age
gender
partnership status (couple/single/living at parental home),
family status (dependent children under age of 18),
health (work limiting health condition)
educational attainment (NQF equivalents)
ethnicity
local employment (rate of employment among non-student population – allocated to decile groups)

6. Some Practical Issues
The LFS data pre-dates the information from the ESF survey
Introducing local labour market indicators requires access to detailed geographical identifiers
Time elapsed between pre-ESF and current activity varies among ESF respondents – not set at 12 months
a) the varying duration of ESF interventions and
b) the different end dates OF these interventions
Use career history data to identify activity 1 year following pre-ESF
Only variables unaffected by participation within an ESF project should be used in matching – we use some variables that are measured at the time of the surveys
Unemployed more homogenous than economically inactive

7. Career Profiles of Previously Unemployed – 2009 Survey

8. 12 Month Transition Rates in to Employment Among the Previously Unemployed
Local Area Employment Rates
(non-student population of working age)
LFS
Transition Rates
ESF
Transition Rate
(2009 Survey)
1st Decile (<68.8%)
29%
40%
2nd Decile ( %)
32%
3rd Decile ( %)
33%
4th Decile ( %)
34%
5th Decile ( %)
35%
6th Decile ( %)
42%
7th Decile ( %)
41%
8th Decile ( %)
44%
9th Decile ( %)
47%
10th Decile (>80.8%)
52%
Total
38%

9. Propensity Score Matching Results
Different matching techniques to consider sensitivity of results:
Nearest Neighbour
Select the individual who has ‘closest’ propensity score
Radius Matching
Average characteristics of controls within a specified radius
With and without replacement
allow controls to be matched to multiple ESF respondents
Adjust size of callipers
the maximum acceptable difference in propensity scores
Include/exclude:
ESF early withdrawers -
Proxy respondents from the LFS
Continuous career histories from the LFS

10. Conclusions, Limitations and Future Directions
Additional insights can be provided by PSM if ESF surveys are designed to align with other available sources of data
Limitations remain
can only match individuals on the basis of measurable characteristics
our control group may have participated in some other form of assistance
some matching variables could have varied over the period during which employment transitions were being measured
Future challenges
refinement of statistical matching to use longitudinal data sources
understanding of what works requires project level analysis