1. Measuring Impact – what can we learn from recent DWP evaluations?
Simon Marlow

2. Overview
What support has been/is available to Jobcentre Plus claimants?
Focus on ‘work experience’ and Future Jobs Fund
Impacts
Cost Benefit Analysis of Future Jobs Fund
Future thoughts
Focus on WE and FJF – number of other programmes for young people WP

3. Claimant journey for 18-24 year olds on JSA
Post mid-2011
New claim
Maximum of 9 months JCP offer
Includes:
1) Core interventions – signing etc.
2) Flexible interventions eg. work experience, Mandatory Work Activity
3) Back to work support
4) Flexible Support Fund
Work Programme – Mandatory for 2 years.
Early access at 3 months for disadvantage groups and with long benefit history
Post Work
Programme
- trial
Length of JSA claim
mid-2011
New claim
Flexible New Deal Phase 1 areas - 1 year
Flexible New Deal – Mandatory for 1 year
Early Access for some groups
The flagship programme in all of this is the WP – where support for the LTU is contracted out of the Department and to external providers
The Key principle of the Work Programme being sustained job outcomes
The Key features are:
Greater emphasis on payment by results
Promotion of competition and everything that can bring – Competitive tendering, Cost-effectiveness, increased outcomes etc.
Promoting innovation, both within JCP and external contractors – for example greater use of the 3rd sector
Establish longer contracts and provide stability for contractors and ultimately greater opportunities to learn and increase effectiveness
Pre-work Programme initiatives
Post Work Programme initiatives
Initiatives can mandatory or voluntary and flexibility/adviser discretion is promoted
Starting to evaluate the new intiatives
Young Person’s Guarantee
(includes Future Jobs Fund)
Flexible New Deal Phase 2
– 6 months
New Deal for Young People – Mandatory
Early Access for some groups

4. Work experience (WE) and Future Jobs Fund (FJF)
WE – January present
Who it is for?
Mainly 18-24yrs old (a few 25+)
Guidance - 13wks + JSA claim duration
JCP adviser discretion to refer
Aim
Help people gain practical experience they need to secure a job
What is it?
2-8 week placement with employer
Employers expected to allow time for job search where possible
Conditionality
Voluntary scheme once referred
Continue signing; claiming benefit
No sanctions for non-completion
Cost - £325/per participant
FJF – October 2009 – March 2011
Who was it for?
Mainly (a few 25+ in hotspots)
Guidance - 26 weeks JSA duration
Voluntary
Aim
Build skills/work experience for disadvantaged jobseekers to help them find unsubsidised employment
What was it?
6m subsidised placement, 25 hrs/wk at least min wage
Additional jobs, benefit local communities
Organisations mainly from public sector and third sector
Conditionality
Voluntary, but from Apr 10 if refused, had to do ‘Community Task Fund’ (another strand of the Young Person’s Guarantee)
Stop claiming benefit
Cost - Employers were
paid £6,500/per placement – about £650m total

5. WE and FJF reach similar levels of take up
By end of Nov 11 WE starts reach 6k/month
By Apr 10 FJF starts reach 6k/month
FJF reached 105,000
So far WE has reached xxx

6. Methodology
Want to understand programme impact on benefit receipt and employment
Isolate impact by trying to establish a plausible counterfactual
In absence of an RCT, Propensity Score Matching used:
Regress on observable characteristics from DWP data to try and get probability of participation (controlling for selection)
Match to non-participant group (JSA claimants from same period without a start on the programme)
Pseudo starts given to non-participant group so that outcomes can be compared
Main assumption: hope that observable characteristics can explain participation (ie needs to explain participant ‘motivation’)

7. Matching quality on observables is very good
Before matching
After matching

8. Work Experience – 1,300 individuals, Jan – March 2013
Benefit Impacts
– 6 ppts at 21 weeks/5 fewer days
Benefit receipt rate .
Employment rate
Employment Impact – 8ppts at 21 weeks/8 additional days

9. FJF impacts on Benefit/Welfare support – first 12,000 individuals from Oct 2009 – April 2010
Benefit impacts
Over 6m fewer days
18-24m – -7 ppts/ 41 fewer days
Benefit receipt rate
‘Welfare support’ (benefit or FJF job) impacts
Over 6m - 33 additional days
18- 24m – average -7 ppts/ 41 fewer days
‘Welfare support’ (benefit or FJF job) receipt rate
Baseline sensitivities – broadly similar results

10. FJF impacts on employment/unsubsidised employment (caveat – using HMRC P45 data)
Employment impacts
0-6m – 99 additional days
18-24m – average 10 ppts/ 57 additional days
Employment receipt rate
Unsubsidised impacts
0- 6m – 45 fewer days
18-24m – average 10 ppts/ 57 additional days
Unsubsidised employment (job other than FJF job) receipt rate

11. Summary of Impacts Caveats
WE - by 21 weeks:
Benefit Impact
6 ppts off benefit – overall 5 fewer days.
Employment Impact
8 ppts in employment – overall 8 additional days
FJF – over 2 years
Benefit/Welfare Impact
0-6m – 127 fewer days off ben/33 more days on welfare support
18-24m – average 7 ppts off ben; 41 fewer days off ben
Employment/Unsub employment impact
0-6m – 99 add days in employment/45 fewer days in unsub employment
18-24m – 10 ppts in employment; 57 additional days in employment
Caveats
Work experience first early cohort – only 1,300 starts; only 21 weeks
More likely than not to be overestimates for both
HMRC data poor quality so employment impacts are less robust
Compare with New Deal for Young People – 64 fewer days off JSA over 4 years
International job creation scheme evidence poor

12. Costs & benefits of the Future Jobs Fund
FJF programme impact
Cost (-) / benefit (+) per participant (£)
Participants
Participants’ employers
Exchequer
Society
Increase in output
+ 4,400
Increase in wages
+ 4,300
- 4,300
FJF employer payments
+ 6,850
- 6,850
Reduction in operational costs
+ 900
Reduction in benefits
- 1,450
+ 1,450
Increase in taxes
- 1,000
- 100
+ 1,100
Increase in travel & childcare costs
- 250
Reduction in healthcare costs
+ 300
Redistributive costs & benefits
+ 2,400
Social cost of exchequer finance
Net Present Value (NPV)
+ 4,000
- 3,100
+ 7,750
4 perspectives
Lots of caveats around this
If cohort is representative, then net cost to exchequer ~ £330m
(against gross cost of programme - £720m)

13. Sensitivity to assumptions
Wide range of uncertainty

14. Future
Developing our ways to measure impacts and cost benefit analysis of programmes
Extend work experience analysis
Day one support trial
Work Programme evaluation

15. References Early work experience impacts
Future Jobs Fund impacts, costs and benefits
Two main sources of guidance for CBA:
The HM Treasury Green Book
The DWP Social-Cost Benefit Analysis Framework