1. Introduction to Cost Benefit Analysis
Tuesday 15 July 2014 Billsley and Yardley Wood Community Centre, Birmingham
#ourplaceCBA

2. Please familiarise yourself with the emergency evacuation procedures
Emergency procedures
Please familiarise yourself with the
emergency evacuation procedures
located within the inside cover of your
delegate handbook.

3. Please turn mobile devices to silent mode

4. Agenda The Our Place programme
What is CBA and why should areas apply it?
Case study exercises (in tables):
Agreeing outcomes
Calculating delivery costs
Tracking impact
Q&A and next steps

5. The Our Place programme
Click on hyperlink to launch video

6. What is Cost Benefit Analysis?

7. What do we mean by costs and benefits?
All additional costs needed to deliver project
Fiscal
Social
Economic
Fiscal
Economic
Social

8. The CBA model – costs Cost Benefit Analysis Tool Costs Economic Case
Capital
Revenue
In Kind
Economic Case
Net Present Public Value
Value for Money Benefit Cost Ratio
Cost Benefit Analysis Tool
Outcomes
Need
Engagement
Impact
Deadweight
Lag and drop-off
Financial Case
Net Present Budget Impact
Financial return on investment
Payback period
Cashability assumptions

9. Approach to calculating costs
What types of costs are borne?
Which agencies bear costs?
When are costs borne?

10. The CBA model – outcomes
Costs
Capital
Revenue
In Kind
Economic Case
Net Present Public Value
Value for Money Benefit Cost Ratio
Cost Benefit Analysis Tool
Outcomes
Need
Engagement
Impact
Deadweight
Lag and drop-off
Financial Case
Net Present Budget Impact
Financial return on investment
Payback period
Cashability assumptions

11. Types of benefits Example Fiscal benefits Public Value benefits
Employment mentoring programme for people with mental health problems
Reduction in unemployment payments as individuals gain employment
Increased output resulting from increased employment
Improved health with related impacts on well-being (e.g. confidence and self esteem)
Programme to tackle antisocial behaviour
Reduction in police, housing and local authority time spent responding to incidents.
Opportunity cost of avoided time spent by public sector agencies.
Increased patronage of local businesses resulting in net growth in local economy once displacement has been taken into account.
Reduced fear of crime amongst local residents.
Drug treatment programme
Savings in reactive health and criminal justice costs - emergency hospital visits, long term health costs, responses to crime.
Improved health and life expectancy of individual

12. Turning outcomes into benefits
Fiscal benefit
Econ benefit
Social benefit
Getting someone into a job (per annum)
£9,800
£14,044
Gaining a Level 3 apprenticeship (per annum)
£1,391
£1,925
Avoiding a child being taken into care (per annum)
£63,362
£7,460
Reduced incident of crime (per incident)
£609
£676
£1,648
Reduced Isolation (full distance travelled)
£8,500

13. Running the CBA model Assumptions tested/updated via evaluation
At risk /
affected population
%Engaged
Value
How many potential beneficiaries?
How many will we reach?
How many will achieve desired outcome?
What would have occurred anyway?
What is the value of the desired outcome?
%Impact
1 - %optimism bias
How confident are we in the evidence?
%Deadweight
%Retained
How many will complete the programme?
Assumptions tested/updated via evaluation
Optimism Bias (OB) correction applied to data
The Excel model follows a sequential process, by which we translate gross outcomes (i.e. what the pilot has achieved) into net outcomes (i.e. what % of this achievement is additional?) to net value (how much is this additional achievement worth to the taxpayer?)
Models often combine primary data with secondary evidence and assumptions:
We have primary evidence from pilots regarding target populations (for example, number of households within the pilot area)
We are having to make assumptions on what % of this population we will engage and what percentage will achieve the desired outcome
We are using national research to place a value on achieving a specific outcome (for example, the value of fewer evictions from RSL properties)
As projects feed primary data back to us we will be able to replace assumptions with actual evidence of how many people we are engaging and how many we are impacting. Thus, the model will become more accurate over time.
Ways of calculating deadweight – control groups, comparator areas, baseline forecasts, similar interventions, primary research etc

14. Case study example - benefits
Employment mentoring programme for people with low level mental health problems
Key outcomes:
Employment – moving off Employment Support Allowance
Improved mental health
Potential other outcomes
Reduced alcohol dependency
What questions do I need to ask to understand the benefits of improved outcomes?
Is the outcome relevant to the whole cohort (e.g. employment) or just a proportion (e.g. alcohol dependency)
What percentage of the cohort can we engage with?
How many will remain engaged for the whole programme?
Do we have evidence of impact from local pilot schemes?
Can we find evidence from elsewhere in the UK or abroad?
Do we have data on the success rates of the Business as Usual provision (e.g. Work Programme)?
Does research evidence indicate what would happen anyway?
How soon will benefits start?
How long will they last for?

15. The CBA model – outputs Cost Benefit Analysis Tool Costs Economic Case
Capital
Revenue
In Kind
Economic Case
Net Present Public Value
Value for Money Benefit Cost Ratio
Cost Benefit Analysis Tool
Outcomes
Need
Engagement
Impact
Deadweight
Lag and drop-off
Financial Case
Net Present Budget Impact
Financial return on investment
Payback period
Cashability assumptions

16. Family Intervention Project

17. Why should areas use CBA?
Localities can use CBA to deliver:
Informed decision-making
Cost-effective decision-making
Equitable decision-making

18. Refreshments
15 min break - refreshments available in conservatory – restart at 11.30am

19. Exercise 1: Agreeing outcomes
Undertaking CBA is not simple – it requires both an understanding of how to model things and the data and information that you are modelling
Rest of today is going to focus on the pre-CBA steps that if undertaken will mean that Our Place areas are well placed to begin CBA later in the year
We encourage local teams to draw up a ‘to do’ list based upon what they learn from remaining sections of the workshop
Pre-step 1 – confirming exactly what you are modelling

20. Has Area X’s project been successful?
If somebody asks you ‘was that project or programme successful?’ you should answer ‘successful at what?’
As an evaluator, before you start any research you need to be clear on:
What a project is hoping to change – what are the underlying issues, evidence of these?
What does the project involve – key activities. Which actions will affect which indicators?
How it is to be judged – what indicators do we expect to see change?
How much are we spending? What would represent VfM?
Who is meant to benefit – are their target groups? In target areas?
At what point in time it needs to judged – will impact be immediate or longer-term?
What of relevance is happening in the wider policy environment?
By asking for answers to these questions – build up a logical, rational evaluation framework – not judging a project unfairly, not wasting research input on unrelated issues
Logic Chain a useful tool in this respect

21. Contextual conditions
Project:
Intended Impacts
Contextual conditions
Key policy conditions
Intended Outcomes
Programme Objectives
Rationale
Talk through the steps in the Logic Chain, explaining how each links
Inputs
Activities
Outputs

22. Project: Greater Manchester Commitment
Intended Impacts
Reduced youth unemployment and NEET rates
Improved labour force skills & productivity
Reduction in employer skills shortages and staff turnover rates
Contextual conditions High rates of youth unemployment and inactivity vs. other groups Issue has been getting worse since start of economic downturn, esp. GM Limited employer engagement/satisfaction with training market
Key policy conditions
National policy – Youth Contract, provides grants, work placements, advice and BIS Skills for Sustainable Growth strategy
Local policy – each authority has their own scheme, FJF legacy
Intended Outcomes
Sustained employment for year olds
More GM providers working with cohort
More GM employers offering apprenticeships
More awareness of opportunities
Better integrated system
Draw down more central government expenditure
Programme Objectives
Increase employability and employment rate of GM year olds
Increase willingness of training providers to work with and willingness of employers to recruit from this group
Rationale
Financial incentives work
Cost < JSA
Single GM system
Training ethos
Inputs
£4.3m
3 FTEs
Local authorities
Training providers
Activities
Marketing
Administration
Grant making
Website
Brokering vacancies
Outputs
Vacancies
Grant payments
AGE payments
Net outputs
Leakage
Deadweight
Displacement

23. Task: Develop a Logic Chain for Poplar NCB case study
Each table will have an A3 copy of the Logic Chain and handouts explaining the Poplar NCB in more detail – using the latter, fill in the former – for some of the boxes the tables will have to be creative

24. Prompts What does the intervention involve?
How will support be delivered?
Where will it be operating? And when?
How many people will be eligible for the project?
What outcomes does the project aim to deliver?
Has baseline work been undertaken?
Which agencies are involved in the project?

25. Project: Poplar neighbourhood community budget
Intended Impacts
Contextual conditions
Key policy conditions
Intended Outcomes
Programme Objectives
Rationale
Inputs
Activities
Outputs

26. Lunch and networking
45 min break – restart promptly at 1.45pm when we will be looking at the issues of calculating costs and tracking impact

27. Exercise 2: Calculating your delivery costs
You cannot produce a CBA if you cannot state how much is being spent on a programme – surprisingly large number of public programmes do not have a firm handle on their delivery costs
In this section of the workshop we will show attendees the technique we use to help programme teams work out their delivery costs

28. Approach to calculating costs
What types of costs are borne?
Which agencies bear costs?
When are costs borne?

29. Case study example – building up costs
Integrated multi-agency safeguarding programme (Children’s Services and Domestic Abuse)
Cross-agency key worker model
Co-location of staff
Better identification, sequencing and availability of interventions
What are the relative costs compared to business as usual?
Potential increases
Potential efficiency savings
New posts (key workers, supervision)
Staff time – avoidance of duplication of effort
Training costs
Premises costs
Increased referrals to other agencies

30. Task: Identify costs associated with Poplar NCB case study

31. Refreshments
15 min break – refreshments available in conservatory - restart at 3.30pm

32. Exercise 3: Tracking your impact
From exercise 2 we have the basis for working out Our Place programme costs.
To complete a CBA we need data and intelligence regarding the impact of Our Place programmes upon specific outcomes – the outcomes that we identified in the Logic Chain exercise.
We could rely solely on secondary evidence of what other programmes have achieved plus some assumptions/informed guesses but in our CBA model this would result in a high optimism bias rating
Better approach is to ensure that primary data is being collected from the outset of the programme – going to look at research techniques that Our Place areas can use to gather such data

33. Maryland Scientific Method Scale
Level
Criteria 1
Correlation – Our Place clients less likely to be under threat of eviction than non-clients 2
Before & after – number of clients under threat of eviction fell after Our Place support 3
Single control - number of clients under threat of eviction fell after Our Place support and same fall not seen in another group 4
Multiple controls - number of clients, by gender/race/education level etc, under threat of eviction fell after Our Place support and same fall not seen in same groups elsewhere 5
Randomisation – Our Place team offered or refused support on random basis, supported group shows greater decrease in those under threat of eviction
Before designing monitoring and evaluation approach ask yourself what is proportionate

34. As a bare minimum, an evaluation should...
Move beyond correlation to causation
Focus on outcomes as much as outputs
Derive uniform, comparable data
Combine quantitative and qualitative methods
When writing up findings:
Include a description of the project
Make its research method clear
These are my suggestions rather than hard and fast rules!

35. Primary research techniques
Every contact with a member of the public is an opportunity to gather primary data – identify these contact points and then design info gathering tools where possible/proportionate
Feedback sheets - Need to know what support was accessed and effect on client circumstance before asking whether they would recommend support - Try to be specific on exactly what about the support was well received - Asking about areas for improvement tells you more than asking what was best received
Surveys – face to face versus electronic; Survey Monkey; filter/cohort profile questions + closed/open; pilot first; use community researchers; link to existing surveys/questions
Focus groups – should form a second round of primary research to explore initial findings; balance of participants; circulate discussion list; transcribe quotes
Case studies – these should be a combination of quantitative and qualitative evidence – powerful human interest approach and a chance to identify lessons learnt, especially regarding process/delivery issues

36. Secondary research techniques
Datasets – many government departments release data covering a wide range of topics at a fairly low spatial level on a regular basis – DOH, DfE, DWP etc. Local authorities sometimes do so too (Ward profiles) but quite a lot of local analytical capacity lost in the cuts – JSNAs a useful repository of local socio-economic data – helpful when identify programme contexts and comparator areas to track
Trouble with relying on official data in evaluations – often quite a time lag – e.g. Census 2011 – sometimes suppressed at low levels
Databases of ‘what works’ – a few examples on the slide – others include Project Oracle, Dartington and Washington Institute for Public Policy – can use these to make informed assumptions about how effective your Our Place programme will be at getting people into work or tackling issues of ill health etc
Early Intervention Foundation (EIF)
National Institute for Clinical Excellence (NICE)
Coalition for Evidence-Based Policy

37. More advanced evaluations will include...
A review of the delivery process
Consideration of Deadweight
Longitudinal tracking of clients
Value for Money calculations
Valuing of outcomes – leading to CBA
Strategic added value assessments
Action learning approaches
Process evaluation – compare to similar projects elsewhere/previously,
Deadweight – what would have happened anyway? Control and comparison groups, asking clients, looking at trends - NET
Acknowledge that benefits won’t appear immediately or may change over time – track clients
VfM – divide outputs/outcomes by delivery cost and if possible compare to benchmarks
Next step from VfM is putting a value on outcomes – SROI, CBA
SAV – is the project having a wider influence on how policy is developed and delivered locally and nationally
Action learning/experimentation – chance to influence things during the project

38. Task: Develop an evaluation framework for Poplar NCB
Refer tables back to Logic Chain and costings exercises

39. Q&A and next steps
4.45pm
Questions from floor

40. Further information julian. cox@neweconomymanchester. com david
Further information Knowledge Hub – Greater Manchester CBA Network

41. Introduction to Cost Benefit Analysis
Tuesday 15 July 2014 Billsley and Yardley Wood Community Centre, Birmingham
#ourplaceCBA