Introduction to Cost Benefit Analysis Tuesday 15 July 2014 Billsley and Yardley Wood Community Centre, Birmingham #ourplaceCBA

Please familiarise yourself with the emergency evacuation procedures Emergency procedures Please familiarise yourself with the emergency evacuation procedures located within the inside cover of your delegate handbook.

Please turn mobile devices to silent mode

Agenda The Our Place programme What is CBA and why should areas apply it? Case study exercises (in tables): Agreeing outcomes Calculating delivery costs Tracking impact Q&A and next steps

The Our Place programme Click on hyperlink to launch video

What is Cost Benefit Analysis?

What do we mean by costs and benefits? All additional costs needed to deliver project Fiscal Social Economic Fiscal Economic Social

The CBA model – costs Cost Benefit Analysis Tool Costs Economic Case Capital Revenue In Kind Economic Case Net Present Public Value Value for Money Benefit Cost Ratio Cost Benefit Analysis Tool Outcomes Need Engagement Impact Deadweight Lag and drop-off Financial Case Net Present Budget Impact Financial return on investment Payback period Cashability assumptions

Approach to calculating costs What types of costs are borne? Which agencies bear costs? When are costs borne?

The CBA model – outcomes Costs Capital Revenue In Kind Economic Case Net Present Public Value Value for Money Benefit Cost Ratio Cost Benefit Analysis Tool Outcomes Need Engagement Impact Deadweight Lag and drop-off Financial Case Net Present Budget Impact Financial return on investment Payback period Cashability assumptions

Types of benefits Example Fiscal benefits Public Value benefits Employment mentoring programme for people with mental health problems Reduction in unemployment payments as individuals gain employment Increased output resulting from increased employment Improved health with related impacts on well-being (e.g. confidence and self esteem) Programme to tackle antisocial behaviour Reduction in police, housing and local authority time spent responding to incidents. Opportunity cost of avoided time spent by public sector agencies. Increased patronage of local businesses resulting in net growth in local economy once displacement has been taken into account. Reduced fear of crime amongst local residents. Drug treatment programme Savings in reactive health and criminal justice costs - emergency hospital visits, long term health costs, responses to crime. Improved health and life expectancy of individual

Turning outcomes into benefits Fiscal benefit Econ benefit Social benefit Getting someone into a job (per annum) £9,800 £14,044 Gaining a Level 3 apprenticeship (per annum) £1,391 £1,925 Avoiding a child being taken into care (per annum) £63,362 £7,460 Reduced incident of crime (per incident) £609 £676 £1,648 Reduced Isolation (full distance travelled) £8,500

Running the CBA model Assumptions tested/updated via evaluation At risk / affected population %Engaged Value How many potential beneficiaries? How many will we reach? How many will achieve desired outcome? What would have occurred anyway? What is the value of the desired outcome? %Impact 1 - %optimism bias How confident are we in the evidence? %Deadweight %Retained How many will complete the programme? Assumptions tested/updated via evaluation Optimism Bias (OB) correction applied to data The Excel model follows a sequential process, by which we translate gross outcomes (i.e. what the pilot has achieved) into net outcomes (i.e. what % of this achievement is additional?) to net value (how much is this additional achievement worth to the taxpayer?) Models often combine primary data with secondary evidence and assumptions: We have primary evidence from pilots regarding target populations (for example, number of households within the pilot area) We are having to make assumptions on what % of this population we will engage and what percentage will achieve the desired outcome We are using national research to place a value on achieving a specific outcome (for example, the value of fewer evictions from RSL properties) As projects feed primary data back to us we will be able to replace assumptions with actual evidence of how many people we are engaging and how many we are impacting. Thus, the model will become more accurate over time. Ways of calculating deadweight – control groups, comparator areas, baseline forecasts, similar interventions, primary research etc

Case study example - benefits Employment mentoring programme for people with low level mental health problems Key outcomes: Employment – moving off Employment Support Allowance Improved mental health Potential other outcomes Reduced alcohol dependency What questions do I need to ask to understand the benefits of improved outcomes? Is the outcome relevant to the whole cohort (e.g. employment) or just a proportion (e.g. alcohol dependency) What percentage of the cohort can we engage with? How many will remain engaged for the whole programme? Do we have evidence of impact from local pilot schemes? Can we find evidence from elsewhere in the UK or abroad? Do we have data on the success rates of the Business as Usual provision (e.g. Work Programme)? Does research evidence indicate what would happen anyway? How soon will benefits start? How long will they last for?

The CBA model – outputs Cost Benefit Analysis Tool Costs Economic Case Capital Revenue In Kind Economic Case Net Present Public Value Value for Money Benefit Cost Ratio Cost Benefit Analysis Tool Outcomes Need Engagement Impact Deadweight Lag and drop-off Financial Case Net Present Budget Impact Financial return on investment Payback period Cashability assumptions

Family Intervention Project

Why should areas use CBA? Localities can use CBA to deliver: Informed decision-making Cost-effective decision-making Equitable decision-making

Refreshments 15 min break - refreshments available in conservatory – restart at 11.30am

Exercise 1: Agreeing outcomes Undertaking CBA is not simple – it requires both an understanding of how to model things and the data and information that you are modelling Rest of today is going to focus on the pre-CBA steps that if undertaken will mean that Our Place areas are well placed to begin CBA later in the year We encourage local teams to draw up a ‘to do’ list based upon what they learn from remaining sections of the workshop Pre-step 1 – confirming exactly what you are modelling

Has Area X’s project been successful? If somebody asks you ‘was that project or programme successful?’ you should answer ‘successful at what?’ As an evaluator, before you start any research you need to be clear on: What a project is hoping to change – what are the underlying issues, evidence of these? What does the project involve – key activities. Which actions will affect which indicators? How it is to be judged – what indicators do we expect to see change? How much are we spending? What would represent VfM? Who is meant to benefit – are their target groups? In target areas? At what point in time it needs to judged – will impact be immediate or longer-term? What of relevance is happening in the wider policy environment? By asking for answers to these questions – build up a logical, rational evaluation framework – not judging a project unfairly, not wasting research input on unrelated issues Logic Chain a useful tool in this respect

Contextual conditions Project: Intended Impacts Contextual conditions Key policy conditions Intended Outcomes Programme Objectives Rationale Talk through the steps in the Logic Chain, explaining how each links Inputs Activities Outputs

Project: Greater Manchester Commitment Intended Impacts Reduced youth unemployment and NEET rates Improved labour force skills & productivity Reduction in employer skills shortages and staff turnover rates Contextual conditions High rates of youth unemployment and inactivity vs. other groups Issue has been getting worse since start of economic downturn, esp. GM Limited employer engagement/satisfaction with training market Key policy conditions National policy – Youth Contract, provides grants, work placements, advice and BIS Skills for Sustainable Growth strategy Local policy – each authority has their own scheme, FJF legacy Intended Outcomes Sustained employment for 16-24 year olds More GM providers working with cohort More GM employers offering apprenticeships More awareness of opportunities Better integrated system Draw down more central government expenditure Programme Objectives Increase employability and employment rate of GM 16-24 year olds Increase willingness of training providers to work with 16-24 and willingness of employers to recruit from this group Rationale Financial incentives work Cost < JSA Single GM system Training ethos Inputs £4.3m 3 FTEs Local authorities Training providers Activities Marketing Administration Grant making Website Brokering vacancies Outputs Vacancies Grant payments AGE payments Net outputs Leakage Deadweight Displacement

Task: Develop a Logic Chain for Poplar NCB case study Each table will have an A3 copy of the Logic Chain and handouts explaining the Poplar NCB in more detail – using the latter, fill in the former – for some of the boxes the tables will have to be creative

Prompts What does the intervention involve? How will support be delivered? Where will it be operating? And when? How many people will be eligible for the project? What outcomes does the project aim to deliver? Has baseline work been undertaken? Which agencies are involved in the project?

Project: Poplar neighbourhood community budget Intended Impacts Contextual conditions Key policy conditions Intended Outcomes Programme Objectives Rationale Inputs Activities Outputs

Lunch and networking 45 min break – restart promptly at 1.45pm when we will be looking at the issues of calculating costs and tracking impact

Exercise 2: Calculating your delivery costs You cannot produce a CBA if you cannot state how much is being spent on a programme – surprisingly large number of public programmes do not have a firm handle on their delivery costs In this section of the workshop we will show attendees the technique we use to help programme teams work out their delivery costs

Approach to calculating costs What types of costs are borne? Which agencies bear costs? When are costs borne?

Case study example – building up costs Integrated multi-agency safeguarding programme (Children’s Services and Domestic Abuse) Cross-agency key worker model Co-location of staff Better identification, sequencing and availability of interventions What are the relative costs compared to business as usual? Potential increases Potential efficiency savings New posts (key workers, supervision) Staff time – avoidance of duplication of effort Training costs Premises costs Increased referrals to other agencies

Task: Identify costs associated with Poplar NCB case study

Refreshments 15 min break – refreshments available in conservatory - restart at 3.30pm

Exercise 3: Tracking your impact From exercise 2 we have the basis for working out Our Place programme costs. To complete a CBA we need data and intelligence regarding the impact of Our Place programmes upon specific outcomes – the outcomes that we identified in the Logic Chain exercise. We could rely solely on secondary evidence of what other programmes have achieved plus some assumptions/informed guesses but in our CBA model this would result in a high optimism bias rating Better approach is to ensure that primary data is being collected from the outset of the programme – going to look at research techniques that Our Place areas can use to gather such data

Maryland Scientific Method Scale Level Criteria 1 Correlation – Our Place clients less likely to be under threat of eviction than non-clients 2 Before & after – number of clients under threat of eviction fell after Our Place support 3 Single control - number of clients under threat of eviction fell after Our Place support and same fall not seen in another group 4 Multiple controls - number of clients, by gender/race/education level etc, under threat of eviction fell after Our Place support and same fall not seen in same groups elsewhere 5 Randomisation – Our Place team offered or refused support on random basis, supported group shows greater decrease in those under threat of eviction Before designing monitoring and evaluation approach ask yourself what is proportionate

As a bare minimum, an evaluation should... Move beyond correlation to causation Focus on outcomes as much as outputs Derive uniform, comparable data Combine quantitative and qualitative methods When writing up findings: Include a description of the project Make its research method clear These are my suggestions rather than hard and fast rules!

Primary research techniques Every contact with a member of the public is an opportunity to gather primary data – identify these contact points and then design info gathering tools where possible/proportionate Feedback sheets - Need to know what support was accessed and effect on client circumstance before asking whether they would recommend support - Try to be specific on exactly what about the support was well received - Asking about areas for improvement tells you more than asking what was best received Surveys – face to face versus electronic; Survey Monkey; filter/cohort profile questions + closed/open; pilot first; use community researchers; link to existing surveys/questions Focus groups – should form a second round of primary research to explore initial findings; balance of participants; circulate discussion list; transcribe quotes Case studies – these should be a combination of quantitative and qualitative evidence – powerful human interest approach and a chance to identify lessons learnt, especially regarding process/delivery issues

Secondary research techniques Datasets – many government departments release data covering a wide range of topics at a fairly low spatial level on a regular basis – DOH, DfE, DWP etc. Local authorities sometimes do so too (Ward profiles) but quite a lot of local analytical capacity lost in the cuts – JSNAs a useful repository of local socio-economic data – helpful when identify programme contexts and comparator areas to track Trouble with relying on official data in evaluations – often quite a time lag – e.g. Census 2011 – sometimes suppressed at low levels Databases of ‘what works’ – a few examples on the slide – others include Project Oracle, Dartington and Washington Institute for Public Policy – can use these to make informed assumptions about how effective your Our Place programme will be at getting people into work or tackling issues of ill health etc Early Intervention Foundation (EIF) www.earlyinterventionfoundation.org.uk/evidence National Institute for Clinical Excellence (NICE) http://www.nice.org.uk/#panel6 Coalition for Evidence-Based Policy http://coalition4evidence.org/

More advanced evaluations will include... A review of the delivery process Consideration of Deadweight Longitudinal tracking of clients Value for Money calculations Valuing of outcomes – leading to CBA Strategic added value assessments Action learning approaches Process evaluation – compare to similar projects elsewhere/previously, Deadweight – what would have happened anyway? Control and comparison groups, asking clients, looking at trends - NET Acknowledge that benefits won’t appear immediately or may change over time – track clients VfM – divide outputs/outcomes by delivery cost and if possible compare to benchmarks Next step from VfM is putting a value on outcomes – SROI, CBA SAV – is the project having a wider influence on how policy is developed and delivered locally and nationally Action learning/experimentation – chance to influence things during the project

Task: Develop an evaluation framework for Poplar NCB Refer tables back to Logic Chain and costings exercises

Q&A and next steps 4.45pm Questions from floor

Further information julian. cox@neweconomymanchester. com david Further information julian.cox@neweconomymanchester.com david.morris@neweconomymanchester.com richard.waggot@neweconomymanchester.com Knowledge Hub – Greater Manchester CBA Network

Introduction to Cost Benefit Analysis Tuesday 15 July 2014 Billsley and Yardley Wood Community Centre, Birmingham #ourplaceCBA