Presentation on theme: "Introduction to Cost Benefit Analysis"— Presentation transcript:
1Introduction to Cost Benefit Analysis Tuesday 15 July 2014 Billsley and Yardley Wood Community Centre, Birmingham#ourplaceCBA
2Please familiarise yourself with the emergency evacuation procedures Emergency proceduresPlease familiarise yourself with theemergency evacuation procedureslocated within the inside cover of yourdelegate handbook.
7What do we mean by costs and benefits? All additional costs needed to deliver projectFiscalSocialEconomicFiscalEconomicSocial
8The CBA model – costs Cost Benefit Analysis Tool Costs Economic Case CapitalRevenueIn KindEconomic CaseNet Present Public ValueValue for Money Benefit Cost RatioCost Benefit Analysis ToolOutcomesNeedEngagementImpactDeadweightLag and drop-offFinancial CaseNet Present Budget ImpactFinancial return on investmentPayback periodCashability assumptions
9Approach to calculating costs What types of costs are borne?Which agencies bear costs?When are costs borne?
10The CBA model – outcomes CostsCapitalRevenueIn KindEconomic CaseNet Present Public ValueValue for Money Benefit Cost RatioCost Benefit Analysis ToolOutcomesNeedEngagementImpactDeadweightLag and drop-offFinancial CaseNet Present Budget ImpactFinancial return on investmentPayback periodCashability assumptions
11Types of benefits Example Fiscal benefits Public Value benefits Employment mentoring programme for people with mental health problemsReduction in unemployment payments as individuals gain employmentIncreased output resulting from increased employmentImproved health with related impacts on well-being (e.g. confidence and self esteem)Programme to tackle antisocial behaviourReduction in police, housing and local authority time spent responding to incidents.Opportunity cost of avoided time spent by public sector agencies.Increased patronage of local businesses resulting in net growth in local economy once displacement has been taken into account.Reduced fear of crime amongst local residents.Drug treatment programmeSavings in reactive health and criminal justice costs - emergency hospital visits, long term health costs, responses to crime.Improved health and life expectancy of individual
12Turning outcomes into benefits Fiscal benefitEcon benefitSocial benefitGetting someone into a job (per annum)£9,800£14,044Gaining a Level 3 apprenticeship (per annum)£1,391£1,925Avoiding a child being taken into care (per annum)£63,362£7,460Reduced incident of crime (per incident)£609£676£1,648Reduced Isolation (full distance travelled)£8,500
13Running the CBA model Assumptions tested/updated via evaluation At risk /affected population%EngagedValueHow many potential beneficiaries?How many will we reach?How many will achieve desired outcome?What would have occurred anyway?What is the value of the desired outcome?%Impact1 - %optimism biasHow confident are we in the evidence?%Deadweight%RetainedHow many will complete the programme?Assumptions tested/updated via evaluationOptimism Bias (OB) correction applied to dataThe Excel model follows a sequential process, by which we translate gross outcomes (i.e. what the pilot has achieved) into net outcomes (i.e. what % of this achievement is additional?) to net value (how much is this additional achievement worth to the taxpayer?)Models often combine primary data with secondary evidence and assumptions:We have primary evidence from pilots regarding target populations (for example, number of households within the pilot area)We are having to make assumptions on what % of this population we will engage and what percentage will achieve the desired outcomeWe are using national research to place a value on achieving a specific outcome (for example, the value of fewer evictions from RSL properties)As projects feed primary data back to us we will be able to replace assumptions with actual evidence of how many people we are engaging and how many we are impacting. Thus, the model will become more accurate over time.Ways of calculating deadweight – control groups, comparator areas, baseline forecasts, similar interventions, primary research etc
14Case study example - benefits Employment mentoring programme for people with low level mental health problemsKey outcomes:Employment – moving off Employment Support AllowanceImproved mental healthPotential other outcomesReduced alcohol dependencyWhat questions do I need to ask to understand the benefits of improved outcomes?Is the outcome relevant to the whole cohort (e.g. employment) or just a proportion (e.g. alcohol dependency)What percentage of the cohort can we engage with?How many will remain engaged for the whole programme?Do we have evidence of impact from local pilot schemes?Can we find evidence from elsewhere in the UK or abroad?Do we have data on the success rates of the Business as Usual provision (e.g. Work Programme)?Does research evidence indicate what would happen anyway?How soon will benefits start?How long will they last for?
15The CBA model – outputs Cost Benefit Analysis Tool Costs Economic Case CapitalRevenueIn KindEconomic CaseNet Present Public ValueValue for Money Benefit Cost RatioCost Benefit Analysis ToolOutcomesNeedEngagementImpactDeadweightLag and drop-offFinancial CaseNet Present Budget ImpactFinancial return on investmentPayback periodCashability assumptions
17Why should areas use CBA? Localities can use CBA to deliver:Informed decision-makingCost-effective decision-makingEquitable decision-making
18Refreshments15 min break - refreshments available in conservatory – restart at 11.30am
19Exercise 1: Agreeing outcomes Undertaking CBA is not simple – it requires both an understanding of how to model things and the data and information that you are modellingRest of today is going to focus on the pre-CBA steps that if undertaken will mean that Our Place areas are well placed to begin CBA later in the yearWe encourage local teams to draw up a ‘to do’ list based upon what they learn from remaining sections of the workshopPre-step 1 – confirming exactly what you are modelling
20Has Area X’s project been successful? If somebody asks you ‘was that project or programme successful?’ you should answer ‘successful at what?’As an evaluator, before you start any research you need to be clear on:What a project is hoping to change – what are the underlying issues, evidence of these?What does the project involve – key activities. Which actions will affect which indicators?How it is to be judged – what indicators do we expect to see change?How much are we spending? What would represent VfM?Who is meant to benefit – are their target groups? In target areas?At what point in time it needs to judged – will impact be immediate or longer-term?What of relevance is happening in the wider policy environment?By asking for answers to these questions – build up a logical, rational evaluation framework – not judging a project unfairly, not wasting research input on unrelated issuesLogic Chain a useful tool in this respect
21Contextual conditions Project:Intended ImpactsContextual conditionsKey policy conditionsIntended OutcomesProgramme ObjectivesRationaleTalk through the steps in the Logic Chain, explaining how each linksInputsActivitiesOutputs
22Project: Greater Manchester Commitment Intended ImpactsReduced youth unemployment and NEET ratesImproved labour force skills & productivityReduction in employer skills shortages and staff turnover ratesContextual conditions High rates of youth unemployment and inactivity vs. other groups Issue has been getting worse since start of economic downturn, esp. GM Limited employer engagement/satisfaction with training marketKey policy conditionsNational policy – Youth Contract, provides grants, work placements, advice and BIS Skills for Sustainable Growth strategyLocal policy – each authority has their own scheme, FJF legacyIntended OutcomesSustained employment for year oldsMore GM providers working with cohortMore GM employers offering apprenticeshipsMore awareness of opportunitiesBetter integrated systemDraw down more central government expenditureProgramme ObjectivesIncrease employability and employment rate of GM year oldsIncrease willingness of training providers to work with and willingness of employers to recruit from this groupRationaleFinancial incentives workCost < JSASingle GM systemTraining ethosInputs£4.3m3 FTEsLocal authoritiesTraining providersActivitiesMarketingAdministrationGrant makingWebsiteBrokering vacanciesOutputsVacanciesGrant paymentsAGE paymentsNet outputsLeakageDeadweightDisplacement
23Task: Develop a Logic Chain for Poplar NCB case study Each table will have an A3 copy of the Logic Chain and handouts explaining the Poplar NCB in more detail – using the latter, fill in the former – for some of the boxes the tables will have to be creative
24Prompts What does the intervention involve? How will support be delivered?Where will it be operating? And when?How many people will be eligible for the project?What outcomes does the project aim to deliver?Has baseline work been undertaken?Which agencies are involved in the project?
26Lunch and networking45 min break – restart promptly at 1.45pm when we will be looking at the issues of calculating costs and tracking impact
27Exercise 2: Calculating your delivery costs You cannot produce a CBA if you cannot state how much is being spent on a programme – surprisingly large number of public programmes do not have a firm handle on their delivery costsIn this section of the workshop we will show attendees the technique we use to help programme teams work out their delivery costs
28Approach to calculating costs What types of costs are borne?Which agencies bear costs?When are costs borne?
29Case study example – building up costs Integrated multi-agency safeguarding programme (Children’s Services and Domestic Abuse)Cross-agency key worker modelCo-location of staffBetter identification, sequencing and availability of interventionsWhat are the relative costs compared to business as usual?Potential increasesPotential efficiency savingsNew posts (key workers, supervision)Staff time – avoidance of duplication of effortTraining costsPremises costsIncreased referrals to other agencies
30Task: Identify costs associated with Poplar NCB case study
31Refreshments15 min break – refreshments available in conservatory - restart at 3.30pm
32Exercise 3: Tracking your impact From exercise 2 we have the basis for working out Our Place programme costs.To complete a CBA we need data and intelligence regarding the impact of Our Place programmes upon specific outcomes – the outcomes that we identified in the Logic Chain exercise.We could rely solely on secondary evidence of what other programmes have achieved plus some assumptions/informed guesses but in our CBA model this would result in a high optimism bias ratingBetter approach is to ensure that primary data is being collected from the outset of the programme – going to look at research techniques that Our Place areas can use to gather such data
33Maryland Scientific Method Scale LevelCriteria1Correlation – Our Place clients less likely to be under threat of eviction than non-clients2Before & after – number of clients under threat of eviction fell after Our Place support3Single control - number of clients under threat of eviction fell after Our Place support and same fall not seen in another group4Multiple controls - number of clients, by gender/race/education level etc, under threat of eviction fell after Our Place support and same fall not seen in same groups elsewhere5Randomisation – Our Place team offered or refused support on random basis, supported group shows greater decrease in those under threat of evictionBefore designing monitoring and evaluation approach ask yourself what is proportionate
34As a bare minimum, an evaluation should... Move beyond correlation to causationFocus on outcomes as much as outputsDerive uniform, comparable dataCombine quantitative and qualitative methodsWhen writing up findings:Include a description of the projectMake its research method clearThese are my suggestions rather than hard and fast rules!
35Primary research techniques Every contact with a member of the public is an opportunity to gather primary data – identify these contact points and then design info gathering tools where possible/proportionateFeedback sheets - Need to know what support was accessed and effect on client circumstance before asking whether they would recommend support - Try to be specific on exactly what about the support was well received - Asking about areas for improvement tells you more than asking what was best receivedSurveys – face to face versus electronic; Survey Monkey; filter/cohort profile questions + closed/open; pilot first; use community researchers; link to existing surveys/questionsFocus groups – should form a second round of primary research to explore initial findings; balance of participants; circulate discussion list; transcribe quotesCase studies – these should be a combination of quantitative and qualitative evidence – powerful human interest approach and a chance to identify lessons learnt, especially regarding process/delivery issues
36Secondary research techniques Datasets – many government departments release data covering a wide range of topics at a fairly low spatial level on a regular basis – DOH, DfE, DWP etc. Local authorities sometimes do so too (Ward profiles) but quite a lot of local analytical capacity lost in the cuts – JSNAs a useful repository of local socio-economic data – helpful when identify programme contexts and comparator areas to trackTrouble with relying on official data in evaluations – often quite a time lag – e.g. Census 2011 – sometimes suppressed at low levelsDatabases of ‘what works’ – a few examples on the slide – others include Project Oracle, Dartington and Washington Institute for Public Policy – can use these to make informed assumptions about how effective your Our Place programme will be at getting people into work or tackling issues of ill health etcEarly Intervention Foundation (EIF)National Institute for Clinical Excellence (NICE)Coalition for Evidence-Based Policy
37More advanced evaluations will include... A review of the delivery processConsideration of DeadweightLongitudinal tracking of clientsValue for Money calculationsValuing of outcomes – leading to CBAStrategic added value assessmentsAction learning approachesProcess evaluation – compare to similar projects elsewhere/previously,Deadweight – what would have happened anyway? Control and comparison groups, asking clients, looking at trends - NETAcknowledge that benefits won’t appear immediately or may change over time – track clientsVfM – divide outputs/outcomes by delivery cost and if possible compare to benchmarksNext step from VfM is putting a value on outcomes – SROI, CBASAV – is the project having a wider influence on how policy is developed and delivered locally and nationallyAction learning/experimentation – chance to influence things during the project
38Task: Develop an evaluation framework for Poplar NCB Refer tables back to Logic Chain and costings exercises