Presentation on theme: "Introduction to Cost Evaluation for Child Welfare Programs James Bell, MA James Bell Associates, Inc. June 27, 2011."— Presentation transcript:
1 Introduction to Cost Evaluation for Child Welfare Programs James Bell, MA James Bell Associates, Inc. June 27, 2011
2 Why Evaluate Costs?Cost evaluation informs child welfare program management and policy decision making.For example, future adopters of an evidence-based service model want to know the cost implications of implementing a new program.An informative and rigorous cross-site evaluation is determining the degree to which you made concerted efforts to provide and arrange appropriate services that resulted in improved evidence of child safety, permanent and stable living situations, continued family relationships, and enhanced capacity of families to care for their children’s needs.
3 Today’s AimIncrease attendees’ knowledge of cost evaluation terminology and methods.Increase attendees’ practical ability to design and conduct a cost evaluation of a child welfare program.
4 Today’s Agenda Basic cost evaluation terms and methods. Collection of program-level and case-level service utilization & cost data.Analysis, interpretation, & reporting of program-level and case-level cost data.Questions & open discussion.
5 Basic Terms: Cost and Cost Components Cost is the dollar amount spent providing a service.Total cost is the sum of all direct costs and indirect costs.Other types of cost include time cost, social cost, opportunity cost, etc.
6 Basic Terms: Cost and Cost Components (cont.) Direct costs are those costs that can be assigned to a particular case or program; for example, the monetary value of a case worker’s person-time used to counsel a client.Indirect costs are costs that are not identified specifically with a particular case or program; for example, the monetary value of a case worker’s fringe benefits or the cost of facilities.
7 Basic Terms: Cost and Cost Components (cont.) Both direct and indirect costs are comprised of multiple discreet cost categories, such as labor, facilities, equipment, supplies, equipment, etc.These items are recorded as line items in budgets or accounting/financial reports.For most service programs, labor costs drive total cost and are the cornerstone of analyses.
8 Basic Cost Evaluation Methods Describe/analyze program costs using aggregate data that distinguish cost categories by type of programmatic activity (or phases of program maturation).Describe/analyze program costs using case-level data that distinguish cost categories by type of programmatic activity for each case.Describe/analyze program costs using both aggregate and case-level data.
9 Advanced Cost Analysis Methods Cost-effectiveness analysis compares 2 or more interventions on costs and effectiveness.Effectiveness is measured in client level outcomes (e.g., child maltreatment events).Result: cost per unit of effect (e.g., cost per prevented child maltreatment.)
10 Advanced Cost Analysis Methods (cont.) Cost Utility Analysis is like cost effectiveness analysis, except effectiveness is measured as length of life (e.g., survival) or quality of life such as, “cost per quality-adjusted life year (QALY) saved.”
11 Advanced Cost Analysis Methods (cont.) Cost-Benefit Analysis compares 2 or more interventions on costs and monetized benefits.Benefits are expressed in monetary units, which allows for easier comparison of results across programs with different types of effects.
12 Advanced Cost Analysis Methods (cont.) Return on investment analysis is like cost benefit analysis, except monetized benefits are aggregated over an extended period and costs include all phases of program development, implementation and operation.
14 Why Collect Categorized Aggregate Data? More informative to decision-making than budget reports or accounting system output.Less expensive and burdensome than case-level cost analysis.Complements case-level cost analysis and other more advanced types of cost analysis.
15 Categorizing Aggregate Program Cost Data The amounts in each budgeting and accounting line item are re-distributed into programmatically relevant categories usually based on post-hoc estimates of the use of labor (personnel costs) in a preceding period.For example, personnel (labor) costs associated with delivering services to clients can be divided between activities “directly involving a client” and activities conducted “on behalf of a client”
16 Categorizing Aggregate Program Cost Data (cont.) These two board categories of client-related personnel costs can be further disaggregated into subcategories; for example, distinguishing between “in-take assessment” and “parenting skills enhancement counseling.”Labor costs for administration—personnel not providing front-line services--can be distinguished between client-related and management-related administrative activities.
17 Service Activities Conducted Programmatic Cost Analysis: Personnel Data Elements (Family Connections)Service Activities ConductedWith a Clienton Behalf of a Client· Advocate with client· Advocate without client· Assess· Clinical documentation· Counseling/ support· Research· Provide· Consult/ collaborate· Plan· Locate resources· Refer· Risk management meeting· Teach· Schedule· Staff travel· Transport· Contact attempt
18 Programmatic Cost Analysis: Personnel Data Elements (Family Connections) Administrative Activities Related to Client ActivitiesAdministrative Activities Related to Management Activities· Give supervision· Referral screening· Receive supervision· Outreach/ marketing· Team meeting/ Group consultation· Writing· Provide training· MIS entry· Attend training· Staff organizations meetings/committees· Mentor program technicalassistance· Other administrative activities· Professional/ community· Organization committees· Other administrative activities
19 Why Collect Case-Level Data? Examine cross-case cost variation.Increase measurement precision inside the “black box”.Enable analysis of outcomes in relation to amounts or types of services received (dose-intensity/type analysis.)Improve ability to conduct multivariate statistical analysis; especially if intent-to-treat analysis fails in a randomized study.
20 Identify and Define Program Activity Categories Case-related staff activitiesService activities with a client (face-to-face or phone/ )Service activities on behalf of a client (preparation for or following interaction with or on behalf of client)Activities required to operate the program, but not related to individual clients.Administrative activities.
21 Building Case-Level Costs Relies on counts of the amount of staff time (labor) devoted to each activity by each provider involved with a case.Multiplying the amount(s) of staff time by the “loaded” hourly labor rate (monetary value/dollars) yields “cost per activity.”Sum of per activity costs yields “cost per case.”Loaded hourly labor rate includes base compensation (salary), fringe benefits, and overhead expenses such as facilities, equipment, and general administrative support not assigned to a specific case.
22 Involve Key Programmatic Staff, Including Frontline Caseworkers Invaluable feedback about identifying activity categories and defining/populating categories.Problem-solve efficient data collection processes.Sets foundation for assuring data quality.Builds commitment and buy-in.Helps to increase the validity of the interpretation of findings.
23 Determine the Measurement Timeframe What is the period of time during which cost (and outcome) data are collected?How long will data be collected?Pre-Implementation/Planning Phase?Seasonal component of intervention?Length of time between startup and full capacity?Full intervention period or sampling of intervention period?The time horizon has possible implications (e.g., need for a discount/depreciation rate).
24 Identify and Define Costs PersonnelDetermine how staff time will be trackedFTE (including sick, vacation, and other non-productive time)Time spent working on projectNon-personnelSpace and utilitiesTravelConsumable and non-consumable supplies and equipmentPossible to include donated resourcesVolunteer time and servicesDonated facilities, equipment, supplies
25 Collect Additional Case-Related Information Dates that client started and ended participation in programDemographic characteristicsGenderRace and ethnicityEmployment statusMarital statusEducation statusDate of birthNumber in household
26 Prepare Staff Develop detailed procedures manual for data collection. Provide initial and ongoing training (data collection procedures and data collection form/system.)Conduct a pilot trial period.Develop data collection supervision and support systems (e.g. help line for data collectors to submit questions.)
27 Data Collection Preparation Establish rules for coding activities and allocating staff time increments on activities (minutes/fractions of hours.)Develop a activity tracking form/database (time used, staff ID, client ID, activity code, location.)Develop a reporting planHow frequently should staff submit activity tracking forms?
28 Example: Data Collection Form Establish rules for coding activities and allocating staff time increments on activities (minutes/fractions of hours.)Develop a activity tracking form/database (time used, staff ID, client ID, activity code, location.)Develop a reporting planHow frequently should staff submit activity tracking forms?
29 Data Repository Preparation Develop data entry processComplete paper form and:Key-enter data using database screensScan documents into databaseDirect data entry using agency workstationsData entry through internet connectionsComponent of existing database or stand-alone database?
30 Mode of Visit (1=Face-to-face, 2=Phone, 3=Written, 4=Other) Example: DatabaseStaffCase #DATELoc of VisitMode of Visit (1=Face-to-face, 2=Phone, 3=Written, 4=Other)Advocate WITH ClientAssessCounseling/SupportScheduleFilene100012/4/2006HOMEFACE TO FACE120 20 1001OFFICEPHONE 51230COMM120100912/8/200660205
31 Data Quality Assurance Develop systems and procedures for data quality assurance:Data entry screens with built-in completeness and log-error checksAdditional logic-error checks across casesMonitor cross-provider consistencyCompare service use records with case/clinical notes for completeness (all clients in day) and accuracy (for all services and duration)
32 Data Quality Assurance (cont.) Ensure data collection completeness:To reduce invalid estimates, assign daily deadlinesConduct ongoing supervisory checks of dataExamine data monthly (so not forgotten)Reward staff with highest timely completed rates
33 Common Challenges in Case-Level Data Collection Staff turnoverData collector driftUnexpected influences of local events/contextual factorsMultiple staff and sites/agency locations collecting data
34 Accounting for Overhead & Administrative Costs These costs can not be assigned to a specific case (fringe benefits, space and other forms of infrastructure support and labor associated with functions such as payroll, invoicing, accounts receivable, etc.).Two approaches: 1)use established, existing rates, or 2)create rates for fringe, overhead/general administration.
36 Determine the Measurement/Analysis Perspective Who is the intended audience?Society?Policy makers?Funding agency?Agency implementing the intervention?Other agencies considering implementing the intervention?Intended audience determines which type of economic evaluation is needed and which costs (and benefits) to examine.
37 Example: Total Programmatic Costs by Site and Year
38 Case-Level Cost Analysis A sequence of progressively more sophisticated case-level cost analyses is conducted building from univariate analyses to multivariate analyses.A seminal analyses focuses on the relationship between per-case cost and case outcomes using cost and outcome data and applying multivariate statistical analysis techniques such as repeated ANOVA, hierarchical linear modeling, or general equation modeling.
39 Special Considerations on Conducting Multi-Site Cost Analysis Encourage sites to be involved in all phases based on their interest-each site should receive numerous interim and final products from these analyses.The privacy of individual sites’ data and analyses should be protected according to agreed-upon rules for data sharing (though it should be understood that the case-level cost data analysis is an open, transparent, collaborative multi-site endeavor.)Develop rules with regard to publications, access to data and statistical analysis programming code, and other shared issues.
40 Conduct Service Utilization Analyses Develop and collaboratively review output reports (graphics and printouts) on service activity frequency distributions.Reports should describe the types and amounts of labor expended on each case during the observation period.
41 Example: Total Minutes by Category for One Year
42 Intervention vs. Control by Category in Minutes (One Year)
43 Example: Minutes per Direct Activity (One Year)
44 Example: Minutes per Direct Activity per Staff ID for Case 1
45 Example Display of Service Duration and Intensity (One Year)
46 Obtain and Merge Client & Program Staff Characteristics Data Client characteristic variables enable analyses that compare service utilization and cost patterns across subgroups defined by client characteristics.Staff characteristic variables enable analyses that compare service provision and cost patterns across subgroups defined by staff characteristics.
48 Establish Unit Cost Factors Develop staff member/class unit cost factors or hourly labor rates and overhead rates expressed in dollarsStaff compensationFringe benefitsOverhead expenses
49 Example: Staff Hourly Labor & Fringe RatesStaff IDBase Salary A HourlyBase Salary B HourlyAvg. Annual Base SalaryEmpl. StatusFringe RateAnnual Hourly Total Comp.A30.87“”FT26.8%39.14B24.0424.7624.4030.94C17.4218.2917.8622.64D14.1717.97E17.2521.87F15.2415.7015.46PT(50%)19.61G16.2920.66H19.0724.18I20.8321.4521.1426.81J24.2830.79
50 Remaining Analysis Steps Descriptive Cost AnalysesThese analyses rely on time-based service utilization variables and dollar-based cost variables. The first activity is to calculate total cost per case using the unit cost factors developed earlier. This enables both a summary of costs for all cases and cross-case comparisonsConduct Multivariate Cost AnalysesThese analyses treat case-level cost as the primary dependent variable and incorporate variables, such as client characteristics and service utilization patterns, to investigate predictors of cost variation across cases.
51 Interpretation & Reporting Interpretation of dataInclude program staff in discussion across all stages of analysesReportingGraphic presentationDocument and/or present findings in language that providers and policymakers can understand (i.e., no economic jargon)
53 ReferencesCorso, P. & Filene, JF. (in press). Programmatic cost analysis of interventions to prevent child maltreatment.Corso, P.S., & Lutzker, J.R. (2006). The need for economic analysis in research on child maltreatment. Child Abuse & Neglect, 30,Drummond, M.F., O’Brien, B., Stoddart, G.L., & Torrance, G.W. (1997). Methods for the economic evaluation of health care programs (2nd ed.). Oxford: Oxford University Press.Foster, E.M., Johnson-Shelton, D., & Taylor, T.K. (2007). Measuring time costs in interventions designed to reduce behavior problems among children and youth. American Journal of Community Psychology, 40,Foster, E.M., & Jones, D. (2006). Can a costly intervention be cost-effective? An analysis of violence prevention. Archives of General Psychiatry, 63,Foster, E.M., Porter, M.M., Ayers, T.S., Kaplan, D.L., & Sandler, I. (2007). Estimating the costs of preventative interventions. Evaluation Review, 31,
54 References (cont.)Haddix, A.C., Teutsch, S.M., & Corso, P.S. (Eds). (2003). Prevention effectiveness: A guide to decision analysis and economic evaluation. New York, NY: Oxford University Press.Herman, P.H., Avery, D.J., Schemp, C.S., & Walsh, M.E. (2009) Are cost-inclusive evaluations worth the effort? Evaluation and Program Planning, 32,Hornick, J.P., Paetsch, J.J., & Bertrand, L.D. (2002). A manual on conducting economic analysis of crime prevention programs. National Crime Prevention Center.Levin, H.M., & McEwan, P.J. (2001). Cost-effectiveness analysis (2nd ed.). Thousand Oaks, CA: Sage Publications.NIDA ManualYates, B.T. (2009). Cost-inclusive evaluation: A banquet of approaches for including costs, benefits, and cost-effectiveness and cost-benefit analyses in your next evaluation. Evaluation and Program Planning, 32,
55 Follow-up Questions?Contact Information:James Bell