Presentation is loading. Please wait.

Presentation is loading. Please wait.

Introduction to Cost Evaluation for Child Welfare Programs James Bell, MA James Bell Associates, Inc. June 27, 2011.

Similar presentations


Presentation on theme: "Introduction to Cost Evaluation for Child Welfare Programs James Bell, MA James Bell Associates, Inc. June 27, 2011."— Presentation transcript:

1 Introduction to Cost Evaluation for Child Welfare Programs James Bell, MA James Bell Associates, Inc. June 27, 2011

2 Why Evaluate Costs? Cost evaluation informs child welfare program management and policy decision making. Cost evaluation informs child welfare program management and policy decision making. For example, future adopters of an evidence-based service model want to know the cost implications of implementing a new program. For example, future adopters of an evidence-based service model want to know the cost implications of implementing a new program. 2

3 Todays Aim Increase attendees knowledge of cost evaluation terminology and methods. Increase attendees knowledge of cost evaluation terminology and methods. Increase attendees practical ability to design and conduct a cost evaluation of a child welfare program. Increase attendees practical ability to design and conduct a cost evaluation of a child welfare program. 3

4 Todays Agenda Basic cost evaluation terms and methods. Basic cost evaluation terms and methods. Collection of program-level and case-level service utilization & cost data. Collection of program-level and case-level service utilization & cost data. Analysis, interpretation, & reporting of program-level and case-level cost data. Analysis, interpretation, & reporting of program-level and case-level cost data. Questions & open discussion. Questions & open discussion. 4

5 Basic Terms: Cost and Cost Components Cost is the dollar amount spent providing a service. Cost is the dollar amount spent providing a service. Total cost is the sum of all direct costs and indirect costs. Total cost is the sum of all direct costs and indirect costs. Other types of cost include time cost, social cost, opportunity cost, etc. Other types of cost include time cost, social cost, opportunity cost, etc. 5

6 Direct costs are those costs that can be assigned to a particular case or program; for example, the monetary value of a case workers person-time used to counsel a client. Direct costs are those costs that can be assigned to a particular case or program; for example, the monetary value of a case workers person-time used to counsel a client. Indirect costs are costs that are not identified specifically with a particular case or program; for example, the monetary value of a case workers fringe benefits or the cost of facilities. Indirect costs are costs that are not identified specifically with a particular case or program; for example, the monetary value of a case workers fringe benefits or the cost of facilities. Basic Terms: Cost and Cost Components (cont.) 6

7 Both direct and indirect costs are comprised of multiple discreet cost categories, such as labor, facilities, equipment, supplies, equipment, etc. Both direct and indirect costs are comprised of multiple discreet cost categories, such as labor, facilities, equipment, supplies, equipment, etc. These items are recorded as line items in budgets or accounting/financial reports. These items are recorded as line items in budgets or accounting/financial reports. For most service programs, labor costs drive total cost and are the cornerstone of analyses. For most service programs, labor costs drive total cost and are the cornerstone of analyses. Basic Terms: Cost and Cost Components (cont.) 7

8 Describe/analyze program costs using aggregate data that distinguish cost categories by type of programmatic activity (or phases of program maturation). Describe/analyze program costs using aggregate data that distinguish cost categories by type of programmatic activity (or phases of program maturation). Describe/analyze program costs using case-level data that distinguish cost categories by type of programmatic activity for each case. Describe/analyze program costs using case-level data that distinguish cost categories by type of programmatic activity for each case. Describe/analyze program costs using both aggregate and case-level data. Describe/analyze program costs using both aggregate and case-level data. Basic Cost Evaluation Methods 8

9 Cost-effectiveness analysis compares 2 or more interventions on costs and effectiveness. Cost-effectiveness analysis compares 2 or more interventions on costs and effectiveness. Effectiveness is measured in client level outcomes (e.g., child maltreatment events). Effectiveness is measured in client level outcomes (e.g., child maltreatment events). Result: cost per unit of effect (e.g., cost per prevented child maltreatment.) Result: cost per unit of effect (e.g., cost per prevented child maltreatment.) Advanced Cost Analysis Methods 9

10 Cost Utility Analysis is like cost effectiveness analysis, except effectiveness is measured as length of life (e.g., survival) or quality of life such as, cost per quality-adjusted life year (QALY) saved. Cost Utility Analysis is like cost effectiveness analysis, except effectiveness is measured as length of life (e.g., survival) or quality of life such as, cost per quality-adjusted life year (QALY) saved. Advanced Cost Analysis Methods (cont.) 10

11 Cost-Benefit Analysis compares 2 or more interventions on costs and monetized benefits. Cost-Benefit Analysis compares 2 or more interventions on costs and monetized benefits. Benefits are expressed in monetary units, which allows for easier comparison of results across programs with different types of effects. Benefits are expressed in monetary units, which allows for easier comparison of results across programs with different types of effects. Advanced Cost Analysis Methods (cont.) 11

12 Return on investment analysis is like cost benefit analysis, except monetized benefits are aggregated over an extended period and costs include all phases of program development, implementation and operation. Return on investment analysis is like cost benefit analysis, except monetized benefits are aggregated over an extended period and costs include all phases of program development, implementation and operation. Advanced Cost Analysis Methods (cont.) 12

13 DATA COLLECTION Phase 1 13

14 More informative to decision-making than budget reports or accounting system output. More informative to decision-making than budget reports or accounting system output. Less expensive and burdensome than case- level cost analysis. Less expensive and burdensome than case- level cost analysis. Complements case-level cost analysis and other more advanced types of cost analysis. Complements case-level cost analysis and other more advanced types of cost analysis. Why Collect Categorized Aggregate Data? 14

15 The amounts in each budgeting and accounting line item are re-distributed into programmatically relevant categories usually based on post-hoc estimates of the use of labor (personnel costs) in a preceding period. The amounts in each budgeting and accounting line item are re-distributed into programmatically relevant categories usually based on post-hoc estimates of the use of labor (personnel costs) in a preceding period. For example, personnel (labor) costs associated with delivering services to clients can be divided between activities directly involving a client and activities conducted on behalf of a client For example, personnel (labor) costs associated with delivering services to clients can be divided between activities directly involving a client and activities conducted on behalf of a client Categorizing Aggregate Program Cost Data 15

16 These two board categories of client-related personnel costs can be further disaggregated into subcategories; for example, distinguishing between in-take assessment and parenting skills enhancement counseling. These two board categories of client-related personnel costs can be further disaggregated into subcategories; for example, distinguishing between in-take assessment and parenting skills enhancement counseling. Labor costs for administrationpersonnel not providing front-line services--can be distinguished between client-related and management-related administrative activities. Labor costs for administrationpersonnel not providing front-line services--can be distinguished between client-related and management-related administrative activities. Categorizing Aggregate Program Cost Data (cont.) 16

17 Service Activities Conducted With a Client Service Activities Conducted on Behalf of a Client · Advocate with client· Advocate without client · Assess· Clinical documentation · Counseling/ support· Research · Provide· Consult/ collaborate · Plan· Locate resources · Refer· Risk management meeting · Teach· Schedule · Staff travel · Transport · Contact attempt Programmatic Cost Analysis: Personnel Data Elements (Family Connections) 17

18 Administrative Activities Related to Client Activities Administrative Activities Related to Management Activities · Give supervision· Referral screening · Receive supervision· Outreach/ marketing · Team meeting/ Group consultation· Writing · Provide training· MIS entry · Attend training· Staff organizations meetings/ committees · Mentor program technical assistance · Other administrative activities · Professional/ community committees · Organization committees · Other administrative activities Programmatic Cost Analysis: Personnel Data Elements (Family Connections) 18

19 Examine cross-case cost variation. Examine cross-case cost variation. Increase measurement precision inside the black box. Increase measurement precision inside the black box. Enable analysis of outcomes in relation to amounts or types of services received (dose-intensity/type analysis.) Enable analysis of outcomes in relation to amounts or types of services received (dose-intensity/type analysis.) Improve ability to conduct multivariate statistical analysis; especially if intent-to-treat analysis fails in a randomized study. Improve ability to conduct multivariate statistical analysis; especially if intent-to-treat analysis fails in a randomized study. Why Collect Case-Level Data? 19

20 Case-related staff activities Case-related staff activities –Service activities with a client (face-to-face or phone/ ) –Service activities on behalf of a client (preparation for or following interaction with or on behalf of client) Activities required to operate the program, but not related to individual clients. Activities required to operate the program, but not related to individual clients. Administrative activities. Administrative activities. Identify and Define Program Activity Categories 20

21 Relies on counts of the amount of staff time (labor) devoted to each activity by each provider involved with a case. Relies on counts of the amount of staff time (labor) devoted to each activity by each provider involved with a case. Multiplying the amount(s) of staff time by the loaded hourly labor rate (monetary value/dollars) yields cost per activity. Multiplying the amount(s) of staff time by the loaded hourly labor rate (monetary value/dollars) yields cost per activity. Sum of per activity costs yields cost per case. Sum of per activity costs yields cost per case. Loaded hourly labor rate includes base compensation (salary), fringe benefits, and overhead expenses such as facilities, equipment, and general administrative support not assigned to a specific case. Loaded hourly labor rate includes base compensation (salary), fringe benefits, and overhead expenses such as facilities, equipment, and general administrative support not assigned to a specific case. Building Case-Level Costs 21

22 Invaluable feedback about identifying activity categories and defining/populating categories. Invaluable feedback about identifying activity categories and defining/populating categories. Problem-solve efficient data collection processes. Problem-solve efficient data collection processes. Sets foundation for assuring data quality. Sets foundation for assuring data quality. Builds commitment and buy-in. Builds commitment and buy-in. Helps to increase the validity of the interpretation of findings. Helps to increase the validity of the interpretation of findings. Involve Key Programmatic Staff, Including Frontline Caseworkers 22

23 What is the period of time during which cost (and outcome) data are collected? What is the period of time during which cost (and outcome) data are collected? How long will data be collected? How long will data be collected? –Pre-Implementation/Planning Phase? –Seasonal component of intervention? –Length of time between startup and full capacity? –Full intervention period or sampling of intervention period? The time horizon has possible implications (e.g., need for a discount/depreciation rate). The time horizon has possible implications (e.g., need for a discount/depreciation rate). Determine the Measurement Timeframe 23

24 Personnel Personnel Determine how staff time will be tracked Determine how staff time will be tracked –FTE (including sick, vacation, and other non-productive time) –Time spent working on project Non-personnel Non-personnel Space and utilities Space and utilities Travel Travel Consumable and non-consumable supplies and equipment Consumable and non-consumable supplies and equipment Possible to include donated resources Possible to include donated resources Volunteer time and services Volunteer time and services Donated facilities, equipment, supplies Donated facilities, equipment, supplies Identify and Define Costs 24

25 Dates that client started and ended participation in program Dates that client started and ended participation in program Demographic characteristics Demographic characteristics –Gender –Race and ethnicity –Employment status –Marital status –Education status –Date of birth –Number in household Collect Additional Case- Related Information 25

26 Develop detailed procedures manual for data collection. Develop detailed procedures manual for data collection. Provide initial and ongoing training (data collection procedures and data collection form/system.) Provide initial and ongoing training (data collection procedures and data collection form/system.) Conduct a pilot trial period. Conduct a pilot trial period. Develop data collection supervision and support systems (e.g. help line for data collectors to submit questions.) Develop data collection supervision and support systems (e.g. help line for data collectors to submit questions.) Prepare Staff 26

27 Establish rules for coding activities and allocating staff time increments on activities (minutes/fractions of hours.)Establish rules for coding activities and allocating staff time increments on activities (minutes/fractions of hours.) Develop a activity tracking form/database ( time used, staff ID, client ID, activity code, location.) Develop a activity tracking form/database ( time used, staff ID, client ID, activity code, location.) Develop a reporting plan Develop a reporting plan –How frequently should staff submit activity tracking forms? Data Collection Preparation 27

28 Establish rules for coding activities and allocating staff time increments on activities (minutes/fractions of hours.)Establish rules for coding activities and allocating staff time increments on activities (minutes/fractions of hours.) Develop a activity tracking form/database ( time used, staff ID, client ID, activity code, location.) Develop a activity tracking form/database ( time used, staff ID, client ID, activity code, location.) Develop a reporting plan Develop a reporting plan –How frequently should staff submit activity tracking forms? Example: Data Collection Form 28

29 Develop data entry process Develop data entry process Complete paper form and: Complete paper form and: –Key-enter data using database screens –Scan documents into database Direct data entry using agency workstations Direct data entry using agency workstations Data entry through internet connections Data entry through internet connections Component of existing database or stand- alone database? Component of existing database or stand- alone database? Data Repository Preparation 29

30 Example: Database StaffCase #DATE Loc of Visit Mode of Visit (1=Face-to-face, 2=Phone, 3=Written, 4=Other) Advocate WITH Client Assess Counseling/ Support Schedule Filene100012/4/2006HOMEFACE TO FACE Filene100112/4/2006OFFICEPHONE 5 Filene123012/4/2006COMMFACE TO FACE120 Filene100912/8/2006HOMEFACE TO FACE

31 Develop systems and procedures for data quality assurance: Develop systems and procedures for data quality assurance: –Data entry screens with built-in completeness and log-error checks –Additional logic-error checks across cases –Monitor cross-provider consistency –Compare service use records with case/clinical notes for completeness (all clients in day) and accuracy (for all services and duration) Data Quality Assurance 31

32 Ensure data collection completeness: Ensure data collection completeness: –To reduce invalid estimates, assign daily deadlines –Conduct ongoing supervisory checks of data –Examine data monthly (so not forgotten) –Reward staff with highest timely completed rates Data Quality Assurance (cont.) 32

33 Staff turnover Staff turnover Data collector drift Data collector drift Unexpected influences of local events/contextual factors Unexpected influences of local events/contextual factors Multiple staff and sites/agency locations collecting data Multiple staff and sites/agency locations collecting data Common Challenges in Case-Level Data Collection 33

34 These costs can not be assigned to a specific case (fringe benefits, space and other forms of infrastructure support and labor associated with functions such as payroll, invoicing, accounts receivable, etc.). These costs can not be assigned to a specific case (fringe benefits, space and other forms of infrastructure support and labor associated with functions such as payroll, invoicing, accounts receivable, etc.). Two approaches: 1)use established, existing rates, or 2)create rates for fringe, overhead/general administration. Two approaches: 1)use established, existing rates, or 2)create rates for fringe, overhead/general administration. Accounting for Overhead & Administrative Costs 34

35 Analylsis & Reporting Phase 2 Phase 2 35

36 Who is the intended audience? Who is the intended audience? –Society? –Policy makers? –Funding agency? –Agency implementing the intervention? –Other agencies considering implementing the intervention? Intended audience determines which type of economic evaluation is needed and which costs (and benefits) to examine. Intended audience determines which type of economic evaluation is needed and which costs (and benefits) to examine. Determine the Measurement/Analysis Perspective 36

37 Example: Total Programmatic Costs by Site and Year 37

38 A sequence of progressively more sophisticated case-level cost analyses is conducted building from univariate analyses to multivariate analyses. A sequence of progressively more sophisticated case-level cost analyses is conducted building from univariate analyses to multivariate analyses. A seminal analyses focuses on the relationship between per-case cost and case outcomes using cost and outcome data and applying multivariate statistical analysis techniques such as repeated ANOVA, hierarchical linear modeling, or general equation modeling. A seminal analyses focuses on the relationship between per-case cost and case outcomes using cost and outcome data and applying multivariate statistical analysis techniques such as repeated ANOVA, hierarchical linear modeling, or general equation modeling. Case-Level Cost Analysis 38

39 Encourage sites to be involved in all phases based on their interest-each site should receive numerous interim and final products from these analyses. Encourage sites to be involved in all phases based on their interest-each site should receive numerous interim and final products from these analyses. The privacy of individual sites data and analyses should be protected according to agreed-upon rules for data sharing (though it should be understood that the case-level cost data analysis is an open, transparent, collaborative multi- site endeavor.) The privacy of individual sites data and analyses should be protected according to agreed-upon rules for data sharing (though it should be understood that the case-level cost data analysis is an open, transparent, collaborative multi- site endeavor.) Develop rules with regard to publications, access to data and statistical analysis programming code, and other shared issues. Develop rules with regard to publications, access to data and statistical analysis programming code, and other shared issues. Special Considerations on Conducting Multi-Site Cost Analysis 39

40 Develop and collaboratively review output reports (graphics and printouts) on service activity frequency distributions. Develop and collaboratively review output reports (graphics and printouts) on service activity frequency distributions. Reports should describe the types and amounts of labor expended on each case during the observation period. Reports should describe the types and amounts of labor expended on each case during the observation period. Conduct Service Utilization Analyses 40

41 Example: Total Minutes by Category for One Year 41

42 Intervention vs. Control by Category in Minutes (One Year) 42

43 Example: Minutes per Direct Activity (One Year) 43

44 Example: Minutes per Direct Activity per Staff ID for Case 1 44

45 Example Display of Service Duration and Intensity (One Year) 45

46 Client characteristic variables enable analyses that compare service utilization and cost patterns across subgroups defined by client characteristics. Client characteristic variables enable analyses that compare service utilization and cost patterns across subgroups defined by client characteristics. Staff characteristic variables enable analyses that compare service provision and cost patterns across subgroups defined by staff characteristics. Staff characteristic variables enable analyses that compare service provision and cost patterns across subgroups defined by staff characteristics. Obtain and Merge Client & Program Staff Characteristics Data 46

47 Example: Staff Characteristics Staff IDGenderRace/EthnEducationDegree Years Exp. in SW FieldPosition Title AFWhiteMastersMSW, LMSW16-20Program Manager BFWhiteMastersMSW, LCSW16-20 Program Coordinator CFWhiteMastersM.Ed Counseling, LPC, LMFT2-5Staff Clinician DFWhiteCollege GradBS Psychology2-5Case Manager EFBlack/HispanicMastersMS Counseling, LPC Intern2-5Staff Clinician FFBlack/AACollege GradBSW, LBSW> 20Case Manager GFWhiteMastersMA Counseling, LPC, NCC, RPT2-5Staff Clinician HFWhiteMastersMSW, LMSW, MPH2-5Process Evaluator IFOtherMastersMA Sociology11-15Outcome Evaluator JFWhitePh.DDrPH Behavioral Health11-15 Evaluation Supervisor 47

48 Develop staff member/class unit cost factors or hourly labor rates and overhead rates expressed in dollars Develop staff member/class unit cost factors or hourly labor rates and overhead rates expressed in dollars –Staff compensation –Fringe benefits –Overhead expenses Establish Unit Cost Factors 48

49 Example: Staff Hourly Labor & Fringe Rates Staff ID Base Salary A Hourly Base Salary B Hourly Avg. Annual Base Salary Empl. Status Fringe Rate Annual Hourly Total Comp. A FT26.8%39.14 B FT26.8%30.94 C FT26.8%22.64 D FT26.8%17.97 E FT26.8%21.87 F PT(50% )26.8%19.61 G FT26.8%20.66 H FT26.8%24.18 I PT(50% )26.8%26.81 J FT26.8%

50 Descriptive Cost Analyses Descriptive Cost Analyses –These analyses rely on time-based service utilization variables and dollar-based cost variables. The first activity is to calculate total cost per case using the unit cost factors developed earlier. This enables both a summary of costs for all cases and cross-case comparisons Conduct Multivariate Cost Analyses Conduct Multivariate Cost Analyses –These analyses treat case-level cost as the primary dependent variable and incorporate variables, such as client characteristics and service utilization patterns, to investigate predictors of cost variation across cases. Remaining Analysis Steps 50

51 Interpretation of data Interpretation of data –Include program staff in discussion across all stages of analyses Reporting Reporting –Graphic presentation –Document and/or present findings in language that providers and policymakers can understand (i.e., no economic jargon) Interpretation & Reporting 51

52 Open Discussion 52

53 Corso, P. & Filene, JF. (in press). Programmatic cost analysis of interventions to prevent child maltreatment. Corso, P. & Filene, JF. (in press). Programmatic cost analysis of interventions to prevent child maltreatment. Corso, P.S., & Lutzker, J.R. (2006). The need for economic analysis in research on child maltreatment. Child Abuse & Neglect, 30, Corso, P.S., & Lutzker, J.R. (2006). The need for economic analysis in research on child maltreatment. Child Abuse & Neglect, 30, Drummond, M.F., OBrien, B., Stoddart, G.L., & Torrance, G.W. (1997). Methods for the economic evaluation of health care programs (2 nd ed.). Oxford: Oxford University Press. Drummond, M.F., OBrien, B., Stoddart, G.L., & Torrance, G.W. (1997). Methods for the economic evaluation of health care programs (2 nd ed.). Oxford: Oxford University Press. Foster, E.M., Johnson-Shelton, D., & Taylor, T.K. (2007). Measuring time costs in interventions designed to reduce behavior problems among children and youth. American Journal of Community Psychology, 40, Foster, E.M., Johnson-Shelton, D., & Taylor, T.K. (2007). Measuring time costs in interventions designed to reduce behavior problems among children and youth. American Journal of Community Psychology, 40, Foster, E.M., & Jones, D. (2006). Can a costly intervention be cost- effective? An analysis of violence prevention. Archives of General Psychiatry, 63, Foster, E.M., & Jones, D. (2006). Can a costly intervention be cost- effective? An analysis of violence prevention. Archives of General Psychiatry, 63, Foster, E.M., Porter, M.M., Ayers, T.S., Kaplan, D.L., & Sandler, I. (2007). Estimating the costs of preventative interventions. Evaluation Review, 31, Foster, E.M., Porter, M.M., Ayers, T.S., Kaplan, D.L., & Sandler, I. (2007). Estimating the costs of preventative interventions. Evaluation Review, 31, References 53

54 Haddix, A.C., Teutsch, S.M., & Corso, P.S. (Eds). (2003). Prevention effectiveness: A guide to decision analysis and economic evaluation. New York, NY: Oxford University Press. Haddix, A.C., Teutsch, S.M., & Corso, P.S. (Eds). (2003). Prevention effectiveness: A guide to decision analysis and economic evaluation. New York, NY: Oxford University Press. Herman, P.H., Avery, D.J., Schemp, C.S., & Walsh, M.E. (2009) Are cost- inclusive evaluations worth the effort? Evaluation and Program Planning, 32, Herman, P.H., Avery, D.J., Schemp, C.S., & Walsh, M.E. (2009) Are cost- inclusive evaluations worth the effort? Evaluation and Program Planning, 32, Hornick, J.P., Paetsch, J.J., & Bertrand, L.D. (2002). A manual on conducting economic analysis of crime prevention programs. National Crime Prevention Center. Hornick, J.P., Paetsch, J.J., & Bertrand, L.D. (2002). A manual on conducting economic analysis of crime prevention programs. National Crime Prevention Center. Levin, H.M., & McEwan, P.J. (2001). Cost-effectiveness analysis (2 nd ed.). Thousand Oaks, CA: Sage Publications. Levin, H.M., & McEwan, P.J. (2001). Cost-effectiveness analysis (2 nd ed.). Thousand Oaks, CA: Sage Publications. NIDA Manual NIDA Manual Yates, B.T. (2009). Cost-inclusive evaluation: A banquet of approaches for including costs, benefits, and cost-effectiveness and cost-benefit analyses in your next evaluation. Evaluation and Program Planning, 32, Yates, B.T. (2009). Cost-inclusive evaluation: A banquet of approaches for including costs, benefits, and cost-effectiveness and cost-benefit analyses in your next evaluation. Evaluation and Program Planning, 32, References (cont.) 54

55 Contact Information: James Bell Follow-up Questions? 55


Download ppt "Introduction to Cost Evaluation for Child Welfare Programs James Bell, MA James Bell Associates, Inc. June 27, 2011."

Similar presentations


Ads by Google