1. Introduction to Cost Evaluation for Child Welfare Programs James Bell, MA James Bell Associates, Inc. June 27, 2011

2. Why Evaluate Costs?
Cost evaluation informs child welfare program management and policy decision making.
For example, future adopters of an evidence-based service model want to know the cost implications of implementing a new program.
An informative and rigorous cross-site evaluation is determining the degree to which you made concerted efforts to provide and arrange appropriate services that resulted in improved evidence of child safety, permanent and stable living situations, continued family relationships, and enhanced capacity of families to care for their children’s needs.

3. Today’s Aim
Increase attendees’ knowledge of cost evaluation terminology and methods.
Increase attendees’ practical ability to design and conduct a cost evaluation of a child welfare program.

4. Today’s Agenda Basic cost evaluation terms and methods.
Collection of program-level and case-level service utilization & cost data.
Analysis, interpretation, & reporting of program-level and case-level cost data.
Questions & open discussion.

5. Basic Terms: Cost and Cost Components
Cost is the dollar amount spent providing a service.
Total cost is the sum of all direct costs and indirect costs.
Other types of cost include time cost, social cost, opportunity cost, etc.

6. Basic Terms: Cost and Cost Components (cont.)
Direct costs are those costs that can be assigned to a particular case or program; for example, the monetary value of a case worker’s person-time used to counsel a client.
Indirect costs are costs that are not identified specifically with a particular case or program; for example, the monetary value of a case worker’s fringe benefits or the cost of facilities.

7. Basic Terms: Cost and Cost Components (cont.)
Both direct and indirect costs are comprised of multiple discreet cost categories, such as labor, facilities, equipment, supplies, equipment, etc.
These items are recorded as line items in budgets or accounting/financial reports.
For most service programs, labor costs drive total cost and are the cornerstone of analyses.

8. Basic Cost Evaluation Methods
Describe/analyze program costs using aggregate data that distinguish cost categories by type of programmatic activity (or phases of program maturation).
Describe/analyze program costs using case-level data that distinguish cost categories by type of programmatic activity for each case.
Describe/analyze program costs using both aggregate and case-level data.

9. Advanced Cost Analysis Methods
Cost-effectiveness analysis compares 2 or more interventions on costs and effectiveness.
Effectiveness is measured in client level outcomes (e.g., child maltreatment events).
Result: cost per unit of effect (e.g., cost per prevented child maltreatment.)

10. Advanced Cost Analysis Methods (cont.)
Cost Utility Analysis is like cost effectiveness analysis, except effectiveness is measured as length of life (e.g., survival) or quality of life such as, “cost per quality-adjusted life year (QALY) saved.”

11. Advanced Cost Analysis Methods (cont.)
Cost-Benefit Analysis compares 2 or more interventions on costs and monetized benefits.
Benefits are expressed in monetary units, which allows for easier comparison of results across programs with different types of effects.

12. Advanced Cost Analysis Methods (cont.)
Return on investment analysis is like cost benefit analysis, except monetized benefits are aggregated over an extended period and costs include all phases of program development, implementation and operation.

13. Phase 1
Data collection

14. Why Collect Categorized Aggregate Data?
More informative to decision-making than budget reports or accounting system output.
Less expensive and burdensome than case-level cost analysis.
Complements case-level cost analysis and other more advanced types of cost analysis.

15. Categorizing Aggregate Program Cost Data
The amounts in each budgeting and accounting line item are re-distributed into programmatically relevant categories usually based on post-hoc estimates of the use of labor (personnel costs) in a preceding period.
For example, personnel (labor) costs associated with delivering services to clients can be divided between activities “directly involving a client” and activities conducted “on behalf of a client”

16. Categorizing Aggregate Program Cost Data (cont.)
These two board categories of client-related personnel costs can be further disaggregated into subcategories; for example, distinguishing between “in-take assessment” and “parenting skills enhancement counseling.”
Labor costs for administration—personnel not providing front-line services--can be distinguished between client-related and management-related administrative activities.

17. Service Activities Conducted
Programmatic Cost Analysis: Personnel Data Elements (Family Connections)
Service Activities Conducted
With a Client
on Behalf of a Client
· Advocate with client
· Advocate without client
· Assess
· Clinical documentation
·   Counseling/ support
·   Research
·   Provide
·   Consult/ collaborate
·   Plan
·   Locate resources
·   Refer
·   Risk management meeting
·   Teach
·   Schedule
·   Staff travel
·   Transport
·   Contact attempt

18. Programmatic Cost Analysis: Personnel Data Elements (Family Connections)
Administrative Activities Related to Client Activities
Administrative Activities Related to Management Activities
· Give supervision
· Referral screening
·   Receive supervision
·   Outreach/ marketing
·   Team meeting/ Group consultation
·   Writing
·   Provide training
·   MIS entry
·   Attend training
·   Staff organizations meetings/
committees
·    Mentor program technical
assistance
·   Other administrative activities
·    Professional/ community
·    Organization committees
·    Other administrative activities

19. Why Collect Case-Level Data?
Examine cross-case cost variation.
Increase measurement precision inside the “black box”.
Enable analysis of outcomes in relation to amounts or types of services received (dose-intensity/type analysis.)
Improve ability to conduct multivariate statistical analysis; especially if intent-to-treat analysis fails in a randomized study.

20. Identify and Define Program Activity Categories
Case-related staff activities
Service activities with a client (face-to-face or phone/ )
Service activities on behalf of a client (preparation for or following interaction with or on behalf of client)
Activities required to operate the program, but not related to individual clients.
Administrative activities.

21. Building Case-Level Costs
Relies on counts of the amount of staff time (labor) devoted to each activity by each provider involved with a case.
Multiplying the amount(s) of staff time by the “loaded” hourly labor rate (monetary value/dollars) yields “cost per activity.”
Sum of per activity costs yields “cost per case.”
Loaded hourly labor rate includes base compensation (salary), fringe benefits, and overhead expenses such as facilities, equipment, and general administrative support not assigned to a specific case.

22. Involve Key Programmatic Staff, Including Frontline Caseworkers
Invaluable feedback about identifying activity categories and defining/populating categories.
Problem-solve efficient data collection processes.
Sets foundation for assuring data quality.
Builds commitment and buy-in.
Helps to increase the validity of the interpretation of findings.

23. Determine the Measurement Timeframe
What is the period of time during which cost (and outcome) data are collected?
How long will data be collected?
Pre-Implementation/Planning Phase?
Seasonal component of intervention?
Length of time between startup and full capacity?
Full intervention period or sampling of intervention period?
The time horizon has possible implications (e.g., need for a discount/depreciation rate).

24. Identify and Define Costs
Personnel
Determine how staff time will be tracked
FTE (including sick, vacation, and other non-productive time)
Time spent working on project
Non-personnel
Space and utilities
Travel
Consumable and non-consumable supplies and equipment
Possible to include donated resources
Volunteer time and services
Donated facilities, equipment, supplies

25. Collect Additional Case-Related Information
Dates that client started and ended participation in program
Demographic characteristics
Gender
Race and ethnicity
Employment status
Marital status
Education status
Date of birth
Number in household

26. Prepare Staff Develop detailed procedures manual for data collection.
Provide initial and ongoing training (data collection procedures and data collection form/system.)
Conduct a pilot trial period.
Develop data collection supervision and support systems (e.g. help line for data collectors to submit questions.)

27. Data Collection Preparation
Establish rules for coding activities and allocating staff time increments on activities (minutes/fractions of hours.)
Develop a activity tracking form/database (time used, staff ID, client ID, activity code, location.)
Develop a reporting plan
How frequently should staff submit activity tracking forms?

28. Example: Data Collection Form
Establish rules for coding activities and allocating staff time increments on activities (minutes/fractions of hours.)
Develop a activity tracking form/database (time used, staff ID, client ID, activity code, location.)
Develop a reporting plan
How frequently should staff submit activity tracking forms?

29. Data Repository Preparation
Develop data entry process
Complete paper form and:
Key-enter data using database screens
Scan documents into database
Direct data entry using agency workstations
Data entry through internet connections
Component of existing database or stand-alone database?

30. Mode of Visit (1=Face-to-face, 2=Phone, 3=Written, 4=Other)
Example: Database
Staff
Case #
DATE
Loc of Visit
Mode of Visit (1=Face-to-face, 2=Phone, 3=Written, 4=Other)
Advocate WITH Client
Assess
Counseling/Support
Schedule
Filene
1000
12/4/2006
HOME
FACE TO FACE
120 
20 
1001
OFFICE
PHONE  5
1230
COMM
120
1009
12/8/2006 60 20 5 

31. Data Quality Assurance
Develop systems and procedures for data quality assurance:
Data entry screens with built-in completeness and log-error checks
Additional logic-error checks across cases
Monitor cross-provider consistency
Compare service use records with case/clinical notes for completeness (all clients in day) and accuracy (for all services and duration)

32. Data Quality Assurance (cont.)
Ensure data collection completeness:
To reduce invalid estimates, assign daily deadlines
Conduct ongoing supervisory checks of data
Examine data monthly (so not forgotten)
Reward staff with highest timely completed rates

33. Common Challenges in Case-Level Data Collection
Staff turnover
Data collector drift
Unexpected influences of local events/contextual factors
Multiple staff and sites/agency locations collecting data

34. Accounting for Overhead & Administrative Costs
These costs can not be assigned to a specific case (fringe benefits, space and other forms of infrastructure support and labor associated with functions such as payroll, invoicing, accounts receivable, etc.).
Two approaches: 1)use established, existing rates, or 2)create rates for fringe, overhead/general administration.

35. Phase 2
Analylsis & Reporting

36. Determine the Measurement/Analysis Perspective
Who is the intended audience?
Society?
Policy makers?
Funding agency?
Agency implementing the intervention?
Other agencies considering implementing the intervention?
Intended audience determines which type of economic evaluation is needed and which costs (and benefits) to examine.

37. Example: Total Programmatic Costs by Site and Year

38. Case-Level Cost Analysis
A sequence of progressively more sophisticated case-level cost analyses is conducted building from univariate analyses to multivariate analyses.
A seminal analyses focuses on the relationship between per-case cost and case outcomes using cost and outcome data and applying multivariate statistical analysis techniques such as repeated ANOVA, hierarchical linear modeling, or general equation modeling.

39. Special Considerations on Conducting Multi-Site Cost Analysis
Encourage sites to be involved in all phases based on their interest-each site should receive numerous interim and final products from these analyses.
The privacy of individual sites’ data and analyses should be protected according to agreed-upon rules for data sharing (though it should be understood that the case-level cost data analysis is an open, transparent, collaborative multi-site endeavor.)
Develop rules with regard to publications, access to data and statistical analysis programming code, and other shared issues.

40. Conduct Service Utilization Analyses
Develop and collaboratively review output reports (graphics and printouts) on service activity frequency distributions.
Reports should describe the types and amounts of labor expended on each case during the observation period.

41. Example: Total Minutes by Category for One Year

42. Intervention vs. Control by Category in Minutes (One Year)

43. Example: Minutes per Direct Activity (One Year)

44. Example: Minutes per Direct Activity per Staff ID for Case 1

45. Example Display of Service Duration and Intensity (One Year)

46. Obtain and Merge Client & Program Staff Characteristics Data
Client characteristic variables enable analyses that compare service utilization and cost patterns across subgroups defined by client characteristics.
Staff characteristic variables enable analyses that compare service provision and cost patterns across subgroups defined by staff characteristics.

47. Example: Staff Characteristics
Staff ID
Gender
Race/Ethn
Education
Degree
Years Exp. in SW Field
Position Title A F
White
Masters
MSW, LMSW
16-20
Program Manager B
MSW, LCSW
Program Coordinator C
M.Ed Counseling, LPC, LMFT
2-5
Staff Clinician D
College Grad
BS Psychology
Case Manager E
Black/Hispanic
MS Counseling, LPC Intern
Black/AA
BSW, LBSW
> 20 G
MA Counseling, LPC, NCC, RPT H
MSW, LMSW, MPH
Process Evaluator I
Other
MA Sociology
11-15
Outcome Evaluator J
Ph.D
DrPH Behavioral Health
Evaluation Supervisor

48. Establish Unit Cost Factors
Develop staff member/class unit cost factors or hourly labor rates and overhead rates expressed in dollars
Staff compensation
Fringe benefits
Overhead expenses

49. Example: Staff Hourly Labor &
Fringe Rates
Staff ID
Base Salary A Hourly
Base Salary B Hourly
Avg. Annual Base Salary
Empl. Status
Fringe Rate
Annual Hourly Total Comp. A
30.87 “” FT
26.8%
39.14 B
24.04
24.76
24.40
30.94 C
17.42
18.29
17.86
22.64 D
14.17
17.97 E
17.25
21.87 F
15.24
15.70
15.46
PT(50%)
19.61 G
16.29
20.66 H
19.07
24.18 I
20.83
21.45
21.14
26.81 J
24.28
30.79

50. Remaining Analysis Steps
Descriptive Cost Analyses
These analyses rely on time-based service utilization variables and dollar-based cost variables. The first activity is to calculate total cost per case using the unit cost factors developed earlier. This enables both a summary of costs for all cases and cross-case comparisons
Conduct Multivariate Cost Analyses
These analyses treat case-level cost as the primary dependent variable and incorporate variables, such as client characteristics and service utilization patterns, to investigate predictors of cost variation across cases.

51. Interpretation & Reporting
Interpretation of data
Include program staff in discussion across all stages of analyses
Reporting
Graphic presentation
Document and/or present findings in language that providers and policymakers can understand (i.e., no economic jargon)

52. Open Discussion

53. References
Corso, P. & Filene, JF. (in press). Programmatic cost analysis of interventions to prevent child maltreatment.
Corso, P.S., & Lutzker, J.R. (2006). The need for economic analysis in research on child maltreatment. Child Abuse & Neglect, 30,
Drummond, M.F., O’Brien, B., Stoddart, G.L., & Torrance, G.W. (1997). Methods for the economic evaluation of health care programs (2nd ed.). Oxford: Oxford University Press.
Foster, E.M., Johnson-Shelton, D., & Taylor, T.K. (2007). Measuring time costs in interventions designed to reduce behavior problems among children and youth. American Journal of Community Psychology, 40,
Foster, E.M., & Jones, D. (2006). Can a costly intervention be cost-effective? An analysis of violence prevention. Archives of General Psychiatry, 63,
Foster, E.M., Porter, M.M., Ayers, T.S., Kaplan, D.L., & Sandler, I. (2007). Estimating the costs of preventative interventions. Evaluation Review, 31,

54. References (cont.)
Haddix, A.C., Teutsch, S.M., & Corso, P.S. (Eds). (2003). Prevention effectiveness: A guide to decision analysis and economic evaluation. New York, NY: Oxford University Press.
Herman, P.H., Avery, D.J., Schemp, C.S., & Walsh, M.E. (2009) Are cost-inclusive evaluations worth the effort? Evaluation and Program Planning, 32,
Hornick, J.P., Paetsch, J.J., & Bertrand, L.D. (2002). A manual on conducting economic analysis of crime prevention programs. National Crime Prevention Center.
Levin, H.M., & McEwan, P.J. (2001). Cost-effectiveness analysis (2nd ed.). Thousand Oaks, CA: Sage Publications.
NIDA Manual
Yates, B.T. (2009). Cost-inclusive evaluation: A banquet of approaches for including costs, benefits, and cost-effectiveness and cost-benefit analyses in your next evaluation. Evaluation and Program Planning, 32,

55. Follow-up Questions?
Contact Information:
James Bell