1. Costs, Cost-Effectiveness and Cost-Benefit Analysis Brooks Bowden Center for Benefit-Cost Studies of Education Teachers College, Columbia University Clive Belfield Queens College, City University of New York IES PI MeetingDecember 11 2015

2. Outline ① Foundations of CA+CEA+CBA = “EA” ② Need for EA ③ Case studies ④ Future of EA Context: EA research at CBCSE.org EA teaching: IES Methods Training Grant (2014-17, #R305B140003)

4. Case Studies: EA Socio-Emotional Learning interventions Talent SearchReading PartnersCity ConnectsEarly literacy 3

5. Case Studies: Displacement Outcome specification Socio-Emotional Learning interventions Program fidelity Talent Search Treatment contrast Reading Partners Service mediation City Connects Power of the test Early literacy 4

6. COST, COST-EFFECTIVENESS, AND COST-BENEFIT ANALYSIS FOUNDATIONS 4

7. Basic Ideas Money is scarce; must consider if program is affordable ① Costs Analysis (CA): a basic description of how much resource is needed for a program Cannot just look at if a program works; must also look at resources needed for it to work ② Cost-Effectiveness Analysis (CEA): the link between resources needed for programs and their outcomes Must look at the economic consequences of a program ③ Cost-Benefit Analysis (CBA): test whether economic returns justify the cost 5

8. Costs Analysis See if program can be afforded Enumerate all resources to implement a policy Incremental costs over “next best alternative” or “business-as-usual” This is an actual research project using ‘ingredients’ method: o Sampling o Quantities of ingredients o Prices of ingredients o Testing for difference in costs Average Cost per participant AC must be within budget 6

9. Cost Example High school mentoring program for 500 students: 4 teachers @ $100,000 per 4 classrooms @ $25,000 per After-school program for 1,000 students: 6 counselors @ $80,000 per 1 gym @ $120,000 per Mentoring Program After- school Program Total number of students 5001,000 Personnel$400,000$480,000 Facilities$100,000$120,000 Total Cost$500,000$600,000 Average Cost$1,000$600 7

10. Cost-Effectiveness Analysis Find cheapest way to do the thing we have decided to do Compares policy alternatives, based on ratio of their costs to a quantifiable (but not monetized) effectiveness measure Compare alternative policies, not rule on single policy Compare similar policies: scale, target group, expected effects CE ratio = $Cost/Unit of Effectiveness Lowest CE ratio is most cost-effective 9

11. CEA Example HS Dropout rate = 20% School size = 500 Mentoring program: o Costs $1,000 per student o Reduces dropout rate by 5% After-school program: o Costs $600 per student o Reduces dropout rate by 2% Mentoring program After- school program Total cost$500,000$300,000 Baseline dropouts 100 Yield of new high school graduates 52 CE ratio$100,000$150,000 9

12. Cost-Benefit Analysis See if it is worth spending money on a program Compare policy alternatives, based on ratio of their costs to quantifiable and monetized effectiveness measures Monetized effectiveness measures are “benefits” (e.g. earnings gains, lower special education spending) BC ratio = $Benefits/Cost or B−C = Benefit-Costs Worth it if BC ratio > 1 or B−C > 0 11

13. CBA Example Mentoring program reduces dropout rate by 5% Each new high school graduate compared to a dropout over lifetime: o Earns $120,000 more o Saves taxpayer $30,000 in spending on crime Mentoring program Total cost over baseline C$500,000 Yield of new high school graduates 5 Total earnings gain B$600,000 Total crime saving B$150,000 Net Benefit (B-C)$250,000 Benefit-Cost Ratio (B/C)1.50 12

14. COST, COST-EFFECTIVENESS, AND COST-BENEFIT ANALYSIS NEED FOR EA 12

15. Need for EA in Education Many interventions, reforms, programs, policies Intensively investigate “What Works” Very little guidance on “..At What Cost?” “..Is it Worth the Cost?” 13

16. Examples E Effects Class size reduction ✓✓ Reading Recovery ✓✓ High VA teachers ✓✓ Vouchers ✓ Web-learning= 15

17. Examples E C EffectsCosts Class size reduction ✓✓ Expensive Reading Recovery ✓✓ Very expensive High VA teachers ✓✓ No idea Vouchers ✓ Cheaper (?) Web-learning=Cheaper (?) 16

18. Examples E C E? Cost-Effective or Efficient? Class size reduction Perhaps Compared to what? From whose perspective? Is it affordable? Reading Recovery High VA teachers Vouchers Web-learning 17

19. EA Methods They are methods: o CA: ingredients method o CEA: CA + evaluation impacts o CBA: CA + shadow pricing methods There are methodological standards But not like impact evaluations because hypothesis / statistical testing is not determinative 18

20. EA Practice EA is becoming more common Slowly and from a low base Assumptions rarely harmonized Research questions rarely related to each other Limited hypothesis testing 19

21. Value of EA Conventional answer: It will help decision-making If it does not help us decide between X and Y, it is not needed New answer: It will improve education research generally 19

22. SESSION II APPLICATIONS & RECOMMENDATIOS 21

23. Case Studies Outcome specification Socio-Emotional Learning interventions Program fidelity Talent Search Treatment contrast Reading Partners Service mediation City Connects Power of the test Early literacy 22

24. OUTCOME SPECIFICATION CASE STUDY: SOCIO-EMOTIONAL LEARNING 21

25. SEL Interventions Two full reviews of interventions that enhance SE skills (Durlak et al., 2011; Sklad et al., 2012) Focus on SE skills or test scores? (Can’t do both) SE skills are broad-based, not task-specific Need to look at broad set of outcomes Need to weigh those outcomes This implies CBA or CBA framework 24

26. The Economic Value of Social and Emotional Learning 25

27. Outcomes of SEL interventions? Across the literature outcomes were: Inconsistent More or less Student, teacher, classroom, school level Durable or ephemeral Fast-acting or delayed 24

28. 25

29. Conclusions Better “benefit maps” Focus on school-level or class-level effects of SEL Examine how fast-acting and durable interventions are Generally, pre-specify outcomes (Social Impact Bonds) 28

30. Clincher We propose a cost-effectiveness analysis Question always asked is: How are we measuring effectiveness? We don’t know – we are economists 29

31. PROGRAM FIDELITY CASE STUDY: TALENT SEARCH 38

32. Helps low-income and first-generation college students complete high school and gain access to college Many services: academic achievement, access to financial aid, test taking and study skills assistance, academic advising, tutoring, career development, campus visits Talent Search 31

33. Program Flexibility Site-specific variation: Sites can serve students in G6-12 or cohorts from 6G through 12G; at few or many schools Sites are at schools and at colleges Programs vary with respect to services Cost analysis needed to: ① Describe the program ② Estimate in-kind resources ③ Identify most cost-effective sites 32

34. EA of Talent Search Cost Analysis: Collected cost data using ingredients method from 6 TX sites and 3 FL sites CEA: Linked cost data to effects from Constantine et al. (2006) CBA: Shadow priced estimates of earnings gains from high school completion and college attendance 33

35. 34

36. 35

37. 36

38. Read 180 For struggling readers G1-12 to improve decoding, fluency, and comprehension skills Prescriptive program : o 90-minutes daily of whole-group, small-group and individualized instruction + videos + feedback o Class size 15 Collected cost data using ingredients method from three sites (Levin et al., 2007) 37

39. Read 180 Cost per Student Site 1Site 2Site 3 Students served 6,7011,0802,400 Personnel (teachers) $320 $950 $70 Personnel (administrators, technicians, coordinators) $50 $400 $60 Equipment/materials (computers, licenses) $250 $150 $140 Other (prof. dev., sub teachers, other) $- $10 Average Cost $610 $1,510 $280 46

40. JOBSTART Education/training program for low-income youth, including remedial education, GED preparation, and job placement assistance Youth can remain in the program for up to two years MIS data from Cave et al. (2003) 39

41. 40

42. Mean = $14,900 SD = $13,400 N =5050

43. Job Corps: Line Item Costs Cost perMeanSDMax Counseling$4,6204360$50,020 Admin$3,6403410$53,540 Voc. training$2,9302760$33,540 Allowance$2,2001940$11,110 Child care$1,8101670$17,450 Outreach$1,6301440 $9,860 Education$1,6001460$15,210 Med/dental$1,2701180$13,190 Food$1,0501020$12,280

44. Conclusions Cost Analysis yields information on: What the program is Actual program implementation Fidelity Many evaluations include (idiosyncratic?) investigation of program fidelity Base fidelity on resource use indicators 43

45. TREATMENT CONTRAST CASE STUDY: READING PARTNERS 28

46. Reading Partners Supplemental pull-out reading program: o One-on-one tutoring (2x45 minutes per week) o Dedicated school space and materials o Structured and individualized curriculum o Data-driven instruction o Rigorous ongoing training o Instructional supervision and support For struggling students in K-5 Split responsibility: o Run with volunteers supported by AmeriCorps Site Coordinators o Site Coordinators supervised weekly by Program Managers 45

47. Mobilizing Volunteer Tutors 46 www.mdrc.org/sites/default/files/ReadingPartners_2015_FR.pdf

48. Evaluating RP Multi-site randomized design: o 1,250 students randomly assigned at 19 school sites o Grades 2-5, eligible for Reading Partners o Evaluated during school year 2012-13 o Baseline equivalence of treatment and control Cost analysis was part of evaluation: o What resources were needed to implement RP as per the evaluation? o What proportion of RP costs were borne by the school? o How did these resource requirements compare with the resources required to implement other supplemental reading services in these schools? 47

49. RP Context Cost ≠ Price School selects RP as one option for supplemental reading services Children in need must be served – impact relative to other supplemental services (not no intervention) Cost Study Sample: 6 of 19 study sites o Chosen based on strong implementation, geographic diversity, and ability to collect reliable data Data collected via implementation observations, teacher surveys, and interviews with RP staff, teachers, reading support staff, and school personnel Total social cost calculated by summing the costs of all the resources utilized to implement program (Levin & McEwan, 2001) o Included both in-kind and financial costs o Used national prices o 2013 $USD Same approach used to assess the cost of the other supplemental reading programs offered at these six schools 48

50. RP Costs Total social cost calculated by summing costs of all resources utilized to implement program (Levin & McEwan, 2001) o Included both in-kind and financial costs o Used national prices o 2013 $USD Same approach used to assess the cost of the other supplemental reading programs offered at these six schools 49

51. RP Cost Analysis Cost per student RP staff$690 AmeriCorps members$930 School staff$90 Volunteer time + transport$1,520 Facilities$300 Materials/equipment$80 Total Ingredients$3,610 50

52. RP Cost per Student Cost per student Cost to school Cost to others RP staff$690$320$370 AmeriCorps members$930 School staff$90 Volunteer time + transport$1,520 Facilities$300 Materials/equipment$80 Total Ingredients$3,610$710$2,900 20%80% 51

53. 52

54. Cost of Other Supplemental Reading Services Total Cost Per Student Reading PartnersOther Supplemental Services Site A$3,450$1,840 Site B$3,420$2,230 Site C$3,570$2,680 Site D$5,190$1,050 Site E$4,210$1,980 Site F$2,740$4,890 Pooled$3,610$1,780 53

55. RP Cost versus BAU Cost 54

56. Conclusions Schools receive a resource rich intervention [RP] for a fraction of the cost Cost-effective option for under-resourced schools Important to consider treatment contrast and BAU in costs as well as effects. 55

57. SERVICE MEDIATION CASE STUDY: CITY CONNECTS 56

58. Service Mediation Interventions These SMIs have attributes of: Diagnosis/assessment Information Referral Assignment to services Tracking Service management 57

59. Service Mediation Interventions Examples: Diagnosis/assessment: Interventions to change college remediation Information: Letter grades and accountability systems Referral: Texts to motivate class attendance Assignment: Grade retention Tracking effect: FAFSA support to direct college choice Tracking effect: Head Start to motivate future parental investment Service management: City Connects 58

60. Implications (1) Impact of intervention is the sum of diagnosis + service receipt o New placement test impact depends on the test + remedial services received based on the test result o Grade retention policy impact depends on whether student is retained + services received by retained v. non-retained students Not possible to disentangle the impact attributable to the intervention from the impact attributable to service receipt; service change must be measured 59

61. Implications (2) Interventions might save resources in the long run o New placement tests have reduced enrollment in remediation (Scott-Clayton et al., 2014) Interventions with no positive effect might be justified purely in resource terms Illustrate using economic framework 60

62. City Connects (CCNX) CCNX diagnoses student need, refers students to most appropriate support services, manages support services, and provides some information/services to students directly Walsh et al. (2014b, Table 7) identify significant gains in academic achievement from CCNX CCNX yields average gain of 0.39 standard deviations in English Language Arts and math by 5 th Grade 61

63. BCA of City Connects 62

64. Sample, Data, and Methods Sample: o 2 representative K-8 schools in Boston o 5 community partners per site, based on service intensity Data from interviews: o Site coordinators, teachers, principals, City Connects central office staff, and community partners Costs estimated with ingredients method: o Opportunity cost not accounting costs/ expenditures o Separate analyses of quantities/prices and by funder 63

65. Costs for K-5 years per site (PV at K, 2013$) City ConnectsMediated Services Personnel$4,868,900 Central Program Staff$174,580 School Site Coordinators$900,040 School Staff$93,140 Parental Involvement$4,990 Materials$5,040$42,070 Facilities$16,410$186,910 Other $774,900 Total$1,204,560$5,872,800 Students780 Average Cost$1,540$7,530 64

66. Average Cost per Student Core + No Extra Service Core + Full Services Core + Estimated Service Change City Connects program$1,540 Mediated services $7,530$3,030 Full Cost (C)$1,540$9,070$4,570 Effects (E)0.39 C/E ratio$3,950$23,260$11,720 65

67. Service Mediation City Connects is effective – large gains in academic achievement If only direct costs are counted, it is very cost- effective But if all service costs are counted, it is unclear Need to perform CBA to fully evaluate City Connects Require different impact evaluation (outcomes, durations) 66

68. POWER OF TEST CASE STUDY: EARLY LITERACY 50

69. Power of Test Before identifying impacts, specify power of test But economists are not focused on statistical significance And distribution of ratio of two variables is not same as distribution of each variable So use “power of economic test” 68

70. Early Literacy Estimated costs for all effective early literacy interventions Costs based on ingredients of effective implementations of interventions Related costs to effectiveness from independent studies 69

71. Improving Early Literacy 70

72. 71 Cost per Effect Size Gain K-PALS $40 Stepping Stones $570 Sound Partners $600 Reading Recovery $2,090 Fast ForWord $1,480 Corrective Reading $6,360 Wilson Reading Systems $13,390

73. Power of Economic Test How much more effective would Wilson have to be to be as cost-effective as K-PALS? How imprecise would estimates of these effects have to be for K-PALS to lose status as most cost-effective? Minimum detectable effect (MDE): the smallest true effect that has a “good chance” of being found to be statistically significant (Bloom) MDCE: the smallest true CE-ratio that has a “good chance” of being found to be statistically significant? 72

74. CA: General Conclusions Gives better understanding of the program Programs typically cost more than guesstimated or proposed or in budget documents Substantial variation in costs for a given program Most programs are not high cost given what is expected to happen 73

75. CEA: General Conclusions No expectation more costly programs are more effective Very low cost programs are unlikely to be effective Cost-effectiveness varies across programs, across sites, and over time Targeted interventions are likely to be more cost- effective 74

76. CBA: General Conclusions Targeted interventions are likely to have higher BC ratios Effective programs tend to have very high benefits Almost always undercount benefits Test scores not as valuable as other skills 75

77. Friday afternoon!Thank you! Brooks Bowden and Clive Belfield www.cbcse.org 76