2015 EAST AFRICA EVIDENCE SUMMIT JULY 8-9, 2015 | NAIROBI, KENYA COMMUNITY PERFORMANCE-BASED FINANCING IMPACT EVALUATION DISSEMINATION MEETING JEANINE UMUTESI CONDO

PRINCIPAL COLLABORATORS AND INSTITUTIONS 2 MOH-RBC-MCCH Fidel Ngabo Cathy Mugeni University of Rwanda/CMHS/SPH Ina R. Kalisa James Humuza Jeanine Condo Vedaste Ndahindwa The World Bank Gil Shapira Netsanet W. Workie

BACKGROUND AND JUSTIFICATION Health Center PBF impact evaluation results show PBF had a significant impact on: Increasing institutional deliveries Improving quality of prenatal care Increasing child preventive care utilization Vaccination, growth monitoring VCT for couples and more for discordant couples But….PBF did not have a significant impact on: Prenatal / post natal care utilization Modern contraceptive use Short term malnutrition

COMMUNITY PBF PROGRAM DESIGN 4 Model 1: Conditional in-kind incentives for women: Aimed at increase early prenatal care utilization, facility deliveries, and postnatal care in order to diagnose and treat preventable threats This demand-side model endows mothers with gifts for: Timely antenatal consultation: first visit in the first four months of pregnancy In-facility delivery Timely postnatal care: consultation within the ten days after delivery

COMMUNITY PBF PROGRAM DESIGN 5 Model 2: Supply-side financial incentives for CHW Cooperatives: Aim: (i) improve quality of data reported at the sector level, (ii) increase utilization of priority maternal and child health services, and (iii) improve motivation and behavior of CHWs. Pay for reporting: Quarterly payment based on the timely submission of quality data reports related to 29 indicators. Implemented nationally Not evaluated by this study Pay for performance: Offers financial rewards directly to CHWs.

COMMUNITY PBF PROGRAM DESIGN 6 Incentivized indicators Unit Fees (USD) ( /15) Deliveries: Women accompanied/referred to HC for assisted deliveries Antenatal Care: Women accompanied/referred to HC for prenatal care within first 4 months of pregnancy Nutrition Monitoring: % of children monitored for nutritional status (6 -59 months) Family Planning: % of regular users using long term methods (IUD, Norplant, Surgical/NSV contraception) FP: new users referred by CHWs for modern family planning methods

RESEARCH OBJECTIVES AND QUESTIONS 7 Main Research Objective The main objective of this study is to evaluate the effects of demand- side (in-kind incentives) and supply-side (financial incentives) on health services utilization and health outcomes—The study attempts to isolate the causal impact of the incentives packages on health outcomes.

RESEARCH QUESTIONS 8 The primary research questions of the IE are: Do the demand and supply-side incentives affect: The number of woman who receive antenatal care during the first 4 months of pregnancy? The number of antenatal care visits? The number of facility deliveries? The number of woman-child pairs seen during postnatal care follow up care?

RESEARCH QUESTIONS 9 Is there “a multiplicative effect” on outcomes when demand and supply-side incentives are combined? Do the supply-side incentives to CHWs increase the: Use of modern contraceptives services? Increase the time between births? Improve nutritional status in under-fives? Do the supply-side incentives to CHWs affect their motivation and behaviors?

Study Design and IE Surveys

EVALUATION DESIGN The study is a prospective, randomized impact evaluation CPBF interventions are randomly assigned at the sector level into four study arms: o D: Demand-side in-kind incentives to women o S: Supply-side financial incentives to CHW cooperative o DS: Demand-side incentives to women + Supply-side incentives to CHW cooperative o C: Comparison group 11

EVALUATION DESIGN Treatment group : sample of sectors where the interventions were implemented starting in Comparison group: sample of sectors where the interventions were not implemented until 2013 In demand side and comparison sectors, CHW cooperatives received the average incentive payments distributed where the supply side intervention was implemented 12

SAMPLING DESIGN Study Arm# Sectors # CHW Cooperatives # HH# CHWs D: In-kind Incentives S: CHW incentives DS: In-kind + CHW incentives C: Average financing TOTAL Baseline Planned Sample Sizes by Study Arm

SAMPLING DESIGN 14 Map of Sectors by Study Arm

SAMPLING STRATEGY AT BASELINE Cooperative Level All 200 in 200 sectors Household Level The household with the most recent birth in the village (within last 4 months) was selected for the household interview for each of the 2400 randomly selected villages CHW individual Level 2 CHWs selected for CHW individual questionnaire for each of the 2400 randomly selected villages Forcibly include MNH CHW (ASM) Randomly select second 15

SAMPLING STRATEGY AT ENDLINE The endline survey covered the same sectors and villages as the baseline Changes from baseline survey: Doubling households interviewed Reduce by half CHWs interviewed Add health center assessments 16

SAMPLING STRATEGY AT ENDLINE CHWs cooperative Same as baseline Households survey Old sample: Tracking of households interviewed in baseline New sample: Households with the most recent birth in each of the villages visited during the baseline CHWs survey Interview only the CHW in charge of maternal and neonatal heath (ASM) in each village Health facility survey A health facility assessment A health worker survey (ANC and child curative care) Patient exit interviews (ANC and child curative care) 17

CPBF IE DATA-AT BASELINE Baseline data collection: Fielded from February to May 2010 Final baseline sample size 18 Study Arm# Sectors#Households#cooperatives# CHWs Demand-side ,162 Supply-side ,184 Demand- and Supply-side ,186 Control ,136 Total ,668

CPBF IE DATA-AT ENDLINE Endline data collection: Fielded from November 2013 to June 2014 Final Endline sample size 19 Study Arm#Sectors # Baseline HH # New HH #CHWs cooperatives # CHWs Demand-side Supply-side Demand- and Supply-side Control Total ,220

CPBF IE DATA-AT ENDLINE  Quality checks:  During data collection: regular spot checks and supervision by field coordinator and research team  After data collection: 10% revisit of households (98% of households were really interviewed) 20

VALIDATION OF IE DESIGN: BASELINE BALANCE CHECK  T-tests were used to assess the difference between each one of the three treatment arms and the control  F-tests were used to test the hypothesis that the variable means are identical among all 4 study arms. 21

COMMUNITY PBF RESULT 22

COMPARISON OF BASELINE AND ENDLINE SAMPLES OF WOMEN WITH RECENT BIRTHS 23

CHARACTERISTICS OF CHWS AT ENDLINE 24

IMPACTS ON HEALTH SERVICE UTILIZATION OUTCOMES: TIMELY ANC, TIMELY PNC AND IN-FACILITY DELIVERY Indicators targeted by the Demand-side intervention Timely PNC was not targeted in the supply-side intervention Sample: 2334 women who were pregnant or gave birth shortly before the endline survey

Impact on first ANC visit during the first 4 months of the pregnancy: A positive and significant impact of the demand-side in-kind incentives of about 10 percentage points The CHW incentives are not found to have a significant effect No difference between the ‘Demand’ and the ‘Demand+Supply’ treatment arms Treatment Arm

Impact on having at least one ANC visit: Not targeted by the program! Not impacted by the program but this outcome is already almost universal.

Impact on completing at least 4 ANC visits: Not targeted by the program! Higher in the intervention sectors, but not statistically significant at the 10% level

Impact on skilled-attended in-facility deliveries: No statistically significant difference between the treatment arms Rate has increased substantially in the duration of the study for other reasons

Impact on PNC within the 10 days after delivery: A positive and significant impact of the demand-side in-kind incentives of about 7 percentage points The CHW incentives are not found to have a significant effect No difference between the ‘Demand’ and the ‘Demand+Supply’ treatment arms

RESULTS ARE ROBUST TO THE FOLLOWING CHECKS: 1.Regressions with controls : Detected impacts are not due to difference in observable characteristics between the groups. Randomization was done at the sector level -> controlling for individual-level characteristics (for example: age, marital status, education level, household characteristics, distance from health center,…) Include characteristics which were not perfectly balanced 2.Excluding “misclassified” sectors : 3 sectors have been misclassified during the transition to the new computerized data system, although cooperatives were not informed 3.Difference-in difference approach: Account for fixed unobservable characteristics

IMPACT ON FERTILITY AND FAMILY PLANNING The supply-side program incentivizes CHW cooperative for new and regular users of modern contraceptives Sample: 2,157 “baseline women” Were pregnant or gave birth shortly before the baseline survey Re-interviewed in 2014 Not the optimal sample for evaluating impact on fertility outcomes

No impact found on fertility and use of modern contraception 3 indicators: 1.Pregnancy since baseline interview 2.Ever used modern contraceptive method 3.Ever used modern contraceptive method – adjusted for baseline responses

DID THE INTERVENTION IMPACT CHW BEHAVIOR OR MOTIVATION? We found no evidence that either of the interventions impacted the following self-reported indicators: 1.Average number of hours spent on health activities in a week 2.Number of household visited in the past month 3.CHWs seek advice from peers 4.Measures of satisfaction and motivation