1. 2015 EAST AFRICA EVIDENCE SUMMIT JULY 8-9, 2015 | NAIROBI, KENYA COMMUNITY PERFORMANCE-BASED FINANCING IMPACT EVALUATION DISSEMINATION MEETING JEANINE UMUTESI CONDO

2. PRINCIPAL COLLABORATORS AND INSTITUTIONS 2 MOH-RBC-MCCH Fidel Ngabo Cathy Mugeni University of Rwanda/CMHS/SPH Ina R. Kalisa James Humuza Jeanine Condo Vedaste Ndahindwa The World Bank Gil Shapira Netsanet W. Workie

3. BACKGROUND AND JUSTIFICATION 3 2006-2008 Health Center PBF impact evaluation results show PBF had a significant impact on: Increasing institutional deliveries Improving quality of prenatal care Increasing child preventive care utilization Vaccination, growth monitoring VCT for couples and more for discordant couples But….PBF did not have a significant impact on: Prenatal / post natal care utilization Modern contraceptive use Short term malnutrition

4. COMMUNITY PBF PROGRAM DESIGN 4 Model 1: Conditional in-kind incentives for women: Aimed at increase early prenatal care utilization, facility deliveries, and postnatal care in order to diagnose and treat preventable threats This demand-side model endows mothers with gifts for: Timely antenatal consultation: first visit in the first four months of pregnancy In-facility delivery Timely postnatal care: consultation within the ten days after delivery

5. COMMUNITY PBF PROGRAM DESIGN 5 Model 2: Supply-side financial incentives for CHW Cooperatives: Aim: (i) improve quality of data reported at the sector level, (ii) increase utilization of priority maternal and child health services, and (iii) improve motivation and behavior of CHWs. Pay for reporting: Quarterly payment based on the timely submission of quality data reports related to 29 indicators. Implemented nationally Not evaluated by this study Pay for performance: Offers financial rewards directly to CHWs.

6. COMMUNITY PBF PROGRAM DESIGN 6 Incentivized indicators Unit Fees (USD) (2010-2014/15) 201020112012 20132014 1.Deliveries: Women accompanied/referred to HC for assisted deliveries 2.731.370.99 1.Antenatal Care: Women accompanied/referred to HC for prenatal care within first 4 months of pregnancy 2.241.120.81 1.Nutrition Monitoring: % of children monitored for nutritional status (6 -59 months) 3.240.5650.433 1.Family Planning: % of regular users using long term methods (IUD, Norplant, Surgical/NSV contraception) 2.111.060.77 1.FP: new users referred by CHWs for modern family planning methods 2.91.451.05

7. RESEARCH OBJECTIVES AND QUESTIONS 7 Main Research Objective The main objective of this study is to evaluate the effects of demand- side (in-kind incentives) and supply-side (financial incentives) on health services utilization and health outcomes—The study attempts to isolate the causal impact of the incentives packages on health outcomes.

8. RESEARCH QUESTIONS 8 The primary research questions of the IE are: Do the demand and supply-side incentives affect: The number of woman who receive antenatal care during the first 4 months of pregnancy? The number of antenatal care visits? The number of facility deliveries? The number of woman-child pairs seen during postnatal care follow up care?

9. RESEARCH QUESTIONS 9 Is there “a multiplicative effect” on outcomes when demand and supply-side incentives are combined? Do the supply-side incentives to CHWs increase the: Use of modern contraceptives services? Increase the time between births? Improve nutritional status in under-fives? Do the supply-side incentives to CHWs affect their motivation and behaviors?

10. Study Design and IE Surveys

11. EVALUATION DESIGN The study is a prospective, randomized impact evaluation CPBF interventions are randomly assigned at the sector level into four study arms: o D: Demand-side in-kind incentives to women o S: Supply-side financial incentives to CHW cooperative o DS: Demand-side incentives to women + Supply-side incentives to CHW cooperative o C: Comparison group 11

12. EVALUATION DESIGN Treatment group : sample of sectors where the interventions were implemented starting in 2010. Comparison group: sample of sectors where the interventions were not implemented until 2013 In demand side and comparison sectors, CHW cooperatives received the average incentive payments distributed where the supply side intervention was implemented 12

13. SAMPLING DESIGN Study Arm# Sectors # CHW Cooperatives # HH# CHWs D: In-kind Incentives 50 6001200 S: CHW incentives 50 6001200 DS: In-kind + CHW incentives 50 6001200 C: Average financing 50 6001200 TOTAL 200 24004800 13 Baseline Planned Sample Sizes by Study Arm

14. SAMPLING DESIGN 14 Map of Sectors by Study Arm

15. SAMPLING STRATEGY AT BASELINE Cooperative Level All 200 in 200 sectors Household Level The household with the most recent birth in the village (within last 4 months) was selected for the household interview for each of the 2400 randomly selected villages CHW individual Level 2 CHWs selected for CHW individual questionnaire for each of the 2400 randomly selected villages Forcibly include MNH CHW (ASM) Randomly select second 15

16. SAMPLING STRATEGY AT ENDLINE The endline survey covered the same sectors and villages as the baseline Changes from baseline survey: Doubling households interviewed Reduce by half CHWs interviewed Add health center assessments 16

17. SAMPLING STRATEGY AT ENDLINE CHWs cooperative Same as baseline Households survey Old sample: Tracking of households interviewed in baseline New sample: Households with the most recent birth in each of the villages visited during the baseline CHWs survey Interview only the CHW in charge of maternal and neonatal heath (ASM) in each village Health facility survey A health facility assessment A health worker survey (ANC and child curative care) Patient exit interviews (ANC and child curative care) 17

18. CPBF IE DATA-AT BASELINE Baseline data collection: Fielded from February to May 2010 Final baseline sample size 18 Study Arm# Sectors#Households#cooperatives# CHWs Demand-side50600491,162 Supply-side50600501,184 Demand- and Supply-side 50600501,186 Control48576481,136 Total19823761974,668

19. CPBF IE DATA-AT ENDLINE Endline data collection: Fielded from November 2013 to June 2014 Final Endline sample size 19 Study Arm#Sectors # Baseline HH # New HH #CHWs cooperatives # CHWs Demand-side5054959450569 Supply-side5053659150559 Demand- and Supply-side 5053359350556 Control4853956547536 Total198215723431972,220

20. CPBF IE DATA-AT ENDLINE  Quality checks:  During data collection: regular spot checks and supervision by field coordinator and research team  After data collection: 10% revisit of households (98% of households were really interviewed) 20

21. VALIDATION OF IE DESIGN: BASELINE BALANCE CHECK  T-tests were used to assess the difference between each one of the three treatment arms and the control  F-tests were used to test the hypothesis that the variable means are identical among all 4 study arms. 21

22. COMMUNITY PBF RESULT 22

23. COMPARISON OF BASELINE AND ENDLINE SAMPLES OF WOMEN WITH RECENT BIRTHS 23

24. CHARACTERISTICS OF CHWS AT ENDLINE 24

25. IMPACTS ON HEALTH SERVICE UTILIZATION OUTCOMES: TIMELY ANC, TIMELY PNC AND IN-FACILITY DELIVERY Indicators targeted by the Demand-side intervention Timely PNC was not targeted in the supply-side intervention Sample: 2334 women who were pregnant or gave birth shortly before the endline survey

26. Impact on first ANC visit during the first 4 months of the pregnancy: A positive and significant impact of the demand-side in-kind incentives of about 10 percentage points The CHW incentives are not found to have a significant effect No difference between the ‘Demand’ and the ‘Demand+Supply’ treatment arms Treatment Arm

27. Impact on having at least one ANC visit: Not targeted by the program! Not impacted by the program but this outcome is already almost universal.

28. Impact on completing at least 4 ANC visits: Not targeted by the program! Higher in the intervention sectors, but not statistically significant at the 10% level

29. Impact on skilled-attended in-facility deliveries: No statistically significant difference between the treatment arms Rate has increased substantially in the duration of the study for other reasons

30. Impact on PNC within the 10 days after delivery: A positive and significant impact of the demand-side in-kind incentives of about 7 percentage points The CHW incentives are not found to have a significant effect No difference between the ‘Demand’ and the ‘Demand+Supply’ treatment arms

31. RESULTS ARE ROBUST TO THE FOLLOWING CHECKS: 1.Regressions with controls : Detected impacts are not due to difference in observable characteristics between the groups. Randomization was done at the sector level -> controlling for individual-level characteristics (for example: age, marital status, education level, household characteristics, distance from health center,…) Include characteristics which were not perfectly balanced 2.Excluding “misclassified” sectors : 3 sectors have been misclassified during the transition to the new computerized data system, although cooperatives were not informed 3.Difference-in difference approach: Account for fixed unobservable characteristics

32. IMPACT ON FERTILITY AND FAMILY PLANNING The supply-side program incentivizes CHW cooperative for new and regular users of modern contraceptives Sample: 2,157 “baseline women” Were pregnant or gave birth shortly before the baseline survey Re-interviewed in 2014 Not the optimal sample for evaluating impact on fertility outcomes

33. No impact found on fertility and use of modern contraception 3 indicators: 1.Pregnancy since baseline interview 2.Ever used modern contraceptive method 3.Ever used modern contraceptive method – adjusted for baseline responses

34. DID THE INTERVENTION IMPACT CHW BEHAVIOR OR MOTIVATION? We found no evidence that either of the interventions impacted the following self-reported indicators: 1.Average number of hours spent on health activities in a week 2.Number of household visited in the past month 3.CHWs seek advice from peers 4.Measures of satisfaction and motivation