1. Improving data to improve iCCM programs: Implementation of a data quality and use package in Malawi Presenter: Emmanuel Chimbalanga, Save the Children Malawi & IMCI Unit, MOH Malawi On behalf of the Malawi CCM-IDIP working group: Jennifer Bryce, Emmanuel Chimbalanga, Tiyese Chimuna, Kate Gilroy, Tanya Guenther, Elizabeth Hazel, Angella Mtimuni, Humphreys Nsona University Research Co., LLC June 12, 2014

2. Overview of CCM-IDIP Embedded implementation research project focused on improving program monitoring and evaluation for iCCM Funding provided by USAID/URC and partners include JHU-IIP, Save the Children and the Ministry of Health IMCI Unit 4 Focus countries: Malawi, Mali, Ethiopia, Mozambique Malawi program activities: Desk review of M&E system for iCCM and stakeholder consultations Data quality assessments Implementation and evaluation of 2 innovative approaches  This presentation focuses on one of the innovative approaches – a package to improve data quality and use

3. iCCM program in Malawi Health Surveillance Assistants (HSAs) started providing iCCM for malaria, pneumonia and diarrhea in 2008 HSAs provide iCCM services through village clinics that they operate several days each week As of February 2014, about 4000 HSAs have been trained and deployed for CCM across the 29 districts of Malawi A 2012 data quality assessment (DQA) identified data quality issues and low levels of data use

4. HSA Health center District National Form 1A Form 1B Form 1C iCCM reporting system HSA register HSA report – Form 1A

5. Development of a data quality and use improvement package Developed the package with district health staff and partners The DI package included: – general training on data management, use and interpretation; – refresher training on the routine reporting forms; – simple templates for displaying CCM implementation strength data; – provision of calculators to assist with completing monitoring forms; and – working with district staff to identify reporting benchmarks and action thresholds.

6. Templates for HSAs Set of 5 graphs to summarize: Background data and supervision visits # cases treated and referred Total cases and days VC operated

7. Templates for Health Facilities Set of 6 graphs to summarize:  HSA is residing in catchment area  HSA reporting  Stock-outs lasting more than 7 days  Supervision (routine)  Mentoring  Cases treated by HSAs

8. District templates Electronic data entry Generates dashboard with indicators and time trend graphs Excel format; potential to link to DHIS2 in future

9. All relevant district staff, HSA supervisors and HSAs implementing iCCM (n=426) trained in Dowa and Kasungu districts Feb 2013: TOT with IMCI/deputy coordinators, HMIS, Pharm. Tech, others April 2013: District staff conducted trainings for HSAs and senior HSAs at health facilities – Half-day trainings – Trainings supervised by MOH and SC staff Pilot Implementation

10. Sample: 5 health facilities and 3-4 HSAs per facility were randomly selected at baseline in each district. The same facilities and HSAs were followed up at endline Data collection: Baseline data collection in June 2012. Endline data collection in July/August 2013 after 3+ months of implementation. Data analysis: 1. Measured changes in reporting: Consistency: measured through results verification ratio (RVR: verified/reported; 1.0 = perfect consistency) Availability and completeness: forms were submitted and complete for the previous month 2. Assessed data use (display of templates; how used) 3. Documented package costs Evaluation of the package

11. Results Verification Ratio Calculation: HSA level example Count from HSA register (e.g. # of fever cases treated) Count from HSA Form 1 A report (e.g. # of fever cases reported) RVR of 1 means perfect match RVR of less than 1 indicates over-reporting RVR more than 1 indicates under-reporting

12. Consistency: Improvements in HSA reporting After introduction of the package, the monthly data reported by HSAs for cases treated was more consistent with what they recorded in their registers: – Average reporting consistency for cases treated improved – There was less variation in reporting consistency after the package (shown by the smaller boxes). Figure: Comparison of reporting consistency levels for cases treated between baseline and endline

13. Consistency: Health facility level Baseline showed good consistency between HSA reports (Form 1A) and HF summary (Form 1B) for cases treated; consistency sustained at endline Some over-reporting of HSA stock-outs of ACT, AB, ORS at baseline (RVR less than 1); minor improvements in consistency for stock-out reporting at endline (RVRs closer to 1) Small sample sizes (less than 10 facilities) limit conclusions

14. Availability and completeness: district differences At baseline, Dowa was the stronger district with 95% of HSA and HF forms available and complete compared with just 74% in Kasungu district At endline, Kasungu showed some improvements in availability and completeness for HSA forms and 100% of facility level forms were available and complete In Dowa, availability and completeness dropped (63% of HSAs forms and just 16% of HF forms available and complete)

15. All HSAs and nearly all health facilities were using the templates (one HF wasn’t using) Almost all (97%) said the templates were easy to use and not time intensive (most spent less than an hour to complete each month) Most (89%) of HSAs had completed the templates for each month since January 2013; completeness was lower at HF (78%) About 40% of HSAs could not display their templates because they lacked a permanent structure Data use: data display templates

16. Data use: examples  Most HSAs mentioned using data to inform their community health education activities “The display of data makes it easy for the community to see which cases are common which helps in choosing targeted interventions to address the situation” – HSA, Dowa “The community was told that not any cough is fast-breathing; the community perception of demanding cotrim for any cough is gradually changing” – HSA, Dowa  Senior HSAs reported using data to make staffing decisions (deploy HSAs to vacant areas, ask district to allocate more HSAs) and to respond to stock-outs “Our percentage of CCM-trained HSAs with stock-outs >7days in February, March and April was above action threshold so we took action to order drugs on time” – Msakambewa HF

17. Costing of Package 247 USD per health facility; 27 USD per HSA* *Includes cost of printing and calculators

18. Summary of findings Package helped to improve consistency between # cases recorded in HSA register and # cases reported in monthly report Routine data on iCCM treatments aggregated at the HF level, may not be as bad as people think Strength is that now “everyone can see the data”. HSAs and HF staff do use these data to improve the iCCM program at the grassroots level. The benchmarks and action thresholds were seen as helpful guidance. Turn-over and other management/health systems issues at the district level limit its potential effects Package is acceptable and feasible to implement at national level

19. Package Expansion Package is being scaled up to 23 other districts (with support from MOH, Save the Children’s RAcE, MICS and SSDI Services) Modifications: – Some clarification on indicator wording, definitions and targets done – Templates have been modified to meet number of cases HSAs are currently seeing The Ministry of Health through IMCI unit has taken a lead role in ensuring that the package is scaled up

20. Lessons from Expansion What is working well? The package has created interest amongst stakeholders including HSAs, SHSAs, and DHMTs (need to sustain the momentum) Reporting levels and quality of data have improved Improved community participation Aids supervision The package is simple and is not seen as additional work What are the challenges? Inadequate resources seems to be affecting scale up (In some districts only TOT was conducted) Delay in distributing necessary material e.g. templates and calculators has resulted in some HSAs forgetting what they learnt Some supervisors are not using the designed implementation strength indicators summary sheet

21. Next Steps Exploring opportunities to: – Integrate successful elements of the DI package within iCCM training for HSAs and HSA supervisors – Include dashboards at district and national levels within the DHIS 2 – Improve tracking of actions and problem solving – Include displays for other services by HSAs (newborn, family planning) – Include changes in iCCM service provision e.g. mRDTs – Disseminate and advocate for package uptake at district level

22. Thank you Acknowledgements: This study was supported by the American people through the United States Agency for International Development (USAID) and its Translating Research into Action (TRAction). TRAction is managed by University Research Co., LLC (URC) under the Cooperative Agreement Number GHS-A- 00-09-00015-00. For more information on TRAction's work, please visit http://www.tractionproject.org/.http://www.tractionproject.org/ More information and reports from TRAction in Malawi: http://www.jhsph.edu/departments/international- health/centers-and-institutes/institute-for- international-programs/projects/traction/index.html

23. Baseline data quality assessment Assessment Objectives:  To assess data availability, completeness and quality  To explore the use of iCCM data in program management and decision making Conducted May 2012 in 2 districts – Dowa and Kasungu Random selection of 4 health centers + the hospital in each district Random selection of 4 HSAs per facility District staff involved in data collection and interpretation of findings

24. Major Strengths & Weaknesses Strengths: Well-defined structure for reporting with clear deadlines & expectations Reporting forms easy to use System of quality checks in place Good levels of reporting and completeness Reasonable levels of consistency with a few exceptions. HSAs meet regularly with health center staff/community leaders Weaknesses: iCCM data not kept at health center or HSA level Concerns with data quality reported by participants Limited to no training on data use, processing and interpretation. Data use in decision making is low, mostly top-down approach. Supervision checklist was not yet in use (Kasungu) and low in Dowa. Use of mentoring checklist is low. Staffing barriers and high turnover