Health Monitoring and Evaluation MSc Students Assessment of M&E system for Malaria Prevention and Control Program in Jimma Zone Health Monitoring and Evaluation MSc Students 2008/2009 Class

Outline Background Problem statement Rationale of the study Objectives Methodology Findings Discussion Conclusion Recommendations Intervention

Background In (DTTP) different professionals are practicing the theoretical to translate the knowledge into practice in the community. This contribute to development of the community when identified gaps are implemented. M&E students worked on malaria prevention and control M&E system

Background…. Malaria prevention and control program has the following strategies: Case management Vector Control Epidemic prevention and control Supportive Education Human resource development Monitoring and evaluation.

Background cont… The scaling up of these strategies requires: Robust surveillance system Monitoring and Evaluation Health-information systems (HIS) at all levels from local to national level Therefore, M&E is fundamental for decision-making and Program improvement

Problem Statement Malaria is the major Public Health problem in Ethiopia Among the top ten leading cause of morbidity and mortality. 2005 report: 20% admissions 48% of out patient consultation 24.9% inpatient death Affects school attendance by 20% Contributes to 47% of the child death

Problem Statement In 2006, report showed that despite much progress in the fight against Malaria, still the M & E system for the program has several problems: Non-centralized malaria Data base system & lack of network Impaired data capturing and storage The Roll back Malaria follow-up survey was not conducted as intended Untimely and incompleteness of the reports

Problem Statement This has several impacts to the implementation of Malaria prevention & control: planning done using obsolete /no data at all, choice of the treatment options done without information collected from the same settings. The above situations also applies similar to Jimma Zone, Mana district.

Rationale of the study Managers at various levels need quality information to make programmatic decisions. Accurate and timely reporting to national authorities and donors is vital in order to secure continued funding The study aimed at assessing the factors which can be easily acted upon by local authorities.

Scope of the study M&E system- 12 components by WORLD BANK: Partnership, People and Planning Collection, Capturing and Verification of data Data for Decision Due to factors:- time for the DTTP, resource available and areas of M&E system which can be simply recuperated by means of the resources at the local level: components selected include

Scope of the study … routine program monitoring, supervision and data auditing, survey and surveillance M&E data base Using identical rationale, research, evaluation and learning practices was discarded form the survey

Objectives: General objective To asses the process of data collection, capturing and verification of Malaria prevention and control M&E system at Jimma Zone Specific objectives To asses timeliness and completeness of Malaria prevention and control monitoring reports To describe the frequency and coverage of surveys and surveillance.

Specific objectives … To assess how Malaria prevention and control data base operates to generate data To asses whether the supervision and data auditing system of malaria prevention and control is in accordance to the guideline

METHODOLOGY Study Design Cross sectional study. Period 13th April to 15th May . Quantitative and Qualitative methods Study population: Staffs involved in M&E system from Zonal Health Department, district health office, health center and health posts

Methodology… Study Sites Included: Jimma Zonal health Department, Jimma Town Health Office Manna district health office, Jimma Health Center Yebu Health Center, Biturie health post Gudeta bula Health post Jimma Higher2 Health Center Kito/sato Health Post Jiren Health Post

Methodology … Sampling technique Convenience: study sites including health facilities Purposive : health care workers (respondents) Sample size: A total of 22 health workers responded to the prepared tools for data collection

Methodology … Data collection The data collection process included: Reviews of documents and reports received from April 2008 to March 2009. Observation Semi-structured interviews

Methodology … Data analysis: Quantitave data entered into SPSS Version 16 Qualitative data analyzed using thematic approach

Methodology … Ethical consideration: Officially introduction letters to study sites Verbal consented pre requisite. Interviewee free to withdraw at any time .

Findings Demographic characteristics 22 interviews were conducted Respondents included 11 Jimma Town Health office 8 Manna District Health Offices respectively. 3 zonal health department Their working experience varies from 1 to 4 years.

Malaria Program Monitoring Supervision & Data Auditing Table 3: Provision of Training on M&E and presence of Plan for the M&E components District/Zone Malaria Program Monitoring Survey & Surveillance Supervision & Data Auditing M&E Database Jimma Zone HO Training Yes M&E Plan No Jimma Town HO Mana district HO

Routine data collection & reporting Expected – number of reports expected Submitted -number of reports received Completeness- proportions of the reports expected that were submitted

Timeliness Though no reliable data to establish the report timeliness rate, qualitatively it was identified that there was chronic problems of incompleteness, and untimely (timeliness) of the reports at the district level from the health facilities

Table 4: Number of Supervision planned and conducted in each for each strategic components by districts in the last 12 months (xpt Manna) Intervention strategies Jimma zone Jimma town Planed Conducted % Case mgt 4 100 ITN 1 25 IRS Env’tal mgt 2 50 Epidemic control

Table 5: Data auditing practice by District/Zone Data Auditing Plan Focal Person Trained Frequency Jimma town Yes No Not regular Jimma zone Quarterly Manna district

Table 8: Information on malaria survey by district/zone office Ever involved in survey Has fund for survey Feedback from conducting Survey Jimma zone Yes no No Manna district Jimma town

Data base All districts and zone have database in place. The JZHD and JTHO have both paper based and electronic database systems Manna district uses paper based database system. However, all health facilities commonly apply paper based database system

Discussions All the districts surveyed implements five strategic elements of MPC and is in line with the national guideline and Dr. Yemane’s 2009 Presentation. presence of M & E system for each of the elements as also studied by Chilundo et al 2004. monthly reports as the common reporting frequency in both districts, while Manna have reports that are submitted on a weekly basis.

Discussion … At manna, the weekly reports with (87%) incompleteness rate Qualitatively it was also identified that there were problems of incompleteness, untimely and underreporting of the reports at the district level from the health facilities.

Discussion … According to the Ethiopian HIMS 2008, the cut off point for the incompleteness of the reports is when 80% or greater of the expected reports to be submitted are submitted. Similar findings by MCP report 2006, untimely and incompleteness to be rampant problems across the country.

Conclusion Indeed, it is clear from this study that monitoring and evaluation system in Jimma zone is severely crippled in the area of data collection, capturing and verification. Challenges seem to vary with districts and also within different level of health services delivery system. The monthly reports is the frequent used mode of reporting,

Conclusion … The supervision of the health facilities is conducted in all districts; however the frequency varies between the districts and mostly is less than quarterly visits to the health facilities. Surveillance of Malaria cases which is done in majority (71%) of the surveyed facilities has at least one trained focal person in each of the facilities undertaking the activities.

Recommendations Case management reporting needs to be sustained The Weekly reports need to be improved in completeness as per the standard The supervision should be as per guidelines

Recommendations … The survey being conducted with in the district should involve the staff Efficient information management system is highly required for surveillance of disease and control program Process evaluation of the M&E system to involve all stakeholders and come up with explanations of the observed deficit in the system

Intervention Purpose of intervention Aimed on improving the capacity of health professional involved in data collection, capturing and verification. Monday, April 17, 2017 43

Intervention……… The group selected to intervene only with knowledge gap due to time and resource constraints. Manual for training were prepared, they included the following topics: Malaria prevention and control monitoring and evaluation process, Supportive supervision and data auditing/verification, Surveillance and survey, monitoring and evaluation Database management, and recording and reporting. Monday, April 17, 2017 44

Intervention…… Training methodology Power point presentation using LCD Training manual power point presentation materials were distributed to the participants Explanation using flip chart Group Discussion Monday, April 17, 2017

Intervention………. Training contents Objective of the study General findings of the study Monitoring and Evaluation Concepts Purpose Importance Basic Terminologies Monday, April 17, 2017

Training……. Process components of Monitoring and Evaluation Routine program monitoring Performance monitoring Methodologies of performance monitoring Tools of routine monitoring Surveillance and survey Concepts & Differences and similarities Purpose and uses, types of surveillance Key sources of malaria surveillance data Features of good surveillance system Monday, April 17, 2017

Training……… Supportive supervision and data auditing Concepts & components of supportive supervision Steps in conducting supervision Methodology of data auditing/verification Monitoring and Evaluation Database Data entry, Data collection and interpretation Relevance of database Importance of proper filing and keeping of database Monday, April 17, 2017

Results of Pre and post test. Pre and post test result of Jimma town training participants Code Pre-test result Post-test result Difference % change 1 53 75 22 29% 2 89 14 16% 3 38 65 27 42% 4 5 30 25 83% 60 37% 6 69 87 18 21% 7 50   NA 8 71 90 19 9 10 85 20 24% Average % change 34% Monday, April 17, 2017

Results.…… Pre and post test result of Manna district training participants Code Pre-test result Post-test result Difference % change 4 40 60 20 33% H2001 52 64 12 19% 1 24 28 54% 8 32 13% M9 80 56 70% 84 76% 7 16 0% GB1 50% GB2 88 73% Monday, April 17, 2017

Conclusion The result from Paired T-Test show that the differences in mean scores between the Pre-Test and Post Test are Statistically Significant t(8) = -3.541, p = .008. The test show that the chance of the mean occurring by chance alone is about 0.008 (or 0.8%).

Intervention……. Discussion during training Major problems which were discussed by participants included. Untimely reports Incomplete reports Inadequate skill on monitoring and evaluation Shortage of logistic supplies In appropriate format Lack of regular supervision and feedback Lack of graphical presentation of data Monday, April 17, 2017

intervention……….. Proposed solutions Timely submission of Reports Provision of the necessary logistics Managing the database properly Preparing checklist for supportive supervision Monday, April 17, 2017

Solution…….. Conduct regular supervision and feedback Appropriate and standardized report formats Monthly performance assessment of the work done at all level Utilization of tools of monitoring (e.g preparation of graphical data) Monday, April 17, 2017

Dissemination of final findings While discussing with the heads on final dissemination of findings, an agreed up on plan of action has been developed to be consumed by the offices for implementation. Monday, April 17, 2017

THE END