Assessment Training Session 9: Assessment Analysis Performance of Routine Information System Management (PRISM) Assessment Training Session 9: Assessment Analysis MEASURE Evaluation Date:

Session objectives Explain how to analyze PRISM assessment data Become familiar with the types of analyses performed in several countries Describe examples of strengthening interventions based on PRISM assessment findings

Some examples of analyses Percentage of districts, health facilities meeting data accuracy, completeness, and timeliness standards Average scores for use of information (scale of 0−100) Level of RHIS data use; percentage of districts, health facilities having active RHIS data use mechanism(s) Performance Diagnostic Tool Site- level average scores for critical HIS functions MAT Knowledge, confidence, and competence levels Personal staff feelings about motivation and reward systems Staff views on promoting the culture of information OBAT Proportion of sites having RHIS materials available and/or experiencing stockouts Mean numbers of specific health staff Proportion of staff who benefited from RHIS training Facility/Office Checklist

RHIS Performance Diagnostic Tool Analysis RHIS Performance Diagnostic Tool

RHIS performance summary table Data quality Use of information Weak Improving RHIS does not produce quality data and managers make little use of the information RHIS produces data of an acceptable quality but managers make little use of the information RHIS does not produce quality data but information is used by managers RHIS produces data of an acceptable quality and information is used by managers

Data quality analysis

Use of information analysis Describe information use at the site visited: Display of updated information Feedback mechanism(s) Discussion based on RHIS data What kinds of decisions are made? Which steps in the process of using information are performing well and which ones have weaknesses? Compilation of data, display, analysis, interpretation, discussion, decision making What kinds of decisions are made? Day-to-day program management Planning Budgeting Medical supply and drug management Human resource management Monitoring of key objectives, e.g., service improvement Identification of emerging issues, such as emerging epidemics

Use of information analysis

Level of use of information at the province level (Mali)

RHIS data displays and update status at the province level (Mali)

Diagnostic tool results (Ivory Coast) RHIS performance Health facilities Districts 2008 2012 Status Data quality 43% 60% 40% 81% Data use 38% 44% 70% Regional level was not assessed in 2008; data quality was at 50% in 2012 for the regional level. Note: From 2009 to 2011, facilities were trained on the HIV data collection tool and not on data use.

Management Assessment Tool (MAT) Analysis Management Assessment Tool (MAT)

RHIS Management Assessment Identify which elements of RHIS management are performing well and which are exhibiting problems Governance RHIS operations Information infrastructure Human resources Financial resources Other resources RHIS operations: planning, quality standards, supervision Information infrastructure: information and communications technology (ICT) Other resources: RHIS materials, equipment, transportation

Data management processes Describe the strengths and weaknesses of data management processes at the site you visited: Data recording Data transmission Data processing and analysis Data presentation

Mean scores for the RHIS management functions (Mali)

Organizational and Behavioral Assessment Tool (OBAT) Analysis Organizational and Behavioral Assessment Tool (OBAT)

Perceived confidence levels and observed HIS task competence (Mali)

Perceived confidence and observed HIS task competence, by level (Ethiopia)

Facility/Office Checklist Analysis Facility/Office Checklist

Percentage distribution for the cluster Availability of resources at health facilities and woreda (Ethiopia) Resources Availability Percentage distribution for the cluster Health posts Health centers Hospitals Woreda health offices Computer 83 56 6 1 17 28 ≥ 2 11 100 66   Printer 86 58 32 67 52 Internet Yes 20 33 2 Electricity 3 A woreda is the equivalent of a district in Ethiopia.

Analysis Overview of PRISM results and interventions

Overview of the PRISM results HMIS overview and major findings (N= 6 provinces) INPUTS PROCESSES OUTPUTS Technical Factors Data collection forms, procedures 1 Information system integration 83 Computer software 100 IT complexity 67 Behavioral Factors HIS forms content knowledge 52 Data quality checking skill 84 HIS problem solving skill 27 HIS task competence 64 HIS task confidence 81 Motivation 71 RHIS processes Data transmission n/a Data completeness Data accuracy Data processing 83 Data analysis 100 Data presentation Feedback RHIS Performance Data quality M1 M2 M3 Complete-ness 83 95 100 Timeliness 17 Accuracy 63 75 85 Information use 26  Organizational factors Governance 50 Planning 42 Training 25 Supervision 58 Quality assurance 56 Finance 1 Information distribution 83 Culture of information promotion 0-59 60-89 90-100

Comparison of results - Example

RHIS strengthening interventions (Examples from Ivory Coast) Integrating HIV/AIDS indicators in the overall RHIS (Technical) Organizing trainings in the use of information, including problem-solving techniques, at district and health facility levels (Behavioral) Developing feedback bulletins for health offices at all levels (Organizational)

RHIS strengthening interventions (Examples from Liberia) Building staff capacity in the use of the DHIS 2 health information software and in data verification, data analysis, and data use (Technical, Behavioral) On-site mentoring of monitoring and evaluation (M&E) staff (Behavioral) Building organizational capacity by establishing, organizing, and implementing routine performance review forums (Organizational) Setting up data quality assurance mechanisms, such as desk review of data and data quality assessments (Technical)

How to access the PRISM Series This slide deck is one of nine in the PRISM Series Training Kit, which also includes a Participant’s Manual and a Facilitator’s Manual. Also in the PRISM Series is a Toolkit (the centerpiece of the series) and a User’s Kit. The PRISM Series is available in its entirety on MEASURE Evaluation’s website, here: https://www.measureevaluation.org/prism

MEASURE Evaluation is funded by the United States Agency for International Development (USAID) under the terms of Cooperative Agreement AID-OAA-L-14-00004. It is implemented by the Carolina Population Center, University of North Carolina at Chapel Hill, in partnership with ICF International; John Snow, Inc.; Management Sciences for Health; Palladium; and Tulane University. The views expressed in this presentation do not necessarily reflect the views of USAID or the United States government. www.measureevaluation.org