Assessment Implementation Performance of Routine Information System Management (PRISM) Assessment Training Session 8: Assessment Implementation MEASURE Evaluation Date:

Session objectives Describe the PRISM assessment implementation steps Understand the criteria for the adaptation of the PRISM Tools to the local context Understand the sampling method Become familiar with electronic PRISM data entry

PRISM implementation Pre-assessment steps Establish country leadership, buy-in, and stakeholder engagement/coordination Principle outputs: HIS Advisory Group Assessment management and monitoring Working Group Set priorities and plan for the assessment Principle products: HIS priorities in country context: health program focus (nation-wide vs selected provinces/states); desired unit of analysis (comprehensive vs focused) Customized PRISM Tools Assessment Plan: sampling unit, sampling frame, sample size; key informants; assessment schedule; composition of assessment team; assessment team training schedule

Steps to conducting an assessment PRISM implementation Steps to conducting an assessment Manage and monitor data collection Assemble a core team of data collectors and supervisors Train the data collectors Select sites to conduct the assessment; identify key informants to interview Inform the assessment sites and key informants (preferably through official letters/communication media) Arrange logistics for data collectors Use the tools in order (Overview, Performance Diagnostic, eRHIS Performance, MAT, Office/Facility Checklist, OBAT) Supervise data collection; monitor quality of data collected and/or data entered using an electronic data entry tool Principle product: Assessment data (raw data)

Post data-collection steps PRISM implementation Post data-collection steps Analyze and assess current RHIS performance Apply PRISM data analysis plan Prepare tables, charts, assessment score matrix, etc. Review, verify, and adjust (if anomalies/inconsistencies found) Principle product: PRISM Assessment Report – current RHIS performance and its major determinants; identifying major areas of attention Translate evidence into policy, strategy, and interventions RHIS performance improvement plan, policies Implement the plan and monitor progress

Sampling method Probability samples are better than nonprobability samples. Probability samples are randomly selected and produce unbiased estimates. Two commonly used sampling techniques are: Lot Quality Assurance Sampling (LQAS) Sample size of 19 for each supervisory area Five supervisory areas: total sample size of 95 Probability Proportion to Size (PPS) Minimum sample size of 100

What is LQAS? LQAS is a sampling method with flexibility, rapidity, and inexpensiveness – relative to other probability survey methods – to assess performance using a smaller sample size. A sampling method that can: Be used locally, at the level of a “supervision area,” to identify priority areas or indicators that are not reaching average coverage or an established benchmark Provide an accurate measure of coverage or health system quality at a more aggregate level (e.g., program catchment area or district) Produce data for local management decision making and for sharing information across supervision areas Be used for quality assurance using a “minimal sample”, “maximal security” principle: Most frequently used size = <20 per supervision area Larger sizes are seldom needed

What a sample of 19 can tell us A sample of 19 provides an acceptable level of error for making management decisions At least 92% of the time, it is good for: Identifying whether a coverage benchmark has been reached or whether a supervision area is below the average coverage of a program Setting priorities in a supervision area Setting priorities among supervision areas with large differences in coverage Deciding which are the higher performing supervision areas to learn from Deciding which are the lower performing supervision areas Distinguishing knowledge/practices that have high coverage from those with low coverage Samples larger than 19 have practically the same statistical precision as samples of 19. They do not result in better information and they cost more.

What a sample of 19 cannot tell us A sample of 19 is not good for: Calculating exact coverage in a supervision area (but can be used to calculate coverage for an entire program) Setting priorities among supervision areas with little difference in coverage

Burundi: 17 health provinces Mali: 8 health provinces Sampling examples Burundi Mali Burundi: 17 health provinces Country divided into 5 zones Random selection of 1 province in each zone (5 provinces) Selection of 2 districts per province: the provincial capital and 1 random district (10 districts) Random selection of another 5 districts across the 5 selected provinces (now 10+5=15 districts) Random selection of 9 health facilities per district and selection of the district hospital (9+1=10 facilities/district) Total of 150 health units selected Mali: 8 health provinces Selection of 6 provinces Selection of 2 districts per province: the provincial capital and 1 random district (12 districts) Random selection of another 6 districts across the 6 selected provinces (now 12+6=18 districts) Random selection of 9 health facilities per district and selection of the district reference center (9+1=10 facilities/district) Total 180 health units selected

PRISM Tools implementation

PRISM electronic data entry Options available for electronic data entry: SurveyCTO - a survey platform that allows for mobile data collection using phones, tablets, or computers All of the PRISM Tools have been adapted into electronic data collection forms in SurveyCTO Locally build data entry modules using Microsoft Excel, Google Sheets, or another platform Facilitator’s Note: If the country has decided to use SurveyCTO, refer to the guidance document for guidelines of how to upload form definitions onto SurveyCTO, customize the data entry forms, and download data files from SurveyCTO for further analysis. Demonstrate (on a big screen) the steps in using SurveyCTO for data collection. Encourage participants to ask clarifying questions during the demonstration.

How to access the PRISM Series This slide deck is one of nine in the PRISM Series Training Kit, which also includes a Participant’s Manual and a Facilitator’s Manual. Also in the PRISM Series is a Toolkit (the centerpiece of the series) and a User’s Kit. The PRISM Series is available in its entirety on MEASURE Evaluation’s website, here: https://www.measureevaluation.org/prism

MEASURE Evaluation is funded by the United States Agency for International Development (USAID) under the terms of Cooperative Agreement AID-OAA-L-14-00004. It is implemented by the Carolina Population Center, University of North Carolina at Chapel Hill, in partnership with ICF International; John Snow, Inc.; Management Sciences for Health; Palladium; and Tulane University. The views expressed in this presentation do not necessarily reflect the views of USAID or the United States government. www.measureevaluation.org