1. Department of Control of Neglected Tropical Diseases, WHO-HQ
Coverage Evaluation for Preventive Chemotherapy Description of the method
Department of Control of Neglected Tropical Diseases, WHO-HQ

2. Rationale: Background & Context
Achieving uniformly high treatment coverage in every treatment round is critical for the attainment of established NTD disease control and elimination goals.
monitoring treatment coverage using administrative / routinely reported data during mass drug administration (MDA) activities can be unreliable:
Incomplete tallying or reporting
poorly documented shifts in population
reliance on outdated census data
treatment of individuals outside the targeted age group or geographic area

3. Coverage Evaluation Definition:
are population-based surveys that are designed to provide precise statistical estimates of coverage that overcome many of the biases and errors that can undermine routinely reported coverage.
Objective:
To determine if the target coverage threshold has been met
To validate the reported coverage
Justification for preventive chemotherapy:
Sustained high coverage crucial to elimination and control
Reported coverage often inaccurate
Coverage evaluation surveys can save programs time and money
MDA round with low/below target coverage is not effective

4. Uses for Coverage Evaluation for preventive chemotherapy
Estimation of PC coverage – to obtain a precise estimate of PC coverage that can be compared with the target coverage threshold to determine if the MDA was effective.
Validation of reported coverage – to check the accuracy of the data recording and reporting system and take corrective actions where necessary.
Identifying reasons for non-compliance – by identifying common reasons for not swallowing the drugs, programme managers can improve social mobilization prior to the next MDA round.
Detecting problems with the supply chain and distribution systems – can identify clusters of individuals for whom the drugs were never offered and corrective action can be taken
Measuring coverage in specific population: subpopulations, e.g.. Rural vs. urban

5. Expert Consultation Meeting, 2012
The NTD Experience
post-MDA coverage evaluations acknowledged as very important but seldom conducted:
limited time and financial resources,
poor accessibility of households,
lack of available transportation
Lack of M&E staff
lack of expertise
Etc.
"…… identify a coverage survey sampling methodology that is feasible for national NTD programs to implement, produces valid point estimates of coverage, and can be standardized for use across the PC NTDs".
Expert Consultation Meeting, 2012

6. Review of methods
When coverage evaluations are conducted, Expanded Programme on Immunization (EPI) cluster-survey method most common
+ Practicality
+ Simplicity
+ Widespread use
- Falls short of probability sampling
- Results in biased estimates
- EPI program replacing method in favor of rigorous probability sampling
When coverage surveys are conducted a commonly used approach is the Expanded Program on Immunization’s 30-cluster survey (EPI). This EPI approach is recommended for conducting coverage surveys for lymphatic filariasis in the WHO Monitoring & Epidemiological Assessment Guidelines by the WHO. The EPI approach is touted for its practicality and simplicity and as a result has seen widespread use. However the EPI approach falls short of probability sampling and results in biased estimates. Because of this the Expanded Programme on Immunizations is replacing the EPI method in favor of a more rigorous probability sampling approach.
One of the decisions to come out of the 2013 meeting was the need to identify a standard approach for coverage evaluation surveys that is statistically rigorous while still feasible for programs to implement.
Updating current methods to a standard approach for coverage evaluations that is statistically rigorous while feasible for programs to implement

7. Coverage Evaluation Common Methodologies:
Overall Pros and Cons
EPI cluster-survey
Lot Quality Assurance Sampling (LQAS)
Probability Sampling with Segmentation (PSS)
Pros
Proven feasibility
Precise
Widely used
Feasible
Small sample size
Classification of Survey Areas
Unbiased
Cons
Biased results
EPI programme moving away from method
Must visit at least different villages
Imprecise
Cannot directly calculate coverage est.
Requires HH weighting or individual enumeration to avoid HH-size bias
Segmentation can be time consuming when maps not available
Difficult to segment large villages; EAs are much easier to use

8. Key features of coverage evaluation methods
EPI
cluster-survey
Lot Quality Assurance Sampling (LQAS)
Probability Sampling with Segmentation (PSS)
Feasible
Yes
Unbiased No
Depends/subjective
Clusters (EAs3) 30 95
Sample Size
15001
952
Precise
Preparation
List of EAs & pop
(optional) maps
List of EAs & pop.
5 Supervisory Areas
1Sample sizes will vary based on the parameters used, but typically range from 1, ,600 individuals
2Only one individual is selected per cluster, which is why the sample size and number of clusters are the same
3EAs = census Enumeration Areas

9. WHO/STAG NTD-WG M&E recommended: Probability sampling with segmentation (PSS)
Path
Stream
Road
House
School
1. Select 30 EAs using PPES
Segment 1
2. Divide EA into segments of ~50 House Holds (HH) ●
3. Randomly select 1 segment
4. Walk through segment and sample houses systematically according to the sampling interval
The figure shown here is a depiction of steps 2 – 4 (i.e., the steps undertaken once the survey team arrives in a selected EA/village).
PSS like the EPI survey is a 30 cluster survey; however within each cluster or village segmentation is used to reduce the number of household needing enumeration and then a systematic sample of households is taken from within the segment to select individuals such that everybody in the district has an equal probability of selection.
Divide the EA into segments of ~50 households
Randomly select 1 segment
Walk a path through that segment that passes by all households and sample households systematically according to a previously agreed sampling interval.
Segment 2
EA = census enumeration area PPES = probability proportional to estimated size HH = household

10. Main Difference Between EPI Method
1. Spin a bottle or pen in the center of village, then choose one random house between the center of the village and the edge of the village, in the direction of the spin, as the starting household
2. Visit nearest neighbor household, sampling all eligible within the household, until sample size is reached
From this starting household, the remaining households are chosen by going to the nearest neighbor household until the sample size is reached

11. Limitations: EPI
Do all individuals in a village have an equal probability of selection?
No, HH towards the center more likely to be selected. ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ●
In order for a method to be unbiased individuals must have an equal probability of selection, or at least known probability of selection. Is this true of the EPI method? No.
To illustrate this, suppose we are looking at a village with the dots throughout representing HHs. If we spin a bottle from the center of the town and then count the households in that direction until the end of the village you can see that, if we pick a constant width for our swath and choose a random direction, this method will favor the selection of HHs near the center of the village because the swath constitutes a larger and larger arc of circles drawn closer to the center.
This bias becomes very worrisome if households closer to the center of the village are also more likely to have received MDA, which has some plausible credibility
Note that a pie-shaped wedge would help resolve this concern for a circular village (though it would be tough to implement consistently); however it will not resolve this bias when the village is spread out along a road, as is quite common.
Can we calculate what these different sampling probabilities are? No, not in a meaningful way. ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ●

12. Limitations: EPI P(Individualij) = 30∗𝑝𝑜𝑝 𝑖 ` 𝑝𝑜𝑝 𝑖 ` 60 𝑝𝑜𝑝 𝑖
Do individuals in different clusters have the same probability of selection? No. The probability that an individual’s village is selected is based on its estimated size (PPES) but within the selected village (cluster) the probability that an individual is chosen is based on the actual village size.
P(Individualij) = 30∗𝑝𝑜𝑝 𝑖 ` 𝑝𝑜𝑝 𝑖 ` 𝑝𝑜𝑝 𝑖
𝑝𝑜𝑝 𝑖 ` 𝑝𝑜𝑝 𝑖
Estimated village size
What is the probability that individual j in EA i is selected? The probability that a individual’s village is selected is based on its estimated size, typically based on the most recent census estimates – we call this probability proportional to estimated size sampling – but within the selected village (or cluster), the probability that any one individual is selected is based on the actual village size.
So you can see that the probability that an individual is selected will depend on the ratio of the estimated size of their village compared with the true size of their village. This will result in a biased estimate. The more this ratio varies across villages, which is a reasonable assumption when you consider the heterogeneity of urban growth and migration throughout a district the greater the bias.
Actual village size

13. Probability Sampling with Segmentation (PSS)
Does everyone have an equal probability of selection? - YES
P(individual) =
𝑚 𝑠𝑒𝑔𝑚𝑒𝑛𝑡𝑠𝑖 ′ 𝑠𝑒𝑔𝑚𝑒𝑛𝑡𝑠𝑖 ′ 𝑠𝑒𝑔𝑚𝑒𝑛𝑡𝑠𝑖 ′ 𝑓
By using a set sampling fraction that is based on the expected household and segment size, every person has an equal probability of selection.
= 𝑚 𝑠𝑒𝑔𝑚𝑒𝑛𝑡𝑠𝑖 ′ 𝑓

14. Example of Results

15. Coverage Evaluation 3 methods common compared in 4 countries in 2015
PSS
PSS 3
EPI
PSS
LQAS
EPI
PSS
EPI
EPI 2
LQAS
LQAS
LQAS
Different district & team for each method
Same district & team for each method

16. Some surveys exceed the target coverage threshold, while others fail.
Target threshold (LF, STH)
The surveyed coverage shown estimates the true epidemiologic coverage, that is the number of people who reported ingesting the pills divided by the entire population (e.g., not limited to the eligible population). The first thing to note is that all the surveys in Burkina Faso and Malawi successfully exceeded the target threshold.
Note that for Uganda, only the PSS results are displayed because all the surveys (EPI, LQAS and PSS) took place in the same district and are capturing similar information

17. In 14 of the 16 surveys, reported coverage was greater than surveyed coverage.
Of the 16 country-method-drug combinations, in only two instances did the surveyed coverage exceed the reported coverage. This suggests that more often than not the coverage results that are reported are biased high, and in some cases appear to bear no correlation to the actual surveyed coverage results.

18. Comparative costs for Coverage Evaluation Surveys (2014 – 2015)
Country
EPI
LQAS
PSS
Days to complete
Cost
Burkina 18
$ 4,385 19
$ 4,816 17
$ ,525
Honduras 22
$ 1,867a 9
$ 1,167a
$ ,520a
Malawi 14
$ 4,113 10
$ 3,247 16
$ ,546
Uganda 23
$ 4,040 21
$ 3,835 26
$ ,535
AVERAGE
19.25
$ 3,601
14.75
$ 3,266
$ ,782

19. Acknowledgement of Contributors
MOH Burkina Faso
Roland Bougma
District health teams in Batie, Dano and Diebougou
MOH Uganda
Edridah Tukahebwa
Harriet Lwanga (RTI Envision, Uganda)
Survey teams from MOH
MOH Malawi
Square Mkwanda
District health teams in Balaka, Zomba, and Machinga
Secretary of Health Honduras
Reina Teresa
PAHO (Honduras & DC)
Rosa Elena Mejia & Romeo Montoya
Martha Saboya, Laura Catala-Pascual & Ana Morice
Pamela Mbabazi (WHO)
Michael Deming (formerly CDC)
Kristen Renneker (NTD-SC)
Abdel Direny (RTI Envision)
Funding
Bill & Melinda Gates Foundation
USAID

20. M&E tools for improving quality of data reported by national NTD programmes implementing preventive chemotherapy
Supervisor’s Coverage Tool
Coverage Evaluation Survey
Data Quality Assessment
Purpose:
To improve performance during current MDA
To validate reported coverage
(obtain a statistical point estimate)
To assess capacity of data management and reporting systems
Administrative level:
Supervision Area
(sub-district)
Implementation Unit (district)
National and/or District
Sample size:
20 people
>500 people
N/A
Sites visited:
1 supervision area
30 villages
12 service delivery points
Survey team:
Internal, self-assessment
External to programme
Internal and external to programme
Timing:
Within 2 weeks of MDA
Within 6 months of MDA
After MDA data have been reported (3-6 months post-MDA)
Cost:
$0 - $1,000 per SA
~$2,000 – $10,000 per district
$ 12,000 – 15,000 nationally
$1,000 – 2,500 per district
Duration:
<1 week
2-3 weeks
~ 2-3 weeks

21. WHO/NTD M&E Tool kit
Integrated NTD Database
DQA
SCT
Coverage Surveys