Evaluation of surveillance systems Günter Pfaff 2009/10 / Viviane Bremer 2008 / Preben Aavitsland / FETP Canada Günter Pfaff 17th EPIET Introductory Course Lazareto, Menorca, Spain September – October, 2011

The surveillance loop Health Care SystemPublic Health Authority Event Data Information Intervention (Feedback) Reporting Analysis & Interpretation Decision

Importance of evaluation Obligation Does the system deliver? Credibility of public health service In reality Often neglected Basis for improvements Learning process EPIET training objective Do not create one until you have evaluated one

Detect trends? Epidemics? Provide estimates of morbidity and mortality? Identify risk factors? Stimulate epidemiologic research? Assess effects of control measures Lead to improved clinical practice? Lead to new/improved control measures? Lead to better advocacy and increased funding? Does the surveillance system…

Footnote Simplicity Flexibility Acceptability Data quality Sensitivity and Predictive value positive (PvP) Capture-recapture Representativeness Timeliness Criteria to look at CDC guidelines

Footnote Simplicity As simple as possible while meeting the objectives Structure Information needed Number and type of sources Training needs Number of information users Functionality Data transmission System maintenance Data analysis Information dissemination

Components of system Population under surveillance Period of data collection Type of information collected Data source Data transfer Data management and storage Data analysis: how often, by whom, how Dissemination: how often, to whom, how Confidentiality, security

Flowchart (HIV in Norway)

Footnote Flexibility Ability of the system to accommodate changes New event to follow-up New data about an event New sources of information

Acceptability Willingness to participate in the system Participation (%) of sources Refusal (%) Completeness of report forms Timeliness of reporting

Footnote Acceptability Factors influencing the willingness to participate Public health importance Recognition of individual contribution Responsiveness to comments/suggestions Time burden Legal requirements Legal restrictions

Data quality Completeness Proportion of blank / unknown responses Simple counting Validity True data? Comparison Records inspection Patient interviews...

Completeness of information

Sensitivity = reported true cases total true cases = proportion of true cases detected

Disease Notified Total sickTotal not sick Total not notified Total notified True - False + False - True + - Sensitivity = True + / Total sick Specificity = True - / Total Not sick PVP = True+ / Total notified Sensitivity - -

Sensitivity versus specificity The tiered system: confirmed, probable, possible

Frequent "false-positive" reports Inappropriate follow-up of non-cases Incorrect identification of epidemics Wastage of resources Inappropriate public concern (credibility) Footnote Consequences of low PvP

Measuring sensitivity Find total true cases from other data sources medical records disease registers special studies Capture-recapture study

Capture-recapture Used for counting total number of individuals in population using two or more incomplete lists Originally used in wildlife counting (birds, polar bears, wild salmon…)

Uses in epidemiology Estimate prevalence or incidence from incomplete sources Evaluate completeness of a surveillance system

Principles Two/more sources of cases with disease Lists, registries, observations, samples Estimate total number in the source population (captured and uncaptured) from the numbers of captured in each capture

Assumptions 1.The population is closed No change during the investigation 2.Individuals captured on both occasions can be matched No loss of tags 3.For each sample, each individual has the same chance of being included Same catchability 4.Capture in the second sample is independent of capture in the first The two samples are independent, p YZ = p Y p Z

Seaworld Oberhausen, August 2010 Daddy, how many fish are in the aquarium?

Your options as a scientist Dont answer => Expect repeat question Answer something=> How do you know? Consult an expert Estimate yourself

Meet the expert - Pulpo Paul Has nine brains and three hearts Managed to predict all German games during the 2010 Football World Cup right Predicted accurately the finale Netherlands-Spain Binomial distributions only

Two-source model Source Z Source Y bac x=? N=? N= a + b + c + x Z1Z1 Y1Y1

Two-source analysis N = Y 1 Z 1 / a Sensitivity of YYsn = Y/N = (a+c)/N Sensitivity of ZZsn = Z/N = (a+b)/N

How many persons are in the EPIET 2011 Introductory Course? Isla del Lazareto, Dinner on Monday, 10 October 2011 – Case definiton: Countable heads

How many persons are in the EPIET 2011 Introductory Course? Isla del Lazareto, Dinner on Monday, 10 October 2011 – Case definiton: Countable heads, n= Hand does not meet our case definition This is our first view

How many persons are in the EPIET 2011 Introductory Course? Isla del Lazareto, After Dinner Tutorial on Monday, 11 October 2011 – Case definition: Countable heads, N= This is our second view

How many participants at the course? Capture: Source View #1 Recapture: Source View #2 Estimations Assumptions hold?

Number of participants N = 33 * 18 / 13 = 47 Sensitivity of View # 1Sn1 = 33/47 = 70.2% Sensitivity of View # 2Sn2 = 18/47 = 38.3% YesNo Yes1320View #1 = 33 No 5 x View # 2 = 18 N = x Source View #2 – After Dinner Tutorial Source View #1 Dinner

How many persons are in the EPIET 2011 Introductory Course? Isla del Lazareto, After Dinner Tutorial on Monday, 11 October 2011 – Case definition: Countable heads, N= This is our second view (revisited) + 2

Number of participants N = 33 * 20 / 13 = 51 Sensitivity of View # 1Sn1 = 33/51 = 64.7% Sensitivity of View # 2Sn2 = 20/51 = 39.2% YesNo Yes1320View #1 = 33 No 7 x View # 2 = 20 N = x Source View #2, revised – After Dinner Tutorial Source View #1 Dinner

So, just how many are there? Isla del Lazareto, Katharinas Lecture, Monday, 11 October 2010 – Case definition: Persons in room, N= off screen

The problem with the X: Finding a comprehensive view

Assumptions may not hold 1.The population is closed -Usually possible 2.Individuals captured on both occasions can be matched -OK if good recording systems 3.For each sample, each individual has the same chance of being included -Rarely true 4.Capture in the second sample is independent of capture in the first -Rarely true

Sources are independent (most important condition) Being in one source does not influence the probability of being in the other source OR > 1 (positive dependence): underestimates N OR < 1 (negative dependence): overestimates N

Dependent sources Estimation of number of IVDU in Bangkok in 1991 (Maestro 1994) Two sources used: Methadone programme (April – May 1991) Police arrests (June – September 1991) Methadone Need for drugs Probability of being arrested = negative dependence, overestimation of N

Usefulness of capture-recapture If conditions are met Great potential to estimate population size by using incomplete sources Cheaper than exhaustive registers or full counting Two sources Impossible to quantify extent of dependence Multiple sources Can adjust for dependence and variable catchability

Examples of capture-recapture STDs in The NL Reintjes et al. Epidemiol Infect 1999 Foodborne outbreaks in France Gallay et al. Am J Epidemiol 2000 Pertussis in England Crowcroft et al. Arch Dis Child 2002 Invasive meningococcal disease Schrauder et al. Epidemiol Infect 2006

Footnote Representativeness A representative system accurately describes Occurrence of a health event over time Distribution in the population by place and time Difficult to determine Compare reported events with actual events Characteristics of the population Natural history of condition, medical practices Multiple data sources Related to data quality, bias of data collection, completeness of reporting

Timeliness Disease onset Disease diagnosed Reporting of event Action taken Analysis and interpretation Clinician, labs Public Health Authorities

Sensitivity Representativeness Predictive value positive Timeliness Acceptability Flexibility Simplicity Cost Buehlers balance of attributes

Recommendations of evaluation Continue Revise Stop If revising Increase participation rate of sources Simplify notification Increase the frequency of feedback Broaden the net... Activate data collection Improving surveillance systems

Surveillance is like archeology of the immediate past – It requires your responsible imagination of an invisible reality. Carnunthum, Austria Corollary

Thank you!

Literature CDC. Updated guidelines for evaluating public health surveillance systems. MMWR 2001; 50 (RR-13): 1-35 WHO. Protocol for the evaluation of epidemiological surveillance systems. WHO/EMC/DIS/97.2. Romaguera RA, German RR, Klaucke DN. Evaluating public health surveillance. In: Teutsch SM, Churchill RE, eds. Principles and practice of public health surveillance, 2nd ed. New York: Oxford University Press, 2000.

Reading on capture-recapture Wittes JT, Colton T and Sidel VW. Capture-recapture models for assessing the completeness of case ascertainment using multiple information sources. J Chronic Dis 1974;27: Hook EB, Regal RR. Capture-recapture methods in epidemiology. Methods and limitations. Epidemiol Rev 1995; 17: International Working Group for Disease Monitoring and Forecasting. Capture-recapture and multiple- record systems estimation I: History and theoretical development. Am J Epidemiol 1995;142: International Working Group for Disease Monitoring and Forecasting. Capture-recapture and multiple- record systems estimation II: Applications in human diseases. Am J Epidemiol 1995;142: