Presentation is loading. Please wait.

Presentation is loading. Please wait.

Statistical Methods for Alerting Algorithms in Biosurveillance Howard S. Burkom The Johns Hopkins University Applied Physics Laboratory National Security.

Similar presentations

Presentation on theme: "Statistical Methods for Alerting Algorithms in Biosurveillance Howard S. Burkom The Johns Hopkins University Applied Physics Laboratory National Security."— Presentation transcript:

1 Statistical Methods for Alerting Algorithms in Biosurveillance Howard S. Burkom The Johns Hopkins University Applied Physics Laboratory National Security Technology Department Washington Statistical Society Seminar February 3, 2006 National Center for Health Statistics Hyattsville, MD

2 ESSENCE: An Electronic Surveillance System for the Early Notification of Community-based Epidemics Monitoring health care data from ~800 military treatment facilities since Sept Evaluating data sources –Civilian physician visits –OTC pharmacy sales –Prescription sales –Nurse hotline/EMS data –Absentee rate data Developing & implementing alerting algorithms ESSENCE Biosurveillance Systems

3 Outline of Talk Prospective Syndromic Surveillance: introduction, challenges Algorithm Evaluation Approaches Statistical Quality Control in Health Surveillance Data Modeling and Process Control Regression Modeling Approach Generalized Exponential Smoothing Comparison Study Summary & Research Directions

4 Required Disciplines: Medical/Epi Medical/Epidemiological filtering/classifying clinical records => syndromes interpretation/response to system output coding/chief complaint interpretation

5 Required Disciplines: Informatics Information Technology surveillance system architecture data ingestion/cleaning interface between health monitors and system

6 Required Disciplines: Analytics Analytical Statistical hypothesis tests Data mining/automated learning Adaptation of methodology to background data behavior

7 Essential Task Interaction in Volatile Data Background Medical/Epidemiological filtering/classifying clinical records => syndromes interpretation/response to system output coding/chief complaint interpretation Information Technology surveillance system architecture data ingestion/cleaning interface between health monitors and system Analytical Statistical hypothesis tests Data mining/automated learning Adaptation of methodology to background data behavior

8 The Multivariate Temporal Surveillance Problem Multivariate Nature of Problem: Many locations Multiple syndromes Stratification by age, gender, other covariates Surveillance Challenges: Defining anomalous behavior(s) –Hypothesis tests--both appropriate and timely Avoiding excessive alerting due to multiple testing –Correlation among data streams –Varying noise backgrounds Communication with/among users at different levels Data reduction and visualization Varying Nature of the Data: Scale, trend, day-of-week, seasonal behavior depending on grouping:

9 Data issues affecting monitoring –Statistical properties Scale and random dispersion –Periodic effects Day-of-week effects, seasonality –Delayed (often variably) availability in monitoring system –Trends: long/short term: many causes, incl. changes in: Population distribution or demographic composition Data provider participation Consumer health care behavior Coding or billing practices –Prolonged data drop-outs, sometimes with catch-ups –Outliers unrelated to infectious disease levels Often due to problems in data chain Inclement weather Media reports (example: the Clinton effect) Most suitable for modeling without data-specific information

10 Forming the Outcome Variable: Binning by Diagnosis Code

11 Rash Syndrome Grouping of Diagnosis Codes

12 Chief Complaint Query Simulated Data

13 Dynamic Detection Simulated Data Dynamic Detection

14 Example with Detection Statistic Plot Threshold Injected Cases Presumed Attributable to Outbreak Event

15 Comparing Alerting Algorithms Criteria: Sensitivity –Probability of detecting an outbreak signal –Depends on effect of outbreak in data Specificity ( 1 – false alert rate ) –Probability(no alert | no outbreak ) –May be difficult to prove no outbreak exists Timeliness –Once the effects of an outbreak appear in the data, how soon is an alert expected?

16 Modeling the Signal as Epicurve of Primary Cases Need data epicurve: time series of attributable counts above background Plausible to assume proportional to epidemic curve of infected Sartwell lognormal model gives idealized shape for a given disease type Sartwell, PE. The distribution of incubation periods of infectious disease. Am J Hyg 1950; 51:310:318

17 Signal Modeling: Realizations of Smallpox Epicurve maximum likelihood epicurve Each symptomatic case a random draw

18 Assessing Algorithm Performance Sensitivity/Specificity as a function of threshold: Receiver Operating Characteristic (ROC) Timeliness/Specificity as a function of threshold: Activity Monitor Operating Characteristic (AMOC) False Alert Rate (1 – specificity) Detection Probability (sensitivity) False Alert Rate (1 – specificity) Timeliness Score (e.g. Mean or Median Time to Alert) threshold Summary processing: measure dependence of sensitivity or timeliness on false alert rate (ROC or AMOC curves or key sample values at practical rates)

19 Detection Performance Comparison

20 Quality Control Charts and Health Surveillance Benneyan JC, Statistical Quality Control Methods in Infection Control and Hospital Epidemiology, Infection and Hospital Epidemiology, Vol. 19, (3) Part I: Introduction and Basic Theory Part II: Chart use, statistical properties, and research issues 1998 Survey article gives 135 references Many applications: monitoring surgical wound infections, treatment effectiveness, general nosocomial infection rate, … Monitoring process for special causes of variation Organize data into fixed-size groups of observations Look for out-of-control conditions by monitoring mean, standard deviation,… General 2-phase procedure: Phase I: Determine mean, standard deviation of process from historical in-control data; control limits often set to 3 Phase II: Apply control limits prospectively to monitor process graphically

21 Adaptation of Traditional Process Control to Early Outbreak Detection On adapting statistical quality control to biosurveillance: Woodall, W.H. (2000). Controversies and Communications in Statistical Process Control, Journal of Quality Technology 32, pp Researchers rarely…put their narrow contributions into the context of an overall SPC strategy. There is a role for theory, but theory is not the primary ingredient in most successful applications. Woodall, W.H. (2006, in press). The Use of Control Charts in Health Care Monitoring and Public Health Surveillance In industrial quality control it has been beneficial to carefully distinguish between the Phase I analysis of historical data and the Phase II monitoring stage It is recommended that a clearer distinction be made in health-related SPC between Phase I and Phase II … Does infectious disease surveillance require an ongoing Phase I strategy to maintain robust performance?

22 Statistical Process Control in Advanced Disease Surveillance Key application issues: Background data characteristics change over time –Hospital/clinic visits, consumer purchases not governed by physical science, engineering –But monitoring requires robust performance: algorithms must be adaptive Target signal: effect of infectious disease outbreak –Transient signal, not a mean shift –May be sudden or gradual

23 The Challenge of Data Modeling for Daily Health Surveillance Conventional scientific application of regression –Do covariates such as age, gender affect treatment? Does treatment success of differ among sites if we control for covariates? –Studies use static data sets with exploratory analysis In surveillance, we model to predict data levels in the absence of the signal of interest –Need reliable estimates of expected levels to recognize abnormal levels –Data sets dynamiccovariate relationships change

24 The Challenge of Data Modeling for Daily Health Surveillance, contd Modeling to generate expected data levels –Predictive accuracy matters, not just strength of association or overall goodness-of-fit –For a gradual outbreak, recent data can train model to predict abnormal levels Alerting decisions based on model residuals Residual = observed value – modeled value Conventional approach: –assume residuals fit a known distribution (normal, Poisson,…) –hypothesis test for membership in that distribution For surveillance, can also apply control-chart methods to residuals

25 Monitoring Data Series with Systematic Features Problem: How to account for short-term trends, cyclic data features in alerting decisions? Approaches –Data Modeling Regression: GLM, ARIMA, others & combinations –Signal Processing LMS filters and wavelets –Exponential Smoothing: generalizes EWMA

26 Example: OTC Purchasing Behavior Influenced by Many Factors Example: Tracking Daily Sales of Flu Remedies Loglinear Regression Log(Y) = d + 7 t h + 10 w + 11 p + day of week (6 indicators) harmonic (seasonal) sales promotion (indicator) linear trend weather (temp.) deviation (Poisson dist.) daily count of anti-flu sales

27 Modeling emergency department visit patterns for infectious disease complaints: results and application to disease surveillance Judith C Brillman, Tom Burr, David Forslund, Edward Joyce, Rick Picard and Edith Umland BMC Medical Informatics and Decision Making 2005, 5:4, pp Modeling visit counts on day d: Let S(d) = log ( visits(day d) + 1 ), the started log S(d) = [Σ i c i × I i (d)] + [c 8 + c 9 × d] + [c 10 × cos(kd) + c 11 × sin(kd)], k = 2π / c 1 -c 7 day-of-week effects c 9 long-term trend c 10 -c 11 seasonal harmonic terms Training period:3036 days ~ 8.33 years Test period: 1 year Recent Surveillance Method Based on Loglinear Regression

28 Brillman et. al. Figure 1

29 EWMA Monitoring Exponential Weighted Moving Average Average with most weight on recent X k : S k = S k-1 + (1- )X k, where 0 < Test statistic: S k compared to expectation from sliding baseline Basic idea: monitor (S k – k ) / k Added sensitivity for gradual events Larger means less smoothing

30 EWMA Concept & Smoothing Constant Brown, R.G. and Meyer, R.F. (1961), "The Fundamental Theorem of Exponential Smoothing," Operations Research, 9, Exponential smoothing represents an elementary model of how a person learns: x k = x k-1 + x k - x k-1 )where 0 < For the smoothed value S k, S k = S k-1 + (1- )X k, The variance of S k is S X So a smaller is preferred because it gives a more stable S k ; values between 0.1 and 0.3 often used But Chatfield: changes in global behavior will result in a larger optimal

31 Generalized Exponential Smoothing Forecast Function: where: m j = level at time j, b j = trend at time j, c j = periodic multiplier at time j s = periodic interval k = number of steps ahead and m j, b j, c j are updated by exponential smoothing Annex_B_The_Holt-Winters_forecasting_method.pdf Holt-Winters Method: modeling level, trend, and seasonality

32 Holt-Winters Updating Equations Updating Equations, multiplicative method: Level at time t: Slope at time t: Periodic multiplier at time t: And choice of initial values m 0, b 0, c 0,…c s-1 should be calculated from available data

33 Forecasting Local Linearity: Automatic vs Nonautomatic Methods Chatfield, C. (1978), "The Holt-Winters Forecasting Procedure," Applied Statistics, 27, Chatfield, C.and Yar, M. (1988), "Holt-Winters Forecasting: Some Practical Issues, " The Statistician, 37, Modern thinking favors local linearity rather than global linear regression in time… Local linearity is also implicit in ARIMA modelling… –Simple EWMA ~ ARIMA(0,1,1) –EWMA + trend ~ ARIMA(0,2,2) –Multiplicative Holt-Winters has no ARIMA equivalent Practical considerations rule out [Box-Jenkins] if there are insufficient observations or …expertise available –Box-Jenkins… requires the user to identify an appropriate… [ARIMA] model For fair comparison of H-W to B-J, have both automatic or nonautomatic. Assertion: The simplicity of H-W permits easier classification, requiring less historic data. Can an automatic B-J give robust forecasting over a range of input series types?

34 Regression vs Holt-Winters Ongoing study with Galit Shmueli, U. of MD Sean Murphy, JHU/APL 30 time series, 700 days data 5 cities 3 data types 2 syndromes Respiratory: seasonal & day-of-week behavior Gastrointestinal: day-of-week effects

35 Temporal Aggregation for Adaptive Alerting baseline interval Used to get some estimate of normal data behavior Mean, variance Regression coefficients Expected covariate distrib. -- spatial -- age category -- % of claims/syndrome guardband Avoids contamination of baseline with outbreak signal Data stream(s) to monitor in time: Counts to be tested for anomaly Nominally 1 day Longer to reduce noise, test for epicurve shape Will shorten as data acquisition improves test interval

36 Candidate Methods 1. Global loglinear regression of Brillman et. al. 2. Holt-Winters exponential smoothing fixed sets of smoothing parameters for data: with both day-of-week & seasonal behavior with only day-of-week behavior 3. Adaptive Regression Log(Y) = d + 7 t + 8 hol + 9 posthol + 56-day baseline, 2-day guardband 1-6 = day-of-week indicator coefficient 7 = centered ramp coefficient 8 = coefficient for holiday indicator 9 = coefficient for post-holiday indicator 1-day ahead and 7-day-ahead predictions

37 Respiratory Visit Count Data --- Data --- Holt-Winters --- Regression --- Adaptive Regr. All series display this autocorrelation; good test for published regression model

38 GI Visit Count Data --- Data --- Holt-Winters --- Regression --- Adaptive Regr.

39 Stratified Residual Comparisons --- Data --- Holt-Winters --- Regression --- Adaptive Regr.

40 Mean Residual Comparison When mean residuals favor regression, difference is small, and this difference results from largest residuals If the holiday terms in adaptive regression are removed, H-W means uniformly smaller

41 Median Residual Comparison

42 Residual Autocorrelation Comparison --- Data --- Holt-Winters --- Regression --- Adaptive Regr.

43 Residual Autocorrelation Comparison 1-Day Ahead Predictions

44 Residual Autocorrelation Comparison 7-Day Ahead Predictions

45 Summary Data-adaptive methods are required for robust prospective surveillance Appropriate algorithm selection requires an automated data classification methodology, often with little data history Statistical expertise is required to manage practical issues to maintain required detection performance as datasets evolve: –stationarity (causes rooted in population behavior, evolving informatics, others) –late reporting –data dropouts

46 Research Directions Classification of time series for automatic forecasting –Easier for Holt-Winters than for Box-Jenkins? –Determining reliable discriminants: Autocorrelation coefficients Simple means/medians Goodness-of-fit measures –How little startup data history required? Most effective alerting algorithm using residuals, given signal of interest –Apply control chart to residuals? –Need to detect both sudden, gradual signals –Detection performance constraints: Minimum detection sensitivity Maximum background alert rate

Download ppt "Statistical Methods for Alerting Algorithms in Biosurveillance Howard S. Burkom The Johns Hopkins University Applied Physics Laboratory National Security."

Similar presentations

Ads by Google