1. Forecast, Detect, Intervene: Anomaly Detection for Time Series. Deepak Agarwal Yahoo! Research

2. Outline Approach –Forecast –Detect –Intervene Monitoring multiple series –Multiple testing, a Bayesian solution. Application

3. Issues {y t } : univariate, regularly spaced time series to be monitored for anomalies, “novel events”, surprises prospectively. –E.g. query volume, Hang-ups, ER admissions Goal: A semi-automated statistical approach –Forecast accurately : good baseline model. –Detect deviations from baseline: sensitivity/specificity/timeliness –Baseline model adaptive: learn changes automatically Important in applications : better forecasts →fewer false +ve

4. Approach Three components : (West and Harrison, 1976) –Forecast: Bayesian version of Kalman filter –Detection : A new sequential algorithm –Intervention : correct baseline model.

5. Forecasting

6. Kalman filter Observation Equation –Conditional distribution of data given parameters State Equation –Evolution of parameters (states) through time Posterior of states, predictive distribution –Estimated online by recursive algorithm

7. OBSERVATION EQUATION

8. STATE EQUATION: What assumptions are appropriate for the “Truth”?

9. More general models Y t-1 YtYt xtxt X t-1 GtGt Y t-1 X t-1

10. Lemma: Bayes updating for Gaussian model

11. Kalman Filter update at time t:

12. Estimating Variance components

13. Detection

14. An existing method

16. Pitfalls of GS What if predictive not Gaussian? –Mixtures of Gaussians, Poisson etc Bayes factor: specify alternative explicitly –Large number of unspecified parameters –Require explicit model for each alternative

17. Our approach Normal scores derived from p-values –Good for continuous, approximately good for discrete, especially for large means. A sequential procedure with far less tweaking parameters. Our method has more power, we sacrifice on timeliness.

18. Sequential detection procedure At time t, we are in one of these regions: Acceptance region (A): The null model is true, the system is behaving as expected, no anomalies, start a new run. Rejection region (R) : The null model is not true, an anomaly is generated which is reported to the user and/or the forecasting model is reset. Start a new run. Continue (C): Don’t have enough data to reach a decision, keep accumulating evidence by taking another sample.

19. Detecting outliers and mean shifts

20. Detecting variance shifts

21. Gradual changes, auto correlated errors.

22. rPτ 3.8414.91 4.874.40 5.963.77 6.983.98 7.993.66 8.980.10 9.980.10 10.850.09 11.850.09

23. The sequential algorithm at time t

24. Blue:ours; red: Gargallo and Salvador(GS)

26. Intervention

27. Intervention to adjust the baseline. Outlier → A tail or rare event has occurred –Ignore points → short tail; more false +ve –Use points→ elongated tail, more false -ve A robust solution: ignore points but elongate tail –retain same prior mean, increase prior variance. – system adapts, re-initializing the monitor. Use the above for mean shifts and variance increase. Variance decrease: System stable, make prior tight. Slow changes : System under-adaptive, make prior vague.

28. Intervention strategy

29. No intervention, m=1

30. strong intervention, m=3

31. Example: Blue is data, yellow is forecast.

32. Multiple testing

33. Multiple testing: A Bayesian Approach. Monitoring large number of independent streams –testing multiple hypotheses at each time point –Need correction for multiple testing. Main idea: –Derive an empirical null based on observed deviations –Present analyst with interesting cases adjusting for global characteristics of the system. –We use a Bayesian approach to derive shrinkage estimates of deviations –the “shrunk” deviations automatically build in penalty for conducting multiple tests.

34. Bayesian procedure.

35. Experiment comparing multiple testing versus naïve procedure (threshold raw standardized residuals) Simulate K noise points N(0,1) (K=500,1000,..), 100 signal points from [2,11]U[-2,-11]. Adjust threshold of Bayesian residuals to match sensitivity of naive procedure. Compute False Discovery Rate (FDR) for both procedures.

36. FDR of naive and Bayesian procedures. The Bayesian method gets better with increase in number of time series. Calculations based on 100 replications. The differences are statistically significant.

37. Application

38. Goal: To find leading indicators of social disruption events in China before it gets reported in the mainstream media. Approach: Monitor the occurrence of a set of pre-defined patterns on a collection of Chinese websites (mainly news sites, government sites and portals similar to yahoo located in eastern China). Motivating Application (bio-surveillance).

39. English translation of some Chinese patterns being monitored

40. Notations and transformation.

41. Dotted solid lines: Days when reports appeared in mainstream media Dotted gray lines: Days when our system found spikes related to the reports that appeared later.

42. Rough validation using actual media reports. July 24 th : mystery illness kills 17 people in China, we noticed several spikes on July 17 th and 18 th alerting us on this. Sept 29 th and Dec 7 th : On Sept 29 th, news reports of China carving out emergency plans to fight bird flu and prevent it from spreading to humans. On Dec 7 th, a confirmed case of bird flu in humans reported. We reported several spikes on Sept 12 th and 14 th, Nov 2 nd, 7 th, 11 th, and 16 th mostly for the pattern influenza, flu, pneumonia, meningitis. On Nov 21 st, four big spikes on bf3.syd.com.cn on influenza, flu, pneumonia, meningitis; emergency, disaster, crisis; prevention and quarantine.

43. Questions?