1. Common Anomaly Detection Platform
Tony Xing
Senior Product Microsoft

2. Bio Senior Product Manager of Shared Data team @ Microsoft
Data quality and anomaly detection
NRT datasets
Data Ingestion
Senior Product Manager of Skype Data Microsoft
Real time analytics
Anomaly detection
Cross platform SDKs

3. Agenda Context Anomaly detection 101 Problem statement
Design principles
How it works
Algorithms
Challenges and future work

4. Context

5. Shared Data

6. Shared Data
Our Vision
​Shared Data has a vision to have one common SDK, a data bus to allow easy sharing all of OPG & AI&R's data streams in real time, and a set of common data sets in Cosmos (and Spark) ​At a high level we plan to have a pluggable architecture, where we expect many processors and solutions to share a common data backbone.  OPG & AI&R teams use multiple tools to manage and use their data, given that, the Shared Data architecture is designed to align with OPG & AI&R customer needs and the quickly evolving landscape of 3rd party and open-source data tools

7. Anomaly Detection 101

8. What is Anomaly Detection
Anomaly detection is the identification of items, events or observations which do not conform to an expected pattern or other items in a dataset
Widely used in
System health monitoring
Business metric monitoring
Application performance monitoring
“My current value is not what it should be as of right now”

9. Rule setting vs. automated
Automate the process of finding outliers across the streams of data with a time dimension

10. Problem Statement
Manual rule setting is impossible for large number of time series
Single AD algorithm can not fit all signal types
Precision vs. recall
Analysis and diagnostics when issues happen
Near real time detection
Scalable
Customers needs flexibility in plugging in different sources

11. What is CAP
One stop shop for metric monitoring, analysis and diagnostics
Key capabilities
Automation: Full automation from creating rules to detection without human intervention
Extensibility: Can plug in new data sources and anomaly detection algorithms.
Scalability & real time: linear scale out Azure service
Finer Granularity: support time series AD in hour/minute level
REST APIs: REST APIs available for all operations. Allow easy integration into other product experience
Algorithm tuning: allow easier tuning of algorithm

12. How it works – Automation
5/12/2018
How it works – Automation
Onboarding
Helps data owners register the incoming streams
Creating rules & detecting
The creating rules component creates detection rules which are then used by the detecting component to detect potential anomalies
Contain machine learning and statistical analysis algorithms
Alerting
Once anomalies are found, alerting component will send anomaly info to the data owner
© 2014 Microsoft Corporation. All rights reserved. Microsoft, Windows, and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries. The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

13. How it works - Extensibility
Defined a generic interface of training and detection
Each algorithm provider would implement per defined interface
For example for each data point, we expect following from algorithm providers
Whether it is an anomaly
What is the predicted/expected value by algorithm
What is the suggested lower bound
What is the suggested upper bound
Confidence level …

14. How it works – Extensibility

15. How it works - Scalability

16. Algorithms Intro
Based on the initial customer usage, we start with those algorithms and make the generic interface based on the characteristic of them

17. Algorithm - Service Insider
5/12/2018
Algorithm - Service Insider
Good in time series with periodical pattern
Holt-Winters algorithm - Train model and predict
Improvements for robustness:
Use Median Absolute Deviation (MAD) to get robust estimation
Handling for data missing and noise (e.g., data smoothing)
Automatically capture the slow and regular trend and seasonal pattern
GLR (Generalized Likelihood Ratio) - Used to detect anomalies
Improvements
Floating Threshold GLR, to dynamically adjust the model using the new input data
Outlier removal for noisy data
© 2014 Microsoft Corporation. All rights reserved. Microsoft, Windows, and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries. The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

18. Machine Learning, Analytics & Data Science Conference
5/12/2018 2:25 PM
Other Improvements
Automatic detection of time series types (seasonal/non-seasonal)
Automatic detection of seasonality/trend, instead of manual setting
Add the feedback channels for end users to intuitively tune the algorithms
The automatic detection use an integration of FFT (Fast Fourier Transformation) and LOESS (locally weighted scatterplot smoothing)
FFT directly translate the time series into frequency domain, and it helps to capture the strongest frequency signal, but it could have bunch of noisy signals in the results
LOESS does not require the specification of a function to fit a model to all of the data, it is flexible, and it could help to eliminate some noisy results
Integrate the pure unsupervised statistics with semi-supervised feedback tuning
Traditional statistics approaches (like Holt-Winters and ARIMA) are commonly used, it is unsupervised and no labeling required; but it won’t reflect users’ intent (too loose/strict)
When users explicitly adjust on some points, their feedback are used as semi-supervised labels to retrain the parameters to fit more on users’ intent
© 2014 Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

19. Azure ML - Exchangeability Martingale
Good in detecting slow upward/downward trend, spike and dip, change in dynamic range
General framework for online change detection in time series
Has the property we are interested in changed in distribution?
User specifies meaning of “new value strangeness” given history
At each time t we receive a new value
Add it to the history. For each item i in the history
s[i] = strangeness function of (value[i], history)
Let p[t] = (#{i: s[i] > s[t]}+ r*#{i: s[i]==s[t]})/N, where r is uniform in (0,1)
Uniform r makes sure p is uniform

20. Azure ML - Exchangeability Martingale

21. Algorithm – Exponential Smoothing

22. Result Evaluation of exponential smoothing
In some cases with periodical signal with trending, many false positives could be generated

23. Result Evaluation - ServiceInsider

24. Result Evaluation – EM

25. Result Evaluation – ES based

26. Result Evaluation – ServiceInsider and Azure ML

27. Challenges and Future Work
Real time vs. accuracy
Automated handling of data pattern change
Easy tuning or usage of different algorithms

28. Real time vs. Accuracy Real time vs. Accuracy
5/12/2018
Real time vs. Accuracy
Real time vs. Accuracy
Some data streams are not stable from the perspective of data point latency
© 2014 Microsoft Corporation. All rights reserved. Microsoft, Windows, and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries. The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

29. Data Pattern Change

30. Easy Tuning
Tuning the algorithm parameters to achieve right detection precision and recall is a pain to the users
Service insider 2 parameters
EM based: 7 parameters
ES based: 3 parameters
Creative UI to hide those details
Do without human tuning at all!

31. Questions!