Common Anomaly Detection Platform Tony Xing Senior Product Manager @ Microsoft

Bio Senior Product Manager of Shared Data team @ Microsoft Data quality and anomaly detection NRT datasets Data Ingestion Senior Product Manager of Skype Data team @ Microsoft Real time analytics Anomaly detection Cross platform SDKs

Agenda Context Anomaly detection 101 Problem statement Design principles How it works Algorithms Challenges and future work

Context

Shared Data

Shared Data Our Vision ​Shared Data has a vision to have one common SDK, a data bus to allow easy sharing all of OPG & AI&R's data streams in real time, and a set of common data sets in Cosmos (and Spark) ​At a high level we plan to have a pluggable architecture, where we expect many processors and solutions to share a common data backbone.  OPG & AI&R teams use multiple tools to manage and use their data, given that, the Shared Data architecture is designed to align with OPG & AI&R customer needs and the quickly evolving landscape of 3rd party and open-source data tools

Anomaly Detection 101

What is Anomaly Detection Anomaly detection is the identification of items, events or observations which do not conform to an expected pattern or other items in a dataset Widely used in System health monitoring Business metric monitoring Application performance monitoring “My current value is not what it should be as of right now”

Rule setting vs. automated Automate the process of finding outliers across the streams of data with a time dimension

Problem Statement Manual rule setting is impossible for large number of time series Single AD algorithm can not fit all signal types Precision vs. recall Analysis and diagnostics when issues happen Near real time detection Scalable Customers needs flexibility in plugging in different sources

What is CAP One stop shop for metric monitoring, analysis and diagnostics Key capabilities Automation: Full automation from creating rules to detection without human intervention Extensibility: Can plug in new data sources and anomaly detection algorithms. Scalability & real time: linear scale out Azure service Finer Granularity: support time series AD in hour/minute level REST APIs: REST APIs available for all operations. Allow easy integration into other product experience Algorithm tuning: allow easier tuning of algorithm

How it works – Automation 5/12/2018 How it works – Automation Onboarding Helps data owners register the incoming streams Creating rules & detecting The creating rules component creates detection rules which are then used by the detecting component to detect potential anomalies Contain machine learning and statistical analysis algorithms Alerting Once anomalies are found, alerting component will send anomaly info to the data owner © 2014 Microsoft Corporation. All rights reserved. Microsoft, Windows, and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries. The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

How it works - Extensibility Defined a generic interface of training and detection Each algorithm provider would implement per defined interface For example for each data point, we expect following from algorithm providers Whether it is an anomaly What is the predicted/expected value by algorithm What is the suggested lower bound What is the suggested upper bound Confidence level …

How it works – Extensibility

How it works - Scalability

Algorithms Intro Based on the initial customer usage, we start with those algorithms and make the generic interface based on the characteristic of them

Algorithm - Service Insider 5/12/2018 Algorithm - Service Insider Good in time series with periodical pattern Holt-Winters algorithm - Train model and predict Improvements for robustness: Use Median Absolute Deviation (MAD) to get robust estimation Handling for data missing and noise (e.g., data smoothing) Automatically capture the slow and regular trend and seasonal pattern GLR (Generalized Likelihood Ratio) - Used to detect anomalies Improvements Floating Threshold GLR, to dynamically adjust the model using the new input data Outlier removal for noisy data © 2014 Microsoft Corporation. All rights reserved. Microsoft, Windows, and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries. The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

Machine Learning, Analytics & Data Science Conference 5/12/2018 2:25 PM Other Improvements Automatic detection of time series types (seasonal/non-seasonal) Automatic detection of seasonality/trend, instead of manual setting Add the feedback channels for end users to intuitively tune the algorithms The automatic detection use an integration of FFT (Fast Fourier Transformation) and LOESS (locally weighted scatterplot smoothing) FFT directly translate the time series into frequency domain, and it helps to capture the strongest frequency signal, but it could have bunch of noisy signals in the results LOESS does not require the specification of a function to fit a model to all of the data, it is flexible, and it could help to eliminate some noisy results Integrate the pure unsupervised statistics with semi-supervised feedback tuning Traditional statistics approaches (like Holt-Winters and ARIMA) are commonly used, it is unsupervised and no labeling required; but it won’t reflect users’ intent (too loose/strict) When users explicitly adjust on some points, their feedback are used as semi-supervised labels to retrain the parameters to fit more on users’ intent © 2014 Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

Azure ML - Exchangeability Martingale Good in detecting slow upward/downward trend, spike and dip, change in dynamic range General framework for online change detection in time series Has the property we are interested in changed in distribution? User specifies meaning of “new value strangeness” given history At each time t we receive a new value Add it to the history. For each item i in the history s[i] = strangeness function of (value[i], history) Let p[t] = (#{i: s[i] > s[t]}+ r*#{i: s[i]==s[t]})/N, where r is uniform in (0,1) Uniform r makes sure p is uniform

Azure ML - Exchangeability Martingale  

Algorithm – Exponential Smoothing  

Result Evaluation of exponential smoothing In some cases with periodical signal with trending, many false positives could be generated

Result Evaluation - ServiceInsider

Result Evaluation – EM

Result Evaluation – ES based

Result Evaluation – ServiceInsider and Azure ML

Challenges and Future Work Real time vs. accuracy Automated handling of data pattern change Easy tuning or usage of different algorithms

Real time vs. Accuracy Real time vs. Accuracy 5/12/2018 Real time vs. Accuracy Real time vs. Accuracy Some data streams are not stable from the perspective of data point latency © 2014 Microsoft Corporation. All rights reserved. Microsoft, Windows, and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries. The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

Data Pattern Change

Easy Tuning Tuning the algorithm parameters to achieve right detection precision and recall is a pain to the users Service insider 2 parameters EM based: 7 parameters ES based: 3 parameters Creative UI to hide those details Do without human tuning at all!

Questions!