Presentation is loading. Please wait.

Presentation is loading. Please wait.

ADVANCED ANOMALY DETECTION IN CANARY TESTING

Published byἘλισάβετ Σπυρόπουλος Modified over 5 years ago

Similar presentations

Presentation on theme: "ADVANCED ANOMALY DETECTION IN CANARY TESTING"— Presentation transcript:

1 ADVANCED ANOMALY DETECTION IN CANARY TESTING
Presented by Tamas Cser – Functionize, Inc. © All rights reserved

2 Advanced Deployment Methods
Advanced deployment methods - Multiple versions in product with split traffic © All rights reserved

3 Modern Code Deployment
Current App Version 95% New App Version Users 5% Users become the QA for your app! © All rights reserved

4 Canary Testing Approach
Release To All Users Current App Version Build User ML Model Measure Behavior Yes Release To User Subset New App Version Measure Behavior Anomaly Detection Good For Release? No Revise App © All rights reserved

5 User Journeys © All rights reserved

6 Experimental Method - Clustering
Collected session and actions Recorded user sessions on sample application Clusters session data by url/element interaction Identified 5 unique clusters in the data Each session is then analyzed to find how far it is from its cluster mean Cluster standard deviation data is transformed to be normally distributed and confidence interval is created. Sessions falling far enough away from their cluster mean to be outside the confidence interval are deemed anomalous --- Clustering of User Sessions Calculate Confidence Intervals for Clusters Algorithms used: Expectation Maximization Iterative method to find maximum likelihood estimate of parameters Akaike Information Criterion Given a collection of models, selects best one relative to the others © All rights reserved

7 LSTM Model Source: Analytics Vidhya Long Short-Term Memory
Type of Recurrent Neural Network Analyzes data sequences/time series Predicts the probability of the next user action Source: Analytics Vidhya © All rights reserved

8 LSTM model can work together with the EM clustering
Hybrid Approach LSTM model can work together with the EM clustering Build LSTM for each cluster of user sessions Since clusters represent similar user behavior, LSTM can more effectively predict the next user action Anomalous session are easier to identify © All rights reserved

9 Model trained on user data from live sites
Process and Results Model trained on user data from live sites 10,000 user sessions were analyzed 80% of data used for training and 20% for testing Predicted the next user action 85% of the time Since there are hundreds of action/element combinations on a page, this is actually quite accurate © All rights reserved

10 Split Traffic and Canary Modeling
© All rights reserved

11 2% of sessions were identified as outliers
Conclusion Successfully segmented sessions into 5 clusters based on similar behaviors 2% of sessions were identified as outliers Anomalous user sessions can be easily identified LSTM model can accurately predict probabilities of actions in a user session. This analysis can be used for gating releases and canary testing Cluster 1: Single event sessions Cluster 2: Login interactions Cluster 3: Account interactions Cluster 4: Blog interactions Cluster 5: Subdomain interactions © All rights reserved

Download ppt "ADVANCED ANOMALY DETECTION IN CANARY TESTING"

Similar presentations

1 VLDB 2006, Seoul Mapping a Moving Landscape by Mining Mountains of Logs Automated Generation of a Dependency Model for HUG’s Clinical System Mirko Steinle,

1 VLDB 2006, Seoul Mapping a Moving Landscape by Mining Mountains of Logs Automated Generation of a Dependency Model for HUG’s Clinical System Mirko Steinle,
Unsupervised Learning

Unsupervised Learning
Sensor-Based Abnormal Human-Activity Detection Authors: Jie Yin, Qiang Yang, and Jeffrey Junfeng Pan Presenter: Raghu Rangan.

Sensor-Based Abnormal Human-Activity Detection Authors: Jie Yin, Qiang Yang, and Jeffrey Junfeng Pan Presenter: Raghu Rangan.
AP Statistics Section 11.2 B. A 95% confidence interval captures the true value of in 95% of all samples. If we are 95% confident that the true lies in.

AP Statistics Section 11.2 B. A 95% confidence interval captures the true value of in 95% of all samples. If we are 95% confident that the true lies in.
Fundamentals of Data Analysis Lecture 12 Methods of parametric estimation.

Fundamentals of Data Analysis Lecture 12 Methods of parametric estimation.
SKELETON BASED PERFORMANCE PREDICTION ON SHARED NETWORKS Sukhdeep Sodhi Microsoft Corp Jaspal Subhlok University of Houston.

SKELETON BASED PERFORMANCE PREDICTION ON SHARED NETWORKS Sukhdeep Sodhi Microsoft Corp Jaspal Subhlok University of Houston.
Anomaly Detection in the WIPER System using A Markov Modulated Poisson Distribution Ping Yan Tim Schoenharl Alec Pawling Greg Madey.

Anomaly Detection in the WIPER System using A Markov Modulated Poisson Distribution Ping Yan Tim Schoenharl Alec Pawling Greg Madey.
Descriptive statistics Experiment  Data  Sample Statistics Sample mean Sample variance Normalize sample variance by N-1 Standard deviation goes as square-root.

Descriptive statistics Experiment  Data  Sample Statistics Sample mean Sample variance Normalize sample variance by N-1 Standard deviation goes as square-root.
On Community Outliers and their Efficient Detection in Information Networks Jing Gao 1, Feng Liang 1, Wei Fan 2, Chi Wang 1, Yizhou Sun 1, Jiawei Han 1.

On Community Outliers and their Efficient Detection in Information Networks Jing Gao 1, Feng Liang 1, Wei Fan 2, Chi Wang 1, Yizhou Sun 1, Jiawei Han 1.
Descriptive statistics Experiment  Data  Sample Statistics Experiment  Data  Sample Statistics Sample mean Sample mean Sample variance Sample variance.

Descriptive statistics Experiment  Data  Sample Statistics Experiment  Data  Sample Statistics Sample mean Sample mean Sample variance Sample variance.
Unsupervised Training and Clustering Alexandros Potamianos Dept of ECE, Tech. Univ. of Crete Fall 2004-2005.

Unsupervised Training and Clustering Alexandros Potamianos Dept of ECE, Tech. Univ. of Crete Fall
Bagging LING 572 Fei Xia 1/24/06. Ensemble methods So far, we have covered several learning methods: FSA, HMM, DT, DL, TBL. Question: how to improve results?

Bagging LING 572 Fei Xia 1/24/06. Ensemble methods So far, we have covered several learning methods: FSA, HMM, DT, DL, TBL. Question: how to improve results?
Modeling Gene Interactions in Disease CS 686 Bioinformatics.

Modeling Gene Interactions in Disease CS 686 Bioinformatics.
Business Statistics - QBM117 Revising interval estimation.

Business Statistics - QBM117 Revising interval estimation.
Leon-Guerrero and Frankfort-Nachmias,

Leon-Guerrero and Frankfort-Nachmias,
INFERENTIAL STATISTICS – Samples are only estimates of the population – Sample statistics will be slightly off from the true values of its population’s.

INFERENTIAL STATISTICS – Samples are only estimates of the population – Sample statistics will be slightly off from the true values of its population’s.
EVALUATION David Kauchak CS 451 – Fall 2013. Admin Assignment 3 - change constructor to take zero parameters - instead, in the train method, call getFeatureIndices()

EVALUATION David Kauchak CS 451 – Fall Admin Assignment 3 - change constructor to take zero parameters - instead, in the train method, call getFeatureIndices()
Anomaly detection Problem motivation Machine Learning.

Anomaly detection Problem motivation Machine Learning.
Tuesday, September 10, 2013 Introduction to hypothesis testing.

Tuesday, September 10, 2013 Introduction to hypothesis testing.
CS490D: Introduction to Data Mining Prof. Chris Clifton April 14, 2004 Fraud and Misuse Detection.

CS490D: Introduction to Data Mining Prof. Chris Clifton April 14, 2004 Fraud and Misuse Detection.

Similar presentations

Ads by Google