1. Data Mining Anomaly/Outlier Detection Lecture Notes for Chapter 10 Introduction to Data Mining by Tan, Steinbach, Kumar © Tan,Steinbach, Kumar Introduction to Data Mining 4/18/2004 1

2. © Tan,Steinbach, Kumar Introduction to Data Mining 4/8/2015 2 Anomaly/Outlier Detection l What are anomalies/outliers? –The set of data points that are considerably different than the remainder of the data l Variants of Anomaly/Outlier Detection Problems –Given a database D, find all the data points x  D with anomaly scores greater than some threshold t –Given a database D, find all the data points x  D having the top- n largest anomaly scores f(x) –Given a database D, containing mostly normal (but unlabeled) data points, and a test point x, compute the anomaly score of x with respect to D l Applications: –Credit card fraud detection, telecommunication fraud detection, network intrusion detection, fault detection

3. © Tan,Steinbach, Kumar Introduction to Data Mining 4/8/2015 3 Importance of Anomaly Detection Ozone Depletion History l In 1985 three researchers (Farman, Gardinar and Shanklin) were puzzled by data gathered by the British Antarctic Survey showing that ozone levels for Antarctica had dropped 10% below normal levels l Why did the Nimbus 7 satellite, which had instruments aboard for recording ozone levels, not record similarly low ozone concentrations? l The ozone concentrations recorded by the satellite were so low they were being treated as outliers by a computer program and discarded! Sources: http://en.wikipedia.org/wiki/Ozone_depletionhttp://en.wikipedia.org/wiki/Ozone_depletion

4. © Tan,Steinbach, Kumar Introduction to Data Mining 4/8/2015 4 Anomaly Detection l Challenges –How many outliers are there in the data? –Method is unsupervised  Validation can be quite challenging (just like for clustering) –Finding needle in a haystack l Working assumption: –There are considerably more “normal” observations than “abnormal” observations (outliers/anomalies) in the data

5. © Tan,Steinbach, Kumar Introduction to Data Mining 4/8/2015 5 Anomaly Detection Schemes l General Steps –Build a profile of the “normal” behavior  Profile can be patterns or summary statistics for the overall population –Use the “normal” profile to detect anomalies  Anomalies are observations whose characteristics differ significantly from the normal profile l Types of anomaly detection schemes –Graphical –Model-based –Distance-based –Clustering-based

6. © Tan,Steinbach, Kumar Introduction to Data Mining 4/8/2015 6 Graphical Approaches l Boxplot (1-D), Scatter plot (2-D), Spin plot (3-D) l Limitations –Time consuming –Subjective e.g.: data are outliers if the more than 1.5 IQR away from the box!

7. © Tan,Steinbach, Kumar Introduction to Data Mining 4/8/2015 7 Convex Hull Method l Extreme points are assumed to be outliers l Use convex hull method to detect extreme values l http://www2.cs.uh.edu/~ceick/kdd/AEC13.pdf http://www2.cs.uh.edu/~ceick/kdd/AEC13.pdf l Approach: Fit a polygon to a point set; outliers are determined by their distance to the polygon boundary

8. © Tan,Steinbach, Kumar Introduction to Data Mining 4/8/2015 8 Statistical Approaches---Model-based l Fit a parametric model M describing the distribution of the data (e.g., normal distribution) http://en.wikipedia.org/wiki/Distribution_fitting http://en.wikipedia.org/wiki/Maximum_likelihood http://en.wikipedia.org/wiki/Distribution_fittinghttp://en.wikipedia.org/wiki/Maximum_likelihood l Assess the probability of each point M l The lower a point’s probability the more likely the point is an outlier. l Outlier Detection Approach: –Sort points by the probability –Determine Outliers  based on a probability threshold  take the bottom x percent as outliers.

9. © Tan,Steinbach, Kumar Introduction to Data Mining 4/8/2015 9 Grubbs’ Test if Dataset Contains Outliers l Detect outliers in univariate data l Assume data comes from normal distribution l Detects one outlier at a time, remove the outlier, and repeat –H 0 : There is no outlier in data –H A : There is at least one outlier l Grubbs’ test statistic: l Reject H 0 if:

10. © Tan,Steinbach, Kumar Introduction to Data Mining 4/8/2015 10 Limitations of Statistical Approaches l Most of the tests are for a single attribute l In many cases, data distribution/model may not be known l For high dimensional data, it may be difficult to estimate the true distribution, but there is new work based on EM and mixtures of Gaussians

11. © Tan,Steinbach, Kumar Introduction to Data Mining 4/8/2015 11 Distance-based Approaches l Data is represented as a vector of features l Three major approaches –Nearest-neighbor based –Density based –Clustering based

12. © Tan,Steinbach, Kumar Introduction to Data Mining 4/8/2015 12 Nearest-Neighbor Based Approach l Approach: –Compute the distance between every pair of data points –There are various ways to define outliers:  Data points for which there are fewer than p neighboring points within a distance D  The top n data points whose distance to the kth nearest neighbor is greatest  The top n data points whose average distance to the k nearest neighbors is greatest

13. © Tan,Steinbach, Kumar Introduction to Data Mining 4/8/2015 13 Density-based: LOF approach l For each point, compute the density of its local neighborhood; e.g. use DBSCAN’s approach l Compute local outlier factor (LOF) of a sample p as the average of the ratios of the density of sample p and the density of its nearest neighbors l Outliers are points with lowest LOF value p 2  p 1  In the NN approach, p 2 is not considered as outlier, while LOF approach find both p 1 and p 2 as outliers Alternative approach: directly use density function; e.g. DENCLUE’s density function

14. © Tan,Steinbach, Kumar Introduction to Data Mining 4/8/2015 14 Clustering-Based l Idea: Use a clustering algorithm that has some notion of outliers! l Problem what parameters should I choose for the algorithm; e.g. DBSCAN? l Rule of Thumb: Less than x% of the data should be outliers (with x typically chosen between 0.1 and 10); x might be determined with other methods; e.g. statistical tests or domain knowledge. l Once you have an idea, how much outliers you want, the parameter selection problem becomes easier.