Presentation on theme: "Lynn Jones Stottler Henke Associates, Seattle, WA Lockheed Martin, Gaithersburg, MD 12/20/01 Anomaly Detection Using “Normal” Data."— Presentation transcript:
Lynn Jones Stottler Henke Associates, Seattle, WA Lockheed Martin, Gaithersburg, MD 12/20/01 Anomaly Detection Using “Normal” Data
12/20/01slide 2 “Table of Contents” Introduction Related work Project overview SHAI’s anomaly detection Key component: CVFDT Why this will work What else we need Other application areas
12/20/01slide 3 ChAD: Change and Anomaly Detection Model-based Change and Anomaly Detection system. Has nothing to do with voting in Florida. Models normal behavior by observing normal behavior. Detects departures from normal. Does not require profiles or signatures of abnormal behavior (faults, attacks).
12/20/01slide 4 ChAD: Change and Anomaly Detection Learns a model unique to the monitored network and host. Robust when faced with noisy data. Adapts to fluctuations in network usage. Detects when characteristics change. Reports on the rate and significance of observed changes.
12/20/01slide 5 Related work by SHAI Network management and security –Athena: Mixed-initiative Defensive Information Warfare. –ICE: Intelligent Correlation of Evidence (for network intrusion detection).
12/20/01slide 6 Related work by SHAI Datamining –CASAD: Clustering Activity Streams for Anomaly Detection. –MediMiner, IKODA (Intelligent Knowledge Discovery Assistant) data mining algorithms and frameworks.
12/20/01slide 7 Related commercial work Aprisma’s SPECTRUM suite –event correlation and model-based reasoning. –SNMP MIB and other data. SRI’s Emerald (eBayes) –hybrid signature-based / anomaly detection monitoring. –tcpDump data and derived events.
12/20/01slide 8 Related research Cabrera, et.al. –look for differences in behavior of selected “key variables.” INBOUNDS –statistical modeling using “abnormality factors” and “standardization factors.” Eskin, et.al. –Automatic outlier partitioning and learned model replacement.
12/20/01slide 9 ChAD is a component of MASRR MASRR: Multi-Agent System for Network Resource Reliability –Decentralized monitoring and response. –Prediction and detection of attacks, faults, misconfigurations, etc. –Network steering to maintain performance. Funded as a DARPA SBIR Phase II.
12/20/01slide 10 MASRR goals Detection of events not previously seen. Adaptation to changing usage characteristics. Operation in heterogeneous environments. Real-time performance. Scalability in deployment and operation. Autonomous / semi-autonomous operation. Robustness.
12/20/01slide 11 MASRR current focus We have chosen to focus our efforts on Anomaly Detection using normal data.
12/20/01slide 12 SHAI’s anomaly detection Use data mining methods to build a descriptive model that detects changes in the data stream. We believe we can overcome specific issues and problems...
12/20/01slide 15 Key component: CVFDT Concept-adaptive Very Fast Decision Tree –on-line decision tree model. –does not have to see “all” the data first. –accuracy converges to offline models. Network usage changes over time. Rather than a stationary concept, data is “generated by a series of concepts.” »Hulten, Spencer, & Domingos, “Mining Time-Changing Data Streams”, KDD ’01
12/20/01slide 16 Decision Trees, in general A decision tree built from some engine data shows that the life of oil seals depends on the operating temperature, and, less definitively, the pressure. This model might be used in making a maintenance schedule.
12/20/01slide 17 Adaptive Decision Trees The company changes the supplier of its oil seals, and begins seeing early failures of seals when operating pressure is around 15, with a wide variance in temperature. The adaptive tree starts an alternate tree...
12/20/01slide 18 Adaptive Decision Trees New records are processed by both trees. As the alternate tree grows, it eventually becomes more accurate than the original. The alternate is promoted and the original tree is pruned.
12/20/01slide 19 CVFDT - more detail Each node keeps “sufficient statistics” on the examples seen. Sliding window of examples. Nodes maintain statistics, forget examples as the window slides. Structure of the tree is periodically evaluated, using statistics.
12/20/01slide 20 CVFDT - more detail Alternate tree is started using different split attribute. After every n examples, trees are tested for accuracy. If alternate is better, replace original. If alternate fails to improve, it is pruned.
12/20/01slide 21 CVFDT reveals: That system behavior is changing. How it’s changing – which variable(s). The degree to which it’s changing – how dramatically, how rapidly, whether transient or permanent. ChAD applies CVFDT in a novel way to perform anomaly detection using normal, unlabeled data.
12/20/01slide 22 MASRR agents use ChAD Segment usage and model different periods of normal activity. Manage the library of normal models. Interpret results of ChAD models. Share their observations. Adjust parameters to tune model sensitivity.
12/20/01slide 23 Why this will work Utilizes routine fluctuations to create more precise periodic models. Each agent is sensitive to small changes (slow changes, changes across few variables) on the element(s) it monitors.
12/20/01slide 24 Why this will work When the network is compromised in some area, absence of data or agent response is also used as information. Combines general anomaly detection with root cause analysis.
12/20/01slide 25 Why this will work More general than eBayes: can detect various kinds of anomalies across different variables. “Key variable” signatures not required as in Cabrera, et. al. (similar rules might be used for fault/attack identification). Decentralized analysis more sensitive than INBOUNDS’ centralized system.
12/20/01slide 26 Known issues Overhead - processing, disk space. Getting the sensitivity parameters right. Are parameters universal? Or do they depend on the data? Amount of data needed. What about pre-existing conditions? Feature selection.
12/20/01slide 27 What (else) will it take? Testing and refinement with real data. Implementation of the agent reasoning system. Implementation of heuristics. Feature selection experiments.
12/20/01slide 28 Other applications Manufacturing processes monitoring. Condition-based monitoring (military and commercial) - e.g., fault and wear prediction for maintenance scheduling.
12/20/01slide 29 Conclusion SHAI is developing an anomaly detection system that we believe: –is scalable, –works in real-time, –detects attacks or faults not previously observed, –learns in-place using normal, unlabeled data.
12/20/01slide 30 General info on SHAI Artificial Intelligence R&D firm, founded in 1988. Extensive experience –Hundreds of fielded systems. –Variety of AI techniques and application areas.
12/20/01slide 31 Contact info Lynn Jones firstname.lastname@example.org http://22.214.171.124/ReliabilityWeb/ SHAI 1107 NE 45th St. Suite 427 Seattle, WA 98105