A Technical Evaluation and Critique of: Techniques and Tools for Analyzing Intrusion Alerts by Peng Ning, Yun Cui, Douglas S. Reeves, and Dingbang Xu Angela Orebaugh IT862 4/28/05 Angela Orebaugh IT862 4/28/05

Formal Framework

Framework Highlights Correlates alerts on the basis of prerequisites and consequences of attacks Matches the consequences of some prior alerts with the prerequisites of some later ones Constructs attack scenarios Represented in Hyperalert Correlation Graph Uses nodes to represent alerts and edges to represent the relationships between the alerts.

Framework Notation Prerequisite Predicate UDPVulnerableToBOF(VictimIP, VictimPort) Consequence Predicate {GainRootAccess (VictimIP), rhostsModified (VictimIP)} Logical combination of predicates for complex attacks UDPVulnerableToBOF(VictimIP, VictimPort) ^ (UDPAccessibleViaFirewall (VictimeIP, VictimPort)

Framework Notation (2) HyperAlert Type T (fact, prerequisite, consequence) SadmindBufferOverflow = ({VictimIP, VictimPort}, ExistHost (VictimIP) ^ VulnerableSadmind (VictimIP), {GainRootAccess (VictimIP)}) HyperAlert Instance h h SadmindBOF = {(VictimIP = , VictimPort = 1235), (VictimIP = , VictimPort = 1235)} ExistHost ( ) ^ VulnerableSadmind ( ), ExistHost ( ) ^ VulnerableSadmind ( ) GainRootAccess ( ), GainRootAccess ( )

Hyperalert Correlation In a sequence S of hyperalerts, a hyperalert h is... Correlated hyperalert if there exists another hyperalert h in S such that either h prepares for h or h prepares for h. Isolated hyperalert if no such h exists

Hyperalert Correlation Graph

Additional Utilities Aggregation/Disaggregation All hyperalerts of type FTP-BOF combined All hyperalerts that are DoS are combined Focused Analysis SrcIP = V DestIP = Clustering Analysis (A1.SrcIP = A2.SrcIP) ^ (A1.DestIP = A2.DestIP) Frequency Analysis Counting the number of raw alerts that share the same destination IP address to fnd the most frequently hit target Link Analysis How two IP addresses are realted to each other in a collection of alerts Association Analysis Many attacks are from source IP to destination IP at destination port 80

TIAA (Toolkit for Intrusion Alert Analysis)

TIAA Architecture

Most Recent TIAA Software Current version 0.4 Tested on Windows 2000 and XP with MS SQL Server Newly added features Association Analysis (Extracting frequent coourrences of attribute values from a set of alerts) Attack Strategy Extraction (Extracting attack strategies from a correlation graph) Missed Attack Hypotheses (Hypothesizing possibly missed attacks)

TIAA Knowledge Base

Experiments

2000 DARPA intrusion detection dataset Aimed at evaluating the effectiveness of the proposed alert correlation method in constructing attack scenarios and its ability to differentiate true and false alerts. DEFCON 8 CTF Intended to evaluate the usefulness of the analysis utilities in dealing with large collections of intrusion alerts.

DARPA Dataset LLDOS Series of attacks in which an attacker probes, breaks in, installs the components necessary to launch a DDoS attack, and launches a DDoS attack against an off-site server. LLDOS Similar sequence of attacks by a more sophisticated attacker. Each dataset contains network traffic collected from both the DMZ and the internal network. Testing used 4 sets of experiments, each with either the DMZ of the inside network traffic of one dataset.

DEFCON 8 CTF Dataset Capture the flag contest Attacks range from script kiddie to sophisticated attacker Largest graph had 2,940 nodes and 25,321 edges On average each graph had nodes and edges

DARPA Experiment Results TIAA revealed the structure and high-level strategy of the sequence of attacks Real Secure generated duplicate alerts for several attacks Correlated a few false alerts ISS >93% false alert rate TIAA reduced to 5% for LLDOS 1.0 and 23%-40% for LLDOS Correlated normal alerts that were not attacks Missed the Telnet portion of the attack LLDOS results were unsatisfactory

DEFCON Experiment Results Probably some missed alerts - So many attacks occurring at once Alert aggregation reduced the largest graph to 77 nodes and 347 edges 7 clear stages of attacks Utilities helped discover several attack strategies Scanning attacks followed by attacks that may lead to execution of arbitrary code Not good for forensics

Related Work

First Class of Approaches Staniford 2002 Probability distribution for normal traffic to detect portscan attacks SPICE/SPADE Valdes and Skinner 2001 Mathematical framework for correlating alerts that match closely but not perfectly EMERALD Cuppens 2001 Alert clustering and merging via expert system approach Also uses pre and post attack conditions based on LAMDA MIRADOR project Julisch 2001 Alarm clustering to determine root causes Alarm clustering and summarizing

Second Class of Approaches Eckmann 2002 State transition based attack scenarios State Transition Analysis Technique Language (STATL) Cuppens and Ortalo 2000 Attack scenarios with pre and post conditions using the LAMBDA attack description language Debar and Wespi 2001 Detects duplicates and consequences according to explicit rules Built on top of Tivoli Enterprise Console

Third Class of Approaches Templeton and Levitt 2000 Capabilities/concepts attack model describes unknown attacks and predict attacker actions JIGSAW attack specification language Cuppens and Miege 2002 CRIM module based on LAMBDA to cluster, merge, and correlate alerts MIRADOR project Morin 2002 M2D2 data model for correlation Uses correlation function to detect false positives Porras 2002 Mission-impact-based approach MCorrelator uses an internal topology map for correlation Peng Ning, et. al. 2003

Vulnerability Analysis Approach Ritchey and Ammann 2000 Modeling based approach based on host vulnerability, host connectivity, current point of view of attacker, exploits that can change the state of the model Uses a state machine to encode the vulnerabilities Sheyner 2002 Automated technique for generating and analyzing attack graphs Based on intruder preconditions, network preconditions, intruder effects, and network effects Jha 2002 Expands on Sheyner paper Presents a formal and detailed explanation of the model Presents an algorithm to compute the reliability for a network

Summary

Major Contributions Hyperalert correlation graphs Partial satisfaction of attack prerequisites Uses possible consequences instead of actual consequences Analysis Utilities TIAA

Framework Critique Successfully revealed relationships between alerts and strategies behind the attacks Effectively reduces the number of alerts via aggregation Needs to address partial satisfaction of prerequisites more thoroughly It does not address security architecture It doesnt address network issues IDS evasion May not discover stealthy and intelligent attacks Can still be evaded

TIAA Critique Needs to provide more information on the Knowledge base Does the accuracy of the system all come down to the robustness of the knowledge base? the results produced by our correlation techniques are only as good as the hyperalert information provide by the user Only supports IDMEF and ISS Real Secure alerts Only works with a commercial database - MS SQL Server Not meant for the inexperienced use

Testing Critique Uses ISS Real Secure Are the DARPA and DEFCON CTF datasets the best methods of testing? DARPA dataset has received lots of criticism DEFCON CTF is all attack traffic and not much else What is the best testing approach anyway? Test network, live network, replay data taken from other networks, DARPA or other pre-generated datasets Tested DARPA datasets separately

Architecture Does not address a recommended security architecture for optimization Sensor placement is critical to correlation

Additional Research Opportunities Expanded use of hyperalert correlation graphs Attacker profiling Predictive analysis IDS tuning Input to incident response procedures Incorporate forensic analysis Integrate framework with complementary correlation methods for better performance Use TIAA as part of penetration testing team Automatic generation of knowledge base by learning algorithm

Discussion... Critiques of the framework? Critiques of the TIAA toolkit? Critiques of the testing? Additional research/expansion opportunities?