Mitigating the Insider Threat using High-dimensional Search and Modeling SRS Kickoff Meeting, Arlington, VA, July 21, 2004 Telcordia Contact: Eric van den Berg (732) 699 2748 evdb@research.telcordia.com Team: Shambhu Upadhyaya (SUNY Buffalo) Roy Maxion (CMU) Raj Rajagopalan (HP Labs & Rutgers) An SAIC Company

Talk outline Problem description Prior art About this project Summary Key ideas Proposed architecture Technical approach Challenges Metrics for success Summary

Insider Threat: Problem motivation 59% of companies have had one or more ‘Insider abuse of net access’ incidents in 2003 Estimated losses due to ‘Insider net abuse’ and ‘unauthorized access’ $15 M Source: FBI/CSI Computer Crime and Security Survey 2004.

What are the key aspects of the Insider Threat problem? Insider attacks are different from outside attacks: starting with privileges that cannot be denied Resource access Knowledge of targets and vulnerabilities Insider attacks are more difficult to detect and defend against Perimeter defenses look for outside attacks Any user or group of users may potentially launch an attack Can inflict wider damage, quicker High premium on not punishing the good users Detection requires large number of correlated data streams to be processed Insiders may subvert single stream detection Need proactive approach Learning of attacks after-the-fact is often too late because damage is done Learning how to deal with similar attacks in the future is critical: insiders already know previous attack signatures are in place

What is done today? Reactive systems Detect attacks late in cycle Anomaly detection systems Few streams for correlation, suffer from curse of dimensionality Human-in-the-loop systems Response not scalable Prior attacks pulled from administrator experience Consequences of response vs impact attack Collateral damage may be large

What is the project goal? We want to build a system that defends critical services and resources against insiders Can detect attacks by correlating large numbers of sensor measurements, and Can synthesize appropriate pro-active responses to protect critical services while minimizing collateral damage.

What is our idea? The highlights Collect data from as many different kinds of sensors as possible Audit logs, Web/application logs, network flow data, firewall logs Construct formal model of organizational information aspect of insider threat Can guide placement of sensors Store sensor data in a unified format Suitably filtered and/or aggregated Create historical record in database Use high-dimensional search techniques Create clusters on historical records Search for records similar to current sensor snapshot Initially use humans for expert knowledge, to label history and tune searches Predict attacks using labeled clusters of historical data Predict attack as soon as state becomes similar to past attack precursor Create proactive, automated response Conduct impact analysis of attack on availability of critical services Generate candidate responses and evaluate their impact before deployment

Proposed architecture

Sensor network Install sensors at multiple system layers to monitor applications, servers, hosts and other devices on which critical services depend: End-host sensors (applications, cpu-load, audit logs, web logs, registry, user challenges, etc) Network sensors (aggregate traffic, flow data,…) Aggregators and filters (to reduce sensor data volume) Main idea: Make it very hard for malicious insider to avoid triggering some sensor

Translation and mapping Translate sensor data Normalize sensor data Filter sensor data Aggregate sensor data Map sensor data into network state description Group sensor values into high-dimensional vector Vector of sensor values forms both ‘query’ for search engine, and part of historical network state description

Search engine and Network state repository Use Search Engine to compare current state document to historical documented network states. Search Engine will use Singular Value Decomposition (SVD) techniques for dimension reduction of attribute space Also experimenting with random projection methods Output of Search Engine “Top-K” list of similar documents, together with distance or similarity. If current state is sufficiently similar to past attack, send attack type and location to Response Engine for impact analysis Build Network state repository Construct schema to support search Addresses ‘Curse of dimensionality’ in anomaly detection

High dimensional Search via SVD on Labeled Clusters 1.0 Normal Insider DoS Worms S8 S10 S13 S14 S9 S4 S15 query S1 1.0 S2 S6 S16 S5 S11 S12 S7 S17 S3

Insider analyzer and modeler New formal threat model that captures the organizational information aspect of the insider threat Threat model based on new graph model This model allows analysis of the following questions: What are likely/feasible attack paths? What is the corresponding difficulty (‘cost’) of each such attack? This component helps determine: Which parts of the organization need more careful monitoring? Which security policies need to be reinforced? Insider analyzer and modeler may also guide placement of sensors, and help label clusters of network states

Impact Analysis using Response Engine Building upon Smart Firewalls technology from Dynamic Coalitions program Response Engine has overview of current network configuration Response Engine logically validates Policies, expressed in terms of end-to-end service availability Response Engine generates candidate reconfigurations to comply with Policies as much as possible In this project Detected attack type and location is translated into its effect on the current network configuration E.g. Server failure due to a Denial of Service attack Response Engine can analyze the impact of both the attack and its candidate responses on the availability of critical resources Administrator can push response into the network

Technical challenges We are testing a new hypothesis of whether search engine techniques can be used effectively for this problem. Our key insight is that network attacks are often similar to each other but it is hard to predict what the small change is. Telcordia has extensive experience with SVD based searches in text-based information retrieval. Here we are testing SVD search technology in a new domain. Training search engine: tune distance metrics, label data, reduce false alarms New ‘Insider attack-information graph’ problem is hard

How do we know if we succeeded? Example Scenario: 1. Launch known insider attack in one part of testbed network and tag the data. 2. Launch the same attack in a different part of testbed network. 3. Detect attack using Search Engine. 4. Analyze impact on network using Response Engine 5. Respond using appropriate configuration change (similar to response to old attack) Success = detection and appropriate response to attack after completing steps 3 through 5. Repeat the experiment changing different parameters of the attack such as topology, location, scale, source/target choices, and finally attack vector.

Summary If we succeed, we will have a system that defends critical services and resources against insiders, which Can detect attacks by correlating large numbers of sensor measurements, and Can synthesize appropriate pro-active responses to protect critical services while minimizing collateral damage.