1. Mitigating the Insider Threat using High-dimensional Search and Modeling
SRS Kickoff Meeting,
Arlington, VA, July 21, 2004
Telcordia Contact: Eric van den Berg
(732)
Team: Shambhu Upadhyaya (SUNY Buffalo)
Roy Maxion (CMU)
Raj Rajagopalan (HP Labs & Rutgers)
An SAIC Company

2. Talk outline Problem description Prior art About this project Summary
Key ideas
Proposed architecture
Technical approach
Challenges
Metrics for success
Summary

3. Insider Threat: Problem motivation
59% of companies have had one or more ‘Insider abuse of net access’ incidents in 2003
Estimated losses due to ‘Insider net abuse’ and ‘unauthorized access’ $15 M
Source: FBI/CSI Computer Crime and Security Survey 2004.

4. What are the key aspects of the Insider Threat problem?
Insider attacks are different from outside attacks: starting with privileges that cannot be denied
Resource access
Knowledge of targets and vulnerabilities
Insider attacks are more difficult to detect and defend against
Perimeter defenses look for outside attacks
Any user or group of users may potentially launch an attack
Can inflict wider damage, quicker
High premium on not punishing the good users
Detection requires large number of correlated data streams to be processed
Insiders may subvert single stream detection
Need proactive approach
Learning of attacks after-the-fact is often too late because damage is done
Learning how to deal with similar attacks in the future is critical: insiders already know previous attack signatures are in place

5. What is done today? Reactive systems Detect attacks late in cycle
Anomaly detection systems
Few streams for correlation, suffer from curse of dimensionality
Human-in-the-loop systems
Response not scalable
Prior attacks pulled from administrator experience
Consequences of response vs impact attack
Collateral damage may be large

6. What is the project goal?
We want to build a system that defends critical services and resources against insiders
Can detect attacks by correlating large numbers of sensor measurements, and
Can synthesize appropriate pro-active responses to protect critical services while minimizing collateral damage.

7. What is our idea? The highlights
Collect data from as many different kinds of sensors as possible
Audit logs, Web/application logs, network flow data, firewall logs
Construct formal model of organizational information aspect of insider threat
Can guide placement of sensors
Store sensor data in a unified format
Suitably filtered and/or aggregated
Create historical record in database
Use high-dimensional search techniques
Create clusters on historical records
Search for records similar to current sensor snapshot
Initially use humans for expert knowledge, to label history and tune searches
Predict attacks using labeled clusters of historical data
Predict attack as soon as state becomes similar to past attack precursor
Create proactive, automated response
Conduct impact analysis of attack on availability of critical services
Generate candidate responses and evaluate their impact before deployment

8. Proposed architecture

9. Sensor network Install sensors at multiple system layers
to monitor applications, servers, hosts and other devices on which critical services depend:
End-host sensors (applications, cpu-load, audit logs, web logs, registry, user challenges, etc)
Network sensors (aggregate traffic, flow data,…)
Aggregators and filters (to reduce sensor data volume)
Main idea: Make it very hard for malicious insider to avoid triggering some sensor

10. Translation and mapping
Translate sensor data
Normalize sensor data
Filter sensor data
Aggregate sensor data
Map sensor data into network state description
Group sensor values into high-dimensional vector
Vector of sensor values forms both ‘query’ for search engine, and part of historical network state description

11. Search engine and Network state repository
Use Search Engine to compare current state document to historical documented network states.
Search Engine will use Singular Value Decomposition (SVD) techniques for dimension reduction of attribute space
Also experimenting with random projection methods
Output of Search Engine
“Top-K” list of similar documents, together with distance or similarity.
If current state is sufficiently similar to past attack, send attack type and location to Response Engine for impact analysis
Build Network state repository
Construct schema to support search
Addresses ‘Curse of dimensionality’ in anomaly detection

12. High dimensional Search via SVD on Labeled Clusters
1.0
Normal
Insider
DoS
Worms S8
S10
S13
S14 S9 S4
S15
query S1
1.0 S2 S6
S16 S5
S11
S12 S7
S17 S3

13. Insider analyzer and modeler
New formal threat model that captures the organizational information aspect of the insider threat
Threat model based on new graph model
This model allows analysis of the following questions:
What are likely/feasible attack paths?
What is the corresponding difficulty (‘cost’) of each such attack?
This component helps determine:
Which parts of the organization need more careful monitoring?
Which security policies need to be reinforced?
Insider analyzer and modeler may also guide placement of sensors, and help label clusters of network states

14. Impact Analysis using Response Engine
Building upon Smart Firewalls technology from Dynamic Coalitions program
Response Engine has overview of current network configuration
Response Engine logically validates Policies, expressed in terms of end-to-end service availability
Response Engine generates candidate reconfigurations to comply with Policies as much as possible
In this project
Detected attack type and location is translated into its effect on the current network configuration
E.g. Server failure due to a Denial of Service attack
Response Engine can analyze the impact of both the attack and its candidate responses on the availability of critical resources
Administrator can push response into the network

15. Technical challenges
We are testing a new hypothesis of whether search engine techniques can be used effectively for this problem.
Our key insight is that network attacks are often similar to each other but it is hard to predict what the small change is.
Telcordia has extensive experience with SVD based searches in text-based information retrieval. Here we are testing SVD search technology in a new domain.
Training search engine: tune distance metrics, label data, reduce false alarms
New ‘Insider attack-information graph’ problem is hard

16. How do we know if we succeeded?
Example Scenario:
1. Launch known insider attack in one part of testbed network and tag the data.
2. Launch the same attack in a different part of testbed network.
3. Detect attack using Search Engine.
4. Analyze impact on network using Response Engine
5. Respond using appropriate configuration change (similar to response to old attack)
Success = detection and appropriate response to attack after completing steps 3 through 5.
Repeat the experiment changing different parameters of the attack such as topology, location, scale, source/target choices, and finally attack vector.

17. Summary
If we succeed, we will have a system that defends critical services and resources against insiders, which
Can detect attacks by correlating large numbers of sensor measurements, and
Can synthesize appropriate pro-active responses to protect critical services while minimizing collateral damage.