1. School of Computer Science and Information Systems
Identifying Malicious Web Requests through Changes in Locality and Temporal Sequence DIMACS Workshop on Security of Web Services and E-Commerce
Li-Chiou Chen
School of Computer Science and Information Systems
Pace University
May 4th, 2005

2. Needs for anomaly detection in distributed network traces
The fast spreading Internet worms or malicious programs interrupts web services
Early detection and response is a vital approach
These attacks are usually launched from distributed locations
Network traces left at distributed locations are invaluable for searching clues of potential future attacks
E.g. Dshield, the Honeynet Project
© Li-Chiou Chen, 5/6/2005

3. Types of IDS Based on data Based on detection techniques
Network-based IDS
Monitors and inspects network traffic
Host-based IDS
Runs on a single host
Based on detection techniques
Signature-based IDS
Uses pattern matching to identify known attacks
Anomaly-based IDS
Uses statistical, data mining or other techniques to distinguish normal from abnormal activities
© Li-Chiou Chen, 5/6/2005

4. Outline
Toolkits for inferring anomaly patterns from distributed network traces
Previous works
Changes of locality over time
Markov chain analysis
Preliminary results
Summary
Future works
Focusing on anomaly detection in distributed network traces
TIAP
Malicious web requests
© Li-Chiou Chen, 5/6/2005

5. Locality pattern analysis Sequence pattern analysis
TIAP: Toolkits for inferring anomalous patterns in distributed network traces
Network traces (web log, tcpdump, etc)
Data conversion
Alerts from other IDS or TIAP peers
(using IDMEF)
Locality pattern analysis
Sequence pattern analysis
Response module
Alerts to other IDS or TIAP peers
(using IDMEF)
Alerts to administrators
© Li-Chiou Chen, 5/6/2005

6. Web level IDS Anomaly detection Misuse detection
Structure of a HTTP request (Kruegel and Vigna 03)
Normality on streams of data access patterns (Sion et al 03)
Misuse detection
State transition analysis of HTTP requests (Vigna et al 03)
Look for attack signatures (Almgren et al 01)
© Li-Chiou Chen, 5/6/2005

7. Changes in locality patterns and temporal sequence patterns
where the web request is sent, such as the source IP address,
which web server is requested, such as the destination IP address
Temporal sequence
the order of requested objects during a given period of time
© Li-Chiou Chen, 5/6/2005

8. Locality pattern analysis in distributed network traces
ABAA
ABCD
KIKL
ABPO
t1: AB
t2: ....
t3: ….
t4: ….
© Li-Chiou Chen, 5/6/2005

9. An example: web traces in common log format from 6 web servers
tstamp, ip, server, doc_tpe, user_agent
62978, , 1, 2, 3
62979, , 1, 2, 3
62979, , 2, 2, 3
63001, , 1, 2, 3
……..
………
A session
© Li-Chiou Chen, 5/6/2005

10. Data profiles
6 web servers (2 of them have links to each other, 4 of them are independent)
One day web trace
One session: a distinct IP, 10 minutes interval
193,070 HTTP requests,
11,177 sessions
HTTP requests from outside of the organization
© Li-Chiou Chen, 5/6/2005

11. Locality pattern analysis
86 sessions by only two web bots
© Li-Chiou Chen, 5/6/2005

12. Markov chain analysis © Li-Chiou Chen, 5/6/2005
t1 t t3 t4 t t6 t7 t8 t9 t10 t t12 t t14 t15 t16 …………….
N S N S S O S N S O S N S S N S S …………………..
sampling window 1
sampling window 2 N S O
© Li-Chiou Chen, 5/6/2005

13. Data profiles 1 web servers One week web traces Window size 30
Reference list 30
© Li-Chiou Chen, 5/6/2005

14. Change of distinct IP over time- browsers
© Li-Chiou Chen, 5/6/2005

15. Change of distinct IP over time- web bots
© Li-Chiou Chen, 5/6/2005

16. Markov chain results 0.43(0.14) Old (O) 0.42(0.21) 0.43(0.17)
0.13 (0.10)
0.13 (0.08)
New (N)
Same (S)
0.40 (0.22)
0.06 (0.04)
0.83 (0.10)
0.18 (0.16)
© Li-Chiou Chen, 5/6/2005

17. Illustration of the state transition probability
© Li-Chiou Chen, 5/6/2005

18. Summary
The preliminary locality pattern analysis works well with identifying distinct web bot access patterns
The Markov chain analysis provides a way to infer attacks that utilize random IP addresses
A combination of the two approaches is needed
© Li-Chiou Chen, 5/6/2005

19. Ongoing works
Incorporate the analytical results for malware or intrusion detections
A distributed framework of data collection and information sharing for inferring malwares or intrusion attempts across servers/platforms/geographical locations
Collection of attack logs for analytical purpose
Use of the Intrusion Detection Message Exchange Format (IDMEF) for message changes among servers
© Li-Chiou Chen, 5/6/2005