1. Security Monitoring Thru Log Analysis Sam.NG@PISA

3. What the hacker did Startup script was modified, a line is added to the rc.local./usr/bin/t0x5mm (I cant remember the exact name) ls the file, nothing showed I tried to remove the file by rm, rm ran without any error Many months later, I knew thats something call rootkit

4. Log was gone! Some of the log files are truncated I wanted to know how the hacker got in, and I redirected syslog to /dev/lp0 and ran for a couple of weeks Several inches Several inches of paper printed, but there is more than I could handle I cant remember how, but finally I knew the hacker got in by exploiting an IMAP4 vulnerability

5. Lesson Learnt You will need the log in sometime And better still, store it in a safe place Write once read many (WORM) is good, but searching capability is even more important And the most important: you have to prepare it beforehand

6. But how can I use the log for security monitoring?

7. The Challenge I dont have the staff to handle 140,000 alerts. I dont even have the staff to handle 12,000 alerts David MacLeod, Ph.D., CISSP The Regence Group CISO

8. Reference: Counterpanes Security Monitoring Service Bruce Schneier

9. Block Diagram Phase I Phase III Phase IV Phase II Log Collection Noise Reduction Data Mining Alert/Ticket Management

10. Log Collectors Data Mining Alert/Ticket Management Noise Reduction

11. Log Collectors DATABASE Send your log to DATABASE Can be direct (open database connection from the client) or in-direct (e.g. post data to a web application) Each different log may need a different log collector Many logs have built-in support to store data in database

12. Unix syslog syslog can be redirected Can be redirected to printer! (/dev/lp0 in Linux) Can be redirected to remote syslog host But redirect to program (pipe) seems not working!!?? syslog-ng support logging directly to database Win32 solutions to accept remote syslog records and log to database Plain text format, easy to write a program to watch the data and then send to database

13. Demo A smiple tail base perl script to monitor new records in syslog and send it to remote database

14. Windows Eventlog Microsoft logparser can log to database WMI interface allows you to query remote eventlog Many third party solutions support logging to database

15. Demo A smiple C# program to monitor new records in Eventlog and send it to remote database

16. Quiz 1 A.Windows NT B.Windows 2000 C.Windows 2003 D.None of the above Which of the following Windows Server will pass Microsoft Baseline Analyzer (MSBA) 2.0 auditing requirements with default install?

17. DO YOU KNOW? You can pass MSBA 2.0 if you enabled Success and Failure auditing for the Account Logon Events But indeed, it recommends you to enable Success and Failure auditing: Account Logon Events, Account Management, Policy Change, System Events Failure auditing: Directory Service Access, Object Access Data volume may be quite high, a server config in this way may generate a eventlog for every 2 seconds (actual volume depends on your server)

18. Microsoft IIS Web Server Native support log to ODBC (but seems to be only available on server platform only)

19. Microsoft IIS Web Server (2) Microsoft does not recommend logging to database if the IIS is busy (Q245243) But nowadays most web pages are generated with SQL queries Test ODBC logging on your own environment PrepWebLog (Q296093) convert lIS log to SQL insertion statements in plain text format, but still have to run these SQL insertion by some other means Still, the best would be realtime logging to database Writing a tail -f similar program in IIS is difficult

20. DO YOU KOWN? IIS log file is updated in 64k chunks. On servers that do not have a high usage rate, the statistics will not be up-to-date because of the delayed write (Q142557) When it flush, may be only first 20k of the chunk contains data, the rest 44k data is \0 and will be filled in next flush Obviously a mmapped file, may be due to performance consideration… but

21. Suggestions to programmer In C/C++, STDERR, cerr are not buffered In fact, error log should be send out immediately, should not be cached/buffered # man stderr …… …… CONSIDERATIONS The stream stderr is unbuffered. The stream stdout is line- buffered when it points to a terminal. Partial lines will not appear until fflush(3) or exit(3) is called, or a newline is printed. This can produce unexpected results, especially with debugging output. …… ……

22. J2EE Application Server Most application server support log4j or java.util.logging (JDK 1.4+) log4j natively support logging to database thru JDBCAppender

23. My Experience I have a program developed since JDK1.3 At that time, Java dont have java.util.logging And I dont know much about log4j I wrote my own log handling routine (similar to syslog) to insert my own records into database But is not configurable/adaptable, and is not consolidated with the Application Servers log Since logging is widely spread all over the codes, it is very difficult for me to change the program to use these new technology DESIGN PHASE Log architecture should be planned in DESIGN PHASE

24. Snort IDS Comes with database output plugin to send alerts to database The packet analysis thread is also responsible for database insertion

25. My Experience I have experience in using Snort to monitor a ~20M Internet link, with database output plugin, default rules CPU is just about 30%, seems good When I changed to log to local file, CPU sharp jump to 100% Barnyard seems to be a solution but it does not support database!!

26. And many others Checkpoint FW1: thru LEA enabled product http://www.opsec.com/ Microsoft ISA: default support ODBC logging (KB838710) Microsoft Exchange: same as Eventlog Apache httpd: mod_log_sql

27. Noise Reduction Very important Noise will kill the system Noise will kill the system And to improve overall performance Can be done on several layers 1. ignore certain message at the log collection sender 2. ignore certain message at the log collection receiver end 3. delete (mark as ignore) at the database 4. fine tune your IDS rule, firewall logging policy etc.. 5. exception cases for data mining phase

28. Data Mining Data processing using sophisticated data search capabilities and statistical algorithms to discover patterns and correlations in large preexisting databases; a way to discover new meaning in data From http://www.tfd.com

29. Data Mining (2) The heart of Security Monitoring A board term, a general concept Utilize database queries to get the information you want Can be an external program, can be a SQL server schedule job

30. Data Mining Techniques Data Mining Attack Detection Attack Definition Event Correlation Statistical Analysis Anomaly Detection Normal Definition Event Correlation Statistical Analysis

31. Attack Definition GET /..%c0%af../winnt/system32/cmd.exe?/c+dir+c:\ GET /MSADC/root.exe?/c+dir GET /_mem_bin/..%255c../..%255c../..%255c../winnt/system32/cmd.exe?/c+dir GET /_vti_bin/.%252e/.%252e/.%252e/.%252e/winnt/system32/cmd.exe?/c+dir+c:\ GET /_vti_bin/..%255c../..%255c../..%255c../winnt/system32/cmd.exe?/c+dir GET /_vti_bin/..%c0%af..%c0%af..%c0%af..%c0%af..%c0%af../winnt/system32/cmd.exe?/c+dir+c:\ GET /_vti_bin/..%c0%af../..%c0%af../..%c0%af../winnt/system32/cmd.exe?/c+dir+c:\ GET /_vti_cnf/..%c0%af..%c0%af..%c0%af..%c0%af..%c0%af../winnt/system32/cmd.exe?/c+dir+c:\ GET /adsamples/..%c0%af..%c0%af..%c0%af..%c0%af..%c0%af../winnt/system32/cmd.exe?/c+dir+c:\ GET /cgi-bin/..%255c../..%255c../..%255c../winnt/system32/cmd.exe?/c+dir+c:/ GET /cgi-bin/..%c0%af..%c0%af..%c0%af..%c0%af..%c0%af../winnt/system32/cmd.exe?/c+dir+c:/ GET /cgi-bin/..%c0%af..%c0%af..%c0%af..%c0%af..%c0%af../winnt/system32/cmd.exe?/c+dir+c:\ GET /cgi-bin/..%f0%80%80%af../winnt/system32/cmd.exe?/c+dir+c:/ GET /iisadmpwd/..%c0%af..%c0%af..%c0%af..%c0%af..%c0%af../winnt/system32/cmd.exe?/c+dir+c:\ GET /iisadmpwd/..%c0%af../..%c0%af../..%c0%af../winnt/system32/cmd.exe?/c+dir+c:\ GET /msadc/.%252e/.%252e/.%252e/.%252e/winnt/system32/cmd.exe?/c+dir+c:\ GET /msadc/..%c0%af../..%c0%af../winnt/system32/cmd.exe?/c+dir+c:\ GET /msdac/root.exe?/c+dir+c:\ GET /scripts/..%e0%80%af../winnt/system32/cmd.exe?/c+dir+c:\ GET /scripts/..%f0%80%80%af../winnt/system32/cmd.exe?/c+dir+c:\ IIS Unicode Directory Traversal (cmd.exe) attack strings

32. Vulnerability characteristic attack characteristic CodeRed I [/default.idq?NNNNN……] CodeRed II [/default.idq?XXXXX……] In fact, the vulnerability can be exploited if the variable name is around 240 bytes Regular Expression [\/default\.idq\?.{240,}] Difficult to write an effective and accurate definition And not all attack leave audit trail

33. Event Correlation E.g. 1000 login failures followed by ONE successful login from the same IP E.g. IF http_response_code = 500; THEN find_all_other_url_accessed(); ENDIF

34. Anomaly Detection

35. Normal Definition You define what is normal and then monitor it E.g. Operators login time should be corresponding to their shift duty E.g. All server services should not be restarted unless necessary (ignore service start within 3minutes of system startup)

36. Statistical Analysis E.g. on average a event occurs 10 time a day, with a standard deviation of 2.3. But today we have 2000 records. E.g. anything happens more than 200 times in the past 30 minutes E.g. a event never seen in the past 7 days E.g. TOP 10 events/users/hosts, etc…

37. Alert/Ticket Management Works like Bug Tracker Save the alert as a Ticket A ticket is a something like an outstanding job Assign the ticket to a staff to follow up Escalate it if remain unresolved for some time And dont forget People Management (Time and Skill)

38. Alert/Ticket Management (2) According to my experience, at least 90% of the alerts generated by data mining are still FALSE ALARMS People will get use to it and tend to think EVERY alerts are false alarms If possible, fine tune the system to eliminate the false alarm from occurring again

39. Couterpane 2003 Results

40. DO YOU KNOW? TSL providing Email to pager service for about HK$80/mth

41. How should I start? Do it step by step, phase by phase, EventLog, syslog are easy to start with A group brain storming section would give you at least 10 such data mining rules, and is a good starting point Security Monitoring is a (long term) process, do not regard it as a single one-shot install and forget project

42. Contact: samngms@yahoo.com