1. DERBI 13 December 1999 1999 Intrusion Detection Evaluation Joint DARPA ID/SIA PI Meeting 1 DERBI: Diagnosis, Explanation and Recovery from Break-Ins Mabry Tyson Pauline Berry Nate Williams Doug Moran David Blei Artificial Intelligence Center SRI International 333 Ravenswood Avenue Menlo Park CA 94025 http://www.ai.sri.com/~derbi Tyson@AI.SRI.COM

2. DERBI 13 December 1999 1999 Intrusion Detection Evaluation Joint DARPA ID/SIA PI Meeting 2 DERBI Objective Assist SysAdmin after an attack –No special security expertise required –Detailed system analysis as though by a OS/security expert –For sites that didn’t think they needed a real-time ID system Require nothing beyond off-the-shelf OS –No special logging or monitoring Provide guidance on what happened and how to recover How much info can be detected after-the-fact?

3. DERBI 13 December 1999 1999 Intrusion Detection Evaluation Joint DARPA ID/SIA PI Meeting 3 System Description Rules specify bits of evidence and associated exploit Rule Graph embodies relationships of evidence and attack goals –Beliefs of evidence combined to generate overall belief of attack Anthropomorphic characterization of system –Head - High level control –Body - Passes messages between Head and Feet –Feet - Runs around and does the work

4. DERBI 13 December 1999 1999 Intrusion Detection Evaluation Joint DARPA ID/SIA PI Meeting 4 Head Uses PRS (Procedural Reasoning System) Operates on rule graph –Goal is to determine whether attack happened –Goal is achieved by acquiring evidence Handles user interaction –User can add evidence –Rules can query user –Results presented to user –User can drill down

5. DERBI 13 December 1999 1999 Intrusion Detection Evaluation Joint DARPA ID/SIA PI Meeting 5 Body Allows Head to deal with abstract queries Allows Feet to deal with O/S specific queries Deals with multiple hosts –Network communications –Time differences –File system differences

6. DERBI 13 December 1999 1999 Intrusion Detection Evaluation Joint DARPA ID/SIA PI Meeting 6 Feet O/S specific –Knows how to traverse file system Careful to collect file info before altering it –Understands special file locations –Parses log files ID Evaluation primarily exercises the Feet Solaris & Linux –Only Solaris used in ID Evaluation

7. DERBI 13 December 1999 1999 Intrusion Detection Evaluation Joint DARPA ID/SIA PI Meeting 7 Rule Graph The presented slide is not included here -- it could not be adequately converted into a graphic that could be included in a MS PowerPoint file. This slide showed a graph with a large number of nodes representing rules, and was intended to show that although the rules formed a predominantly hierarchical structure, there was substantial crossing-over of the boundaries. A PostScript version of this graph can be found at http://www.ai.sri.com/~derbi/presentations/idpi9912/derbi- graph-1999dec.ps

8. DERBI 13 December 1999 1999 Intrusion Detection Evaluation Joint DARPA ID/SIA PI Meeting 8 Example Evidence Rule: EJECT buffer overflow EVIDENCE-TYPE (exploit (setuid root) buffer-overflow) UNIQUE-NAME eject-1 EVALUATION-NAME eject PATHS (follow-links '("/usr/bin/eject")) EVIDENCE ( ((not (and (command-version-vulnerable-p DIR FILE) ;; not vulnerable command or (window-of-opportunity (TimeAccessed PATH)))) ;; not used in interval of interest 0 0) ;;; assign 0% probability to command being used and 0% believe that it was ((greater-than (TimeAccessed PATH) ;;; use is later than (max (TimeModified "/cdrom") (TimeModified "/floppy"))) ;;; expected effects 40 100)) ;;; 40% probability of exploit, no change in believe about whether it was exploited POSIT ((posit ((TIME (TimeAccessed PATH))) (compromised-shell "root" TIME *unknown-time*))) EXPLANATION (next slide)

9. DERBI 13 December 1999 1999 Intrusion Detection Evaluation Joint DARPA ID/SIA PI Meeting 9 Evidence Rule: EJECT buffer overflow (cont) UNIQUE-NAME eject-1 PATHS (follow-links '("/usr/bin/eject")) EXPLANATION (explain-evidence ( PATH ;;; variable declarations (TIME (print-unix-time (TimeAccessed PATH))) (TIME2 (print-unix-time (TimeModified "/cdrom"))) (TIME3 (print-unix-time (TimeModified "/floppy"))) ) (TimeAccessed PATH) ;;; “as-of” time "The command ~S is version vulnerable to a buffer overflow attack and appears to have been used at time ~A which is more recent than two associated files: /cdrom (~A) and /floppy (~A)." PATH TIME TIME2 TIME3)

10. DERBI 13 December 1999 1999 Intrusion Detection Evaluation Joint DARPA ID/SIA PI Meeting 10 Example Output for an Attack Time: 08-Apr-1999 13:11:57 EDT Exploit: Suspicious-login (Suspicious-login) Login was found for user "doireano" from host 194.27.251.21. This user not seen before. ------------------------------------------------------------ +00:12:05 later Time: 08-Apr-1999 13:24:02 EDT Exploit: FORMAT (FORMAT-1) The command "/usr/bin/fdformat" is a version vulnerable to a buffer overflow attack and appears to have been used at time 08-Apr- 1999 13:24:02 EDT which is more recent than the associated device: "/devices/sbus@1f,0/SUNW,fdtwo@f,1400000:c,r aw" (04-Mar-1999 11:52:23 EST). +00:02:17 later Time: 08-Apr-1999 13:26:19 EDT Exploit: Unauthorized/nonstandard file activity (FILEACT) 1 files were created with no obvious legitimate user having access. Root users currently are *None*. Normal users are (erink doireano ulandusm grzegors). Groups with a member logged in are *None*. Ignored logins are *None*. Groups with an ignored login are *None*. Files' owner: root Files's group: staff Protection: -rw------- /.sh_history

11. DERBI 13 December 1999 1999 Intrusion Detection Evaluation Joint DARPA ID/SIA PI Meeting 11 Checking a Suspect System DERBI

12. 13 December 1999 1999 Intrusion Detection Evaluation Joint DARPA ID/SIA PI Meeting 12 Data Sources for ID Evaluation File system is only source of information –System files –Log files –File system DERBI has capability to query operator –For example, compare file to backup version –Allow operator to indicate remote login normal or suspicious

13. DERBI 13 December 1999 1999 Intrusion Detection Evaluation Joint DARPA ID/SIA PI Meeting 13 Target System Configuration Files Passwd –Notes crackable passwords Hosts.equiv,.rhosts –Notes capability for passwordless logins –Notes world-writable system directories Crontab files –Notes programs run from crontab

14. DERBI 13 December 1999 1999 Intrusion Detection Evaluation Joint DARPA ID/SIA PI Meeting 14 Log Files utmpx, wtmpx, utmp, wtmp, lastlog –All compared for inconsistencies –Note logins without logouts –Note inconsistencies in tty usage –Note currently unknown users –Note remote logins from a new host for that user –Note failed logins

15. DERBI 13 December 1999 1999 Intrusion Detection Evaluation Joint DARPA ID/SIA PI Meeting 15 Log File Information Relationships utmp utmpx wtmp wtmpx lastlog syslog messages authlog sulog File system Shell Init Files cronlogcrontabs Partial redundancy of info Redundancy a common result of the evolution & growth of systems Use to check for tampering Also exposes changes to system clock

16. DERBI 13 December 1999 1999 Intrusion Detection Evaluation Joint DARPA ID/SIA PI Meeting 16 Log Files (2) Syslog, messages, authlog –sendmail messages (mailbomb, locally sent mail) –su times –sshd messages (failures, successful logins/logouts) –ntp anomalies –Verify time of log messages monotonic

17. DERBI 13 December 1999 1999 Intrusion Detection Evaluation Joint DARPA ID/SIA PI Meeting 17 File System Info Executables –Access times usually means execution –Comparison of suid execute-time vs data file access time –Checksums checked for vulnerable or replaced versions Normal files –File access/creation, owner and protection recorded for every file –Files that indicate login/logout are specially noted (dot files, pty and window system files) Special files –Known cracker file names (included deleted files) –Rarely used files that crackers may use

18. DERBI 13 December 1999 1999 Intrusion Detection Evaluation Joint DARPA ID/SIA PI Meeting 18 Evidence Correlated by Time File access/creation and log information sorted by time Unauthorized access detected when no authorized user known to be logged in at time files accessed or created –Complications: Background processes, servers and scheduled jobs Suid executables Attacks usually evident by clustering of evidence –Often see evidence of an exploit –Followed by evidence of unauthorized access to files –However, attack can be inferred from a single anomaly

19. DERBI 13 December 1999 1999 Intrusion Detection Evaluation Joint DARPA ID/SIA PI Meeting 19 Detection of New Attacks “New attack” means new exploit DERBI spots the intentional and secondary effects of the cracker on the system, after the (new) exploit Crackers often leave a large trail of evidence –Exploit files touched –Camouflage attempts often leave footprints –Data collectors & back doors often detectable –However, ID Evaluation attacks often are hit-and-run

20. DERBI 13 December 1999 1999 Intrusion Detection Evaluation Joint DARPA ID/SIA PI Meeting 20 Detectable Attacks Detects R2L, U2R, Data attacks on Solaris (and Linux) Can detect some DoS attacks when logged (mailbomb, ssh, or telnet attempts) Generally can only detect latest use of executables (i.e., only the last eject attack could be detected) Cracker or normal activity can destroy evidence of attack Can’t detect network traffic but not blinded by encryption

21. DERBI 13 December 1999 1999 Intrusion Detection Evaluation Joint DARPA ID/SIA PI Meeting 21 ID Evaluation Results Test procedure artifacts complicated evaluation –Evaluation team affected file system (apparently including running attacks) outside of simulation runs but with clock set to times within simulation periods Dot files accessed and files written in a user’s directory but simulation contained no login Executables such as eject accessed without device accessed as though an attack was done, but no attack at that time during simulation –Also overwrote access times of all files on some days Simulated “attacks” were often just exercise exploit and leave –DERBI picks up evidence of usage of privileges

22. DERBI 13 December 1999 1999 Intrusion Detection Evaluation Joint DARPA ID/SIA PI Meeting 22 ID Evaluation Results 25 attacks in detectable classes 17 attacks detected –score of 16.98 (68%) 47 false alarms –score of 25

23. DERBI 13 December 1999 1999 Intrusion Detection Evaluation Joint DARPA ID/SIA PI Meeting 23 ID Evaluation Results - Misses 8 misses –1 attack missed due to test procedure overwriting access times ffbconfig –5 attacks left no evidence guessftp, xsnoop, xlock, httptunnel usage (x2) –2 attacks indistinguishable from normal activity httptunnel setup - no recognizable suspicious indications ps - telnet from a new host, but otherwise nothing suspicious

24. DERBI 13 December 1999 1999 Intrusion Detection Evaluation Joint DARPA ID/SIA PI Meeting 24 ID Evaluation Results - False Alarms 47 total false alarms (total score of 25) 29 probably due to test procedure (total score 15.2) –18 definite test procedure artifacts (score 4.55) –11 probable test procedure artifacts (score 10.65) 18 other false alarms (total score 9.8) –7 pseudo-tty errors (looked like log file truncation) (score 5.1) –5 login/logout record problems (score 3.6) –3 dot files accessed when user not logged in (score 0.03) –2 root accessed secret files in a sweep of file system (score 1) –1 secret access while logged in locally and remotely (score 0.05)

25. DERBI 13 December 1999 1999 Intrusion Detection Evaluation Joint DARPA ID/SIA PI Meeting 25 ROC - Overall Total Attacks: 25 Hits: 17 (16.98) Total FAs: 47 (25) Hits: 18 (17.98) Total FAs: 18 (9.8)

26. DERBI 13 December 1999 1999 Intrusion Detection Evaluation Joint DARPA ID/SIA PI Meeting 26 ROC - Old vs Overall Total Attacks: 23 Hits: 15 (15) Total FAs: 47 (25) Hits: 16 (16) Total FAs: 18 (9.8)

27. DERBI 13 December 1999 1999 Intrusion Detection Evaluation Joint DARPA ID/SIA PI Meeting 27 ROC - R2L Total Attacks: 12 Hits: 6 (6) Total FAs: 2 (1.7) Hits: 6 (6) Total FAs: 1 (0.7)

28. DERBI 13 December 1999 1999 Intrusion Detection Evaluation Joint DARPA ID/SIA PI Meeting 28 ROC - U2R Total Attacks: 11 Hits: 9 (9) Total FAs: 21(18.45) Hits: 10 (10) Total FAs: 10 (7.5)

29. DERBI 13 December 1999 1999 Intrusion Detection Evaluation Joint DARPA ID/SIA PI Meeting 29 ROC - Data Total Attacks: 3 Hits: 3 (2.98) Total FAs: 26 (6.53) Hits: 3 (2.98) Total FAs: 8 (2.28)

30. DERBI 13 December 1999 1999 Intrusion Detection Evaluation Joint DARPA ID/SIA PI Meeting 30 DERBI Project Ends DERBI has come to its end -- for now Experience at analyzing intrusions as a sysadmin led to the idea a system could be built to do this and to make it easier for less experienced sysadmins

31. DERBI 13 December 1999 1999 Intrusion Detection Evaluation Joint DARPA ID/SIA PI Meeting 31 DERBI is a Success Successful at detecting intrusions on a stock system –Original idea of a post-mortem analysis has been proven –Designed for real intrusions, it performs better the more the cracker does –Difficult to imagine how to further improve detection without modifying O/S

32. DERBI 13 December 1999 1999 Intrusion Detection Evaluation Joint DARPA ID/SIA PI Meeting 32 DERBI is Different The DERBI concept is orthogonal to most other ID systems –This diversity could be useful as the systems have different strengths and weaknesses –Didn’t fit too well with the design of the ID evaluation Not a substitute for intrusion monitoring systems, but can aid those sites that don’t want the overhead of such systems

33. DERBI 13 December 1999 1999 Intrusion Detection Evaluation Joint DARPA ID/SIA PI Meeting 33 Parting Thoughts The problem of intrusions has a variety of responses for a variety of consumers –Read-only systems or network computers –Brick-up-the-door approach –“We can’t let it happen” approach (most IDS) –“It happens” approach (DERBI) ID shouldn’t be an after-market add-on to an OS –Watch for incoming and outgoing attacks