1. CIT 470: Advanced Network and System AdministrationSlide #1 CIT 470: Advanced Network and System Administration Logging

2. CIT 470: Advanced Network and System AdministrationSlide #2 Topics 1.System logs 2.Logging policies 3.Finding logs 4.Syslog 5.Syslog servers 6.Log monitoring

3. CIT 470: Advanced Network and System AdministrationSlide #3 System Logs Logs record status and error conditions. Where do log messages come from? –Kernel –Accounting system –System services Logging methods: –Service records own logs (apache, cron). –Service uses syslog service to manage logs.

4. CIT 470: Advanced Network and System AdministrationSlide #4 Logging Policies 1.Throw away log data. 2.Save for a while, then throw away. 3.Rotate log files 4.Archive log files

5. CIT 470: Advanced Network and System AdministrationSlide #5 How to choose a logging policy? 1.Are there any data retention requirements? 2.How much disk space do you have? 3.How quickly do you need to retrieve logs? 4.Could you find the source of a security issue with the logs you keep?

6. CIT 470: Advanced Network and System AdministrationSlide #6 Throwing Away Not recommended. Leaves you unaware of: –Software and hardware problems –Security incidents It may take time to detect an incident. –Keep logs for at least a month or two.

7. CIT 470: Advanced Network and System AdministrationSlide #7 Rotation Keep backup files for each day/week logfile logfile.1 logfile.2 logfile.3 Rename files each day/week to move old ones back in list. –Compress rotated logs to save disk space. –Remove/archive logs that are X days old.

8. CIT 470: Advanced Network and System AdministrationSlide #8 Rotation #!/bin/sh cd /var/log mv logfile.2 logfile.3 mv logfile.1 logfile.2 mv logfile logfile.1 cp /dev/null logfile chmod 600 logfile

9. CIT 470: Advanced Network and System AdministrationSlide #9 logrotate Program to handle log rotation. –Run via /etc/cron.daily. –Configured via /etc/logrotate.conf. Options –How often to rotate –How long to keep logs –Compression or not –Log file permissions –Pre- and post-rotate scripts

10. CIT 470: Advanced Network and System AdministrationSlide #10 logrotate.conf # rotate log files weekly weekly # keep 4 weeks worth of backlogs rotate 4 # create new (empty) log files after rotating old create # uncomment if you want your log files compressed #compress # RPM packages drop log rotation information into include /etc/logrotate.d # no packages own wtmp -- we'll rotate them here /var/log/wtmp { monthly create 0664 root utmp rotate 1 }

11. CIT 470: Advanced Network and System AdministrationSlide #11 Archiving Logs Store logs to archival media (tape.) –Archive after X days/weeks. –Should be part of regular backup plan. –May want to save logs for all hosts together.

12. CIT 470: Advanced Network and System AdministrationSlide #12 Finding Logs Most logs are stored under –/var/log –/var/adm To find other logs, read startup scripts –/etc/init.d/* –and manuals for services started by scripts.

13. CIT 470: Advanced Network and System AdministrationSlide #13 Log fileProgramContents messagessyslogVarious program/kernel logs. syslog Various program/kernel logs. auth.logsu, ssh, loginAuthorization fail/success. lastloglogin, xdmLogins, commands. wtmploginLogin accounting data. acct/pacctkernelUNIX process accounting. Xorg.logX-WindowsX-Windows failures/info. Finding Logs

14. CIT 470: Advanced Network and System AdministrationSlide #14 Syslog Comprehensive logging system. Frees programmers from managing log files. Gives sysadmins control over log management. Sorts messages by Sources Importance Routes messages to destinations Files Network Terminals

15. CIT 470: Advanced Network and System AdministrationSlide #15 Syslog Components Syslog Daemon that does actual logging. Additional daemon, klog, gets kernel messages. openlog, syslog, closelog C library routines to submit logs to syslog. logger User-level program to submit logs to syslog. Can use from shell scripts.

16. CIT 470: Advanced Network and System AdministrationSlide #16 Example Syslog Messages Feb 11 10:17:01 localhost /USR/SBIN/CRON[1971]: (root) CMD ( run- parts --report /etc/cron.hourly) Feb 11 10:37:22 localhost -- MARK -- Feb 11 10:51:11 localhost dhclient: DHCPREQUEST on eth1 to 192.168.1.1 port 67 Feb 11 10:51:11 localhost dhclient: DHCPACK from 10.42.1.1 Feb 11 10:51:11 localhost dhclient: bound to 10.42.1.55 -- renewal in 35330 seconds. Feb 11 14:37:22 localhost -- MARK -- Feb 11 14:44:21 localhost mysqld[7340]: 060211 14:44:21 /usr/sbin/mysqld: Normal shutdown Feb 12 04:46:42 localhost sshd[29093]: Address 218.38.30.101 maps to ns.thundernet.co.kr, but this does not map back to the address - POSSIBLE BREAKIN ATTEMPT! Feb 12 04:46:44 localhost sshd[29097]: Invalid user matt from ::ffff:218.38.30.101

17. CIT 470: Advanced Network and System AdministrationSlide #17 Configuring Syslog Configured in /etc/syslog.conf Format: selector action Ex: mail.info /var/log/mail.log Selector components Source (facility) List of facilities separated by commas or *. Importance (level) Can be none or *

18. CIT 470: Advanced Network and System AdministrationSlide #18 /etc/syslog.conf # Log anything (except mail) of level info or higher. # Don't log private authentication messages! *.info;mail.none;authpriv.none;cron.none/var/log/messages # The authpriv file has restricted access. authpriv.*/var/log/secure # Log all the mail messages in one place. mail.*/var/log/maillog # Log cron stuff cron.*/var/log/cron # Everybody gets emergency messages *.emerg* # Save news errors of level crit and higher in a special file. uucp,news.crit/var/log/spooler # Save boot messages also to boot.log local7.*/var/log/boot.log

19. CIT 470: Advanced Network and System AdministrationSlide #19 Syslog Facilities FacilityUsed By kernThe kernel userUser processes (default) mailMail servers and related software. daemonSystem daemons (except mail, cron) authSecurity and authorization-related commands. lprPrint server and related commands. cronCron daemon. local0-7Eight local levels for other programs.

20. CIT 470: Advanced Network and System AdministrationSlide #20 Syslog Levels LevelMeaning emergPanic situations (hardware failure, crash) alertUrgent situations critCritical situations errNon-critical errors. warningWarnings. noticeMight merit investigation. infoInformational messages. debugDebugging (typically enabled temporarily.)

21. CIT 470: Advanced Network and System AdministrationSlide #21 Syslog Actions ActionMeaning filenameWrite message to file on local machine. @hostnameSend message to syslogd on hostname. @ipSend message to syslogd at IP address. user1,user2Write message to user screen if logged in. *Write message to all logged-in users.

22. CIT 470: Advanced Network and System AdministrationSlide #22 Testing Syslog stu> for i in {debug,info,notice,warning,err,crit,alert,emerg} > do > logger -p daemon.$i "Test message for daemon, level $i" > done stu> tail /var/log/daemon.log Feb 11 15:57:00 localhost stu: Test message for daemon, level debug Feb 11 15:57:00 localhost stu: Test message for daemon, level info Feb 11 15:57:00 localhost stu: Test message for daemon, level notice Feb 11 15:57:00 localhost stu: Test message for daemon, level warning Feb 11 15:57:00 localhost stu: Test message for daemon, level err Feb 11 15:57:00 localhost stu: Test message for daemon, level crit Feb 11 15:57:00 localhost stu: Test message for daemon, level alert Feb 11 15:57:00 localhost stu: Test message for daemon, level emerg

23. CIT 470: Advanced Network and System AdministrationSlide #23 Syslog Variants Some use m4 macros auth.noticeifdef(‘LOGHOST’, ‘/var/log/authlog’, ‘@loghost’) Red Hat Linux variants Allows spaces as separators. New operators: = (this priority only) Ex: mail.=info New operators: ! (except this pri and higher) Ex: mail.info,mail.!err

24. CIT 470: Advanced Network and System AdministrationSlide #24 Syslog NG Free drop-in replacement for syslog. More configurable Save logs to templated location (auto-rotates.) Filter logs based on program, time, message, etc. Message format customization. Allows easy logging to remote database. Improved networking TCP support as well as UDP. Improved security Doesn’t trust hostnames in remote messages. TCP transmission permits encrypted tunneling (stunnel.)

25. CIT 470: Advanced Network and System AdministrationSlide #25 Log Servers Collect all syslog data on one server. Allows logging to scale to large networks. Logs can be correlated across machines. Security-sensitive logs not on compromised host. Routers and diskless-hosts must log to a server. Need two syslog.conf files Client: sends all logs across network to server. Server: saves logs to database or local files.

26. CIT 470: Advanced Network and System AdministrationSlide #26 Log Monitoring Too much data for a human to process. Logs arrive 24x7 too. Use an automatic monitoring program Triggers on patterns found in log. Examples: logwatch, swatch # 3ware logs watchfor /(?i)3w-xxxx.+no longer fault tolerant/ mail=root,subject=LW warn: disk 3ware RAID not fault tolerant throttle 1:00:00,use=regex

27. CIT 470: Advanced Network and System AdministrationSlide #27 References 1.Michael Bower, Building Secure Servers with Linux, O’Reilly, 2005. 2.Aeleen Frisch, Essential System Administration, 3 rd edition, O’Reilly, 2002. 3.Jeremy Mate, “Log Analysis with Swatch,” http://sial.org/howto/logging/swatch/, 2005. http://sial.org/howto/logging/swatch/ 4.Jeremy Mate, “Logging with syslog-ng,” http://sial.org/howto/logging/syslog-ng/, 2005. http://sial.org/howto/logging/syslog-ng/ 5.Evi Nemeth et al, UNIX System Administration Handbook, 3 rd edition, Prentice Hall, 2001. 6.Shelley Powers et. al., UNIX Power Tools, 3 rd edition, O’Reilly, 2002. 7.Syslog-ng FAQ, http://www.campin.net/syslog- ng/faq.html.