1. Xinwen Fu Linux Logging Mechanisms 91.580.203 Computer & Network Forensics

2. CS@UML 2 Outline  Log files What need to be logged Logging policies Finding log files  Syslog: the system event logger

3. CS@UML 3 Who logs data?  The accounting system  The kernel  Various utilities All produce data that need to be logged Most of the data has a limited useful lifetime, and needs to be summarized, compressed, archived and eventually thrown away

4. CS@UML 4 Logging policies 1.Throw away all data immediately 2.Reset log files at periodic intervals 3.Rotate log files, keeping data for a fixed amount of time 4.Compress and archive to tape or other permanent media

5. CS@UML 5 Which policy to choose  Depends on: how much disk space you have how security-conscious you are  Whatever scheme you select, regular maintenance of log files should be automated using cron

6. CS@UML 6 1. Throwing away log files  Not recommend Security problems (accounting data and log files provide important evidence of break-ins) Helpful for alerting you to hardware and software problems  In general, keep one or two months In a real world, it may take one or two weeks for SA to realize that site has been compromised by a hacker and need to review the logs

7. CS@UML 7 2. Reset log files at periodic intervals  Most sites store each day’s log info on disk, sometimes in a compressed format  These daily files are kept for a specific period of time and then deleted  One common way to implement this policy is called “rotation”

8. CS@UML 8 3. Rotating log files  Keep backup files that are one day old, two days old, and so on. logfile, logfile.1, logfile.2, … logfile.6 Linux: /etc/logrotate.conf  Specify the frequency with which the files are reused  Each day rename the files to push older data toward the end of the chain

9. CS@UML 9 #! /bin/sh cd /var/log mv logfile.2 logfile.3 mv logfile.1 logfile.2 mv logfile logfile.1 cat /dev/null > logfile  Some daemons keep their log files open all the time, this script can’t be used with them. To install a new log file, you must either signal the daemon, or kill and restart it.  In Unix-like operating systems, /dev/null or the null device is a special file that discards all data written to it, and provides no data to any process that reads from it. In Unix programmer jargon, it may also be called the bit bucket or black hole. Script to archive 4 days files

10. CS@UML 10 4. Archiving log files  Some sites must archive all accounting data and log files as a matter of policy, to provide data for a potential audit  Log files should be first rotated on disk, then written to tape or other permanent media

11. CS@UML 11 Finding log files  To locate log files, read the system startup scripts : /etc/rc* or /etc/init.d/* If logging is turned on when daemons are run Where messages are sent  Some programs handle logging via syslog (syslogd or rsyslogd) Check /etc/syslog.conf (or rsyslog.conf on Fedora Core 9) to find out where this data goes

12. CS@UML 12 Finding log files (default configuration)  Different operating systems put log files in different places: /var/log/* /var/cron/log /usr/adm /var/adm …  On Linux, all the log files are in /var/log directory

13. CS@UML 13 Outline  Log files  Syslog: the system event logger how syslog works its configuration file debugging syslog the software that uses syslog

14. CS@UML 14 What is syslog  A comprehensive logging system, used to manage information generated by the kernel and system utilities  Allow messages to be sorted by their sources and importance, and routed to a variety of destinations: Log files, users’ terminals, or even other machines

15. CS@UML 15 Syslog: three parts 1.Syslogd: daemon that does the actual logging Configuration file: /etc/syslog.conf 2.API: openlog, syslog, closelog Library routines that programs use to send data to syslogd 3.logger User-level command for submitting log entries

16. CS@UML 16 Using syslog library routines write log entries to a special file /dev/log syslogd /etc/syslog.conf reads consults dispatches Log files Users’s terminals Other machines syslog-aware programs /dev/klog http://www.calpoly.edu/cgi-bin/man-cgi?syslogd Most system logging daemons listen on one or more Unix sockets, the most typical being /dev/log ; /dev/klog is kernel log socket

17. CS@UML 17 Configuring syslogd  The configuration file /etc/syslog.conf controls syslogd’s behavior  It is a text file with simple format, blank lines and lines beginning with ‘#’ are ignored (comment). selector action for example mail.info/var/log/maillog

18. CS@UML 18 Configuration file - selector  Identifies Program ‘facility’ that is sending a log message Messages’s severity level eg. mail.info  Syntax facility.level Facility names and severity levels must be chosen from a list of defined values

19. CS@UML 19 Configuration file - Facility Names FACILITYPROGRAMS THAT USE IT kernthe kernel userUser process, default if not specified mailThe mail system daemonSystem daemons authSecurity and authorization related commands lprthe BSD line printer spooling system newsThe Usenet news system

20. CS@UML 20 Configuration file - Facility names (Cont.) FACILITYPROGRAMS THAT USE IT uucpReserved for UUCP cronthe cron daemon markTimestamps generated at regular intervals local0-7Eight flavors of local message syslogsyslog internal messages authprivPrivate or system authorization messages ftpthe ftp daemon, ftpd *All facilities except “mark” UUCP stands for Unix to Unix CoPy.

21. CS@UML 21 Configuration file - Facility names (Cont.)  Facility - Mark: Timestamps can be used to log time at regular intervals (by default, every 20 minutes), so you can figure out that your machine crashed between 3:00 and 3:20 am, not just “sometime last night”. This can be a big help if debugging problems occur on a regular basis  Start at command line: syslogd –m 1  Use syslog.conf Start syslog daemon: syslogd Add the line to syslog.conf: mark.*/var/log/messages

22. CS@UML 22 Configuration file - severity level LEVELAPPROXIMATE MEANING emerg (panic)Panic situation alertUrgent situation critCritical condition errOther error conditions warningWarning messages noticeUnusual things that may need investigation infoInformational messages debugFor debugging severe not severe

23. CS@UML 23 Configuration file - selector  Levels indicate the minimum importance that a message must have in order to be logged mail.warning - would match all the messages from the mail system, at the minimum level of warning  Level of ‘none’ will exclude the listed facilities regardless of what other selectors on the same line may say. *.info;mail.noneaction All the facilities, except mail, at the minimum level info will subject to action

24. CS@UML 24 Configuration file – selector (Cont.)  Can include multiple facilities separated with ‘,’ commas e.g., daemon,auth,mail.infoaction  Multiple selectors can be combined with ‘;’ e.g. daemon.level1;mail.level2action Selectors are ‘|’ -- ORed together, a message matching any selector will be subject to the action  Can contain * - meaning all none - meaning nothing

25. CS@UML 25 Configuration file - action (Tells what to do with a message) ACTIONMEANING filenameWrite message to a file on the local machine @hostnameForward messages to the syslogd on hostname @ipaddressForward messages to the host at IP address user1, user2,…Write messages to users’ screens if they are logged in *Write messages to all users logged in

26. CS@UML 26 Configuration file - action (Cont.)  If a filename action used, the filename must be absolute path. The file must exist since syslogd will not create it e.g. /var/log/messages  If a hostname is used, it must be resolved via a translation mechanism such as DNS or NIS  While multiple facilities and levels are allowed in a selector, multiple actions are not allowed.

27. CS@UML 27 Config file examples (1) # Small network or stand-alone syslog.conf file # emergencies: tell everyone who is logged on *.emerg* # important messages *.warning;daemon,auth.info/var/adm/messages # printer errors lpr.debug/var/adm/lpd-errs

28. CS@UML 28 # network client, typically forwards serious messages to # a central logging machine # emergencies: tell everyone who is logged on *.emerg;user.none* #important messages, forward to central logger *.warning;lpr,local1.none@netloghost daemon,auth.info@netloghost # local stuff to central logger too local0,local2,local7.debug@netloghost # card syslogs to local1 - to boulder local1.debug@ialab.cs.uml.edu # printer errors, keep them local lpr.debug/var/adm/lpd-errs # sudo logs to local2 - keep a copy here local2.info/var/adm/sudolog Config file examples (2)

29. CS@UML 29 Sample syslog output 1.Mar 27 09:10:02 tcb-ia-lab-inst sshd[4100]: Accepted password for cis418 from ::ffff:216.254.235.105 port 61940 ssh2 2.Mar 27 18:10:00 tcb-ia-lab-inst sshd[9332]: Failed password for root from ::ffff:216.254.235.105 port 62817 ssh2 3.Mar 27 18:10:08 tcb-ia-lab-inst sshd[9332]: Accepted password for root from ::ffff:216.254.235.105 port 62817 ssh2 4.Mar 27 20:08:27 tcb-ia-lab-inst sshd[10629]: Accepted password for root from ::ffff:10.0.0.111 port 42172 ssh2 5.Mar 27 20:09:48 tcb-ia-lab-inst sshd[10649]: Failed password for root from ::ffff:10.0.0.111 port 48233 ssh2

30. CS@UML 30 Syslogd  A hangup signal (HUP, signal 1) cause syslogd to close its log files, reread its configuration file, and start logging again  If you modify the syslog.conf file, you must HUP syslogd to make your changes take effect ps -ef | grep syslogd Kill -1 pid-of-syslogd

31. CS@UML 31 Software that uses syslog PROGRAMFACILITYLEVELSDESCRIPTION amdautherr-infoNFS automounter dateauthnoticeDisplay and set date ftpddaemonerr-debugftp daemon gateddaemonalert-infoRouting daemon gopherdaemonerrInternet info server halt/rebootauthcritShutdown programs login/rlogindauthcrit-infoLogin programs lpdlprerr-infoBSD line printer daemon

32. CS@UML 32 Software that uses syslog PROGRAMFACILITYLEVELSDESCRIPTION nameddaemonerr-infoName sever (DNS) passwdautherrPassword setting programs sendmailmaildebug-alertMail transport system rwhodaemonerr-noticeromote who daemon suauthcrit, noticesubstitute UID prog. sudolocal2notice, alertLimited su program syslogdsyslog, markerr-infointernet errors, timestamps

33. CS@UML 33 Syslog 's functions  Liberate programmers from the tedious mechanics of writing log files  Put SA in control of logging Before syslog, SA had no control over what information was kept or where it was stored  Can centralize the logging for a network system

34. CS@UML 34 Debugging syslog -- logger  Useful for submitting log entries from shell scripts  Can also use it to test changes in syslogd’s configuration file. For example..

35. CS@UML 35 Add line to syslog.conf: local5.info/var/log/test.log verify it is working, run logger -p local5.info “test messages” a line containing “test messages” should be written to /tmp/test.log If this doesn’t happen: forgot to create the test.log file or forgot to send syslogd a hangup signal

36. CS@UML 36 Remote logging  On a central logging server: 10.0.0.192 syslogd -r  On a local server: 10.0.0.45 authpriv.*;auth.*@10.0.0.192 Question: where are those events written?

37. CS@UML 37 Process Accounting  accton is used to turn on or turn off process accounting  lastcomm tracks commands each user uses touch /var/log/pacct /sbin/accton /var/log/pacct lastcomm -f /var/log/pacct  ac prints out statistics about users' connection times in hours based on the logins and logouts in the current /var/log/wtmp file ac -p -d  sa summarizes accounting information from previously executed commands, software I/O operation times, and CPU times, as recorded in the accounting record file /var/log/pacct sa /var/log/pacct

38. CS@UML 38 Process Accounting (Cont.)  last goes through the /var/log/wtmp file and prints out information about users' connection times  lastb is the same as last, except that by default it shows a log of the file /var/log/btmp, which contains all the bad login attempts.

39. CS@UML 39 Using syslog in programs  openlog( ident, logopt, facility); Messages logged with the options specified by logopt begin with the identification string ident.  syslog( priority, messge, parameters…); Send message to syslogd, which logs it at the sepecified priority level  close( );

40. CS@UML 40 / * c program: syslog using openlog and closelog */ #include main ( ) { openlog ( “SA-BOOK”, LOG_PID, LOG_USER); syslog ( LOG_WARNING, “Testing …. “); closelog ( ); } On the host, this code produce the following log entry: Apr 4 15:21:57 tcb-ia-lab-inst SA-BOOK[7762]: Testing...

41. CS@UML 41 Summary  On linux, check following files: /etc/syslog.conf : syslog configuration file /etc/logrotate.conf : logging policy, rotate /etc/logrotate.d/* /var/log/* : log files  try following commands to find out more... man logrotate man syslogd

42. CS@UML 42 References 1.Chris Prosise, Kevin Mandia, Matt Pepe, Incident Response and Computer Forensics, Second Edition (Paperback), ISBN: 007222696X 2.Brian Hatch, Preventing Syslog Denial of Service attacks, http://www.hackinglinuxexposed.com/articles/20030220.html http://www.hackinglinuxexposed.com/articles/20030220.html 3.Albert M.C. Tam, Enabling Process Accounting on Linux HOWTO, 02/09/2001, http://www.faqs.org/docs/Linux-mini/Process- Accounting.htmlhttp://www.faqs.org/docs/Linux-mini/Process- Accounting.html 4.Keith Gilbertson, Process Accounting, 12/01/2002, http://www.linuxjournal.com/article/6144 http://www.linuxjournal.com/article/6144

43. CS@UML 43 Notes  Change host name /etc/hosts # add the host to the end of 127.0.0.1 /etc/sysconfig/network

44. CS@UML 44 #! /bin/sh cd /var/log mv logfile.2.Z logfile.3.Z mv logfile.1.Z logfile.2.Z mv logfile logfile.1 cat /dev/null > logfile kill -signal pid compress logfile.1 signal - appropriate signal for the program writing the log file pid - process id