Presentation is loading. Please wait.

Presentation is loading. Please wait.

Centralized Logging: Why Would I Want to Do That? Garrett Lanzy Access more information security training for campus technical staff and earn CEUs: its.mnscu.edu/security/training/

Similar presentations


Presentation on theme: "Centralized Logging: Why Would I Want to Do That? Garrett Lanzy Access more information security training for campus technical staff and earn CEUs: its.mnscu.edu/security/training/"— Presentation transcript:

1 Centralized Logging: Why Would I Want to Do That? Garrett Lanzy Access more information security training for campus technical staff and earn CEUs: its.mnscu.edu/security/training/

2 Outline Problems Centralized Logging at Metropolitan State – Architecture and Configuration – Other Considerations Real-Life Scenario – How Does Logging Help? Future Directions Conclusions Q&A Access more information security training for campus technical staff and earn CEUs: its.mnscu.edu/security/training/

3 Problems: Logging Issues Logs spread across servers and network devices Different devices use different log formats Default settings often omit important information Log data retention varies An “event” can involve numerous different devices – hard to search/correlate

4 Key Questions What should be logged? How can this data be gathered? How long should this data be retained? How can searching be simplified? An Answer: Centralized Logging Access more information security training for campus technical staff and earn CEUs: its.mnscu.edu/security/training/

5 Centralized Logging at Metropolitan State University Single central logging server (axe) – Linux (SUSE Linux Enterprise Server) – syslog-ng logging daemon – 30GB partition for logs Stores 65 days worth of logs for 60+ devices (including firewall logs for 2 campuses) – Physical server – Same server used for network monitoring (Nagios)

6 What is syslog? Logging architecture for Unix/Linux Developed since 1980s Provides services for local & centralized logging (single or multi-tier) Communication protocol standardized by IETF – RFC 3164 - “classic” BSD Unix syslog (UDP only) – RFC 5424 – TLS transport support – Neither defines format of log messages

7 What is syslog-ng? “Next generation” syslog Open-source & commercial versions Supports UDP, TCP, and TLS transports Flexible configuration options Many options for storing log data – Simple files – Structured files (CSV, etc) – Database interfaces

8 syslog Message Facilities syslog defines 24 types of sources for logs (called facilities) – common ones: – kernel – messages from OS kernel – user – user-level messages – mail – e-mail related messages – system – messages from system daemons/services – auth – security/authorization messages – syslog – messages generated by syslog itself – local0 – local7 – implementation-defined message Metro design creates separate files per facility for each device

9 syslog Message Severities Each message assigned a severity by creator Can be used to help filter by importance SeverityDescription 0Emergency: system is unusable 1Alert: action must be taken immediately 2Critical: critical conditions 3Error: error conditions 4Warning: warning conditions 5Notice: normal but significant condition 6Informational: informational messages 7Debug: debug-level messages

10 Metro Centralized Logging Design Based on Novell Cool Solutions article Centralized Syslogging with Syslog-NG on SUSE Linux by Scott Flowers http://www.novell.com/coolsolutions/feature/18044.html http://www.novell.com/coolsolutions/feature/18044.html Supports classic syslog on UDP port 514 & syslog over TCP using TCP port 5140 Simpler directory structure for easier searches Added automated compression via logrotate

11 Logging server syslog-ng config Default SuSE configuraton contains logging source src for messages originating on local server Metro design adds logging source nethosts for logs coming form other servers Adds logging destination byhost which writes logs for each host to a specific directory /var/log/HOSTS/hostname Log files named facilityname.log Local logs for central logging server not changed

12 syslog-ng.conf for nethosts source # define source for messages from network hosts Source nethosts { # # process log messages from network: # udp(ip(“0.0.0.0") port(514)); tcp(ip(“0.0.0.0") port(5140) keep-alive(yes) max_connections(100)); };

13 syslog-ng.conf for byhost destination # separate out hosts into individual log directories destination byhost { file("/var/log/HOSTS/$HOST/$FACILITY.log" owner(root) group(root) perm(0640) dir_perm(0750) create_dirs(yes) ); }; log { source(src); source(nethosts); destination(byhost); };

14 Configuring other devices/servers Use TCP over UDP when possible – Exception: Cisco ASA devices stop forwarding if syslog over TCP configured & not working Check server/device logging settings to make sure appropriate information is logged – Example: some servers only log failed login attempts, best to also log successful ones Make sure to adjust firewall configuration on centralized logging server

15 Configuring Linux Server - syslog syslog always sends via UDP (default port 514) Sample config to send messages for all facilities of severity informational or higher to central log server – add lines to syslog.conf: # send all logs of severity info or higher to logging server *.info @x.x.x.x

16 Configuring Linux Server - syslog-ng Define destination loghost for TCP port 5140 on logging server, send all messages there Add following lines to syslog-ng configuration: destination loghost { tcp("x.x.x.x" port(5140)); }; log { source(src); destination(loghost); };

17 Configuring Network Devices Most managed network devices support logging to central syslog server Many configurable via web/GUI interfaces Sample Cisco IOS configuration: Core-6500#conf term Enter configuration commands, one per line. End with CNTL/Z. Core-6500(config)#logging source-interface Vlan1 Core-6500(config)#logging x.x.x.x Core-6500(config)#end

18 Configuring ASA Firewall Logging MnSCU-owned ASA firewalls can be configured to log to campus logging servers - contact MnSCU network support team: hostmaster@mnscu.edu hostmaster@mnscu.edu For campus-owned devices, easy to set up with Cisco ASDM: – Configuration -> Device Management -> Syslog Servers – Configuration -> Device Management -> Syslog Setup

19 Configuring ASA Firewall Logging MnSCU-owned ASA firewalls can be configured to log to campus logging servers - contact MnSCU network support team: hostmaster@mnscu.edu hostmaster@mnscu.edu For campus-owned devices, easy to set up with Cisco ASDM: – Configuration -> Device Management -> Syslog Servers – Configuration -> Device Management -> Syslog Setup

20 Configuring Cisco ACS Logging Can configure logging for each “report name” To start: from web admin interface, select System Configuration -> Logging Select Configure link for syslog column of desired “report” Click Enable Logging checkbox Scroll down, enter IP address of server, set port to 514, and max bytes to 1024 Submit

21 Sample ACS Config (pt. 1)

22 Sample ACS Config (pt. 2)

23 Sample ACS Config (pt. 3)

24 Windows Servers Windows event logging traditionally has had no support for centralized logging Several add-on products available to send Windows events to syslog server – Snare for Windows – open-source product Microsoft has added capability for centralized event collection in Windows 2008 – for info, see: http://www.windowsecurity.com/articles/Centralized-Auditing-here-FREE.html

25 Snare Agent for Windows Snare Agent for Windows is open-source, free software (GNU Public License) Available at: http://www.intersectalliance.com/projects/SnareWindows/ http://www.intersectalliance.com/projects/SnareWindows/ 2 versions: – Snare for Windows: supports NT, 2000, XP, 2003 – Snare for Windows Vista: supports Vista, 2008, Windows 7 Either will run on 32 or 64-bit installations

26 Configuring Snare Agent

27 Windows Event Log Settings Security audit settings – clean Win2008 install

28 A better idea?

29 Other Considerations Clock synchronization Log rotation/space utilization Backups Network Monitoring

30 Clock Synchronization System/device clocks need to be synchronized in order to correlate log data Any state/MnSCU router can be used as an NTP time source Most network devices can sync to NTP Windows standalone servers or domain controllers should be synchronized to NTP http://support.microsoft.com/kb/816042

31 Linux NTP Configuration Linux VMs: try kernel option clock=pit Remove any references to local unsynchronized clock (127.0.0.1) Simple sample ntp.conf driftfile /var/lib/ntp/drift/ntp.drift # path for drift file logfile /var/log/ntp server 172.16.255.238 server 156.98.1.1

32 Log Rotation logrotate – Linux utility to rotate and compress log files Creates new log files when run and deletes after specified interval Metro configuration: – Rotate files daily – Delay compression for 1 day – Keep 65 days worth of logs

33 Log server logrotate.conf additions # rotate logs in /var/logs/HOSTS tree /var/log/HOSTS/*/*.log { daily compress delaycompress dateext missingok notifempty rotate 65 maxage 90 sharedscripts create 640 root root postrotate /etc/init.d/syslog reload endscript }

34 Real-Life Scenario – Have your received one of these? > Subject: Notice ID: 22-79162500 Notice of Unauthorized Use of Paramount Pictures Corporation Property > Date: Fri, 23 Apr 2010 10:07:58 -0700 > From: paramount-no-reply@copyright-compliance.com > Reply-To: paramount-no-reply@copyright-compliance.com > To: abuse@mnscu.edu > -----BEGIN PGP SIGNED MESSAGE----- > Hash: SHA1 > Notice ID: 22-79162500 > Notice Date: 23 Apr 2010 16:58:09 GMT > Minnesota State University System > Dear Sir or Madam: > > BayTSP, Inc. ("BayTSP") swears under penalty of perjury that Paramount > Pictures Corporation ("Paramount") has authorized BayTSP to act as > its non-exclusive agent for copyright infringement notification. > BayTSP's search of the protocol listed below has detected > infringements of Paramount's copyright interests on your IP > addresses as detailed in the below report.

35 … And to the Relevant Information: > Evidentiary Information: > Notice ID: 22-79162500 > Initial Infringement Timestamp: 23 Apr 2010 16:23:27 GMT > Recent Infringement Timestamp: 23 Apr 2010 16:23:27 GMT > Infringers IP Address: 199.17.228.252 > Protocol: BitTorrent > Infringed Work: She's Out of My League > Infringing File Name: She's Out of My League 2010 CAM-lu > Infringing File Size: 608227413 > Bay ID: d116ad77ba844d11a350055c9227e2b0d12a29d9|608227413 > Port ID: 45739 > Infringer's DNS Name: > Infringer's User Name:

36 How to proceed? 199.17.228.252 is a NAT address for the student wireless network on the Midway campus Search the Midway ASA firewall log to determine the local IP address Then search the ACS appliance log to find the user logged in

37 …Um, but wait…. Logs older than 1 day old are compressed, so need to decompress before they can be searched axe:~ # cd /var/log/HOSTS/mwpix/ axe:/var/log/HOSTS/mwpix # bzcat local4.log-20100423.bz2 >/tmp/mwpix.log axe:/var/log/HOSTS/mwpix # cd /var/log/HOSTS/172.16.210.49/ axe:/var/log/HOSTS/172.16.210.49 # bzcat auth.log-20100423.bz2 >/tmp/acs.log axe:/var/log/HOSTS/172.16.210.49 #

38 What does the ASA log look like? Apr 23 09:00:00 mwpix %ASA-6-305011: Built dynamic TCP translation from inside:10.15.38.69/49523 to outside: 199.17.228.252/20192 Apr 23 09:00:00 mwpix %ASA-5-304001: 10.15.38.69 Accessed URL 65.55.17.27:http://www.msn.com/ Apr 23 09:00:00 mwpix %ASA-6-302020: Built inbound ICMP connection for faddr 207.171.116.193/0 gaddr 199.17.231.41/0 laddr 199.17.231.41/0 Apr 23 09:00:00 mwpix %ASA-6-302021: Teardown ICMP connection for faddr 207.171.116.193/0 gaddr 199.17.231.41/0 laddr 199.17.231.41/0 Apr 23 09:00:00 mwpix %ASA-6-305011: Built dynamic UDP translation from inside:10.15.38.67/62560 to outside:199.17.228.252/49397 Apr 23 09:00:00 mwpix %ASA-6-305011: Built dynamic UDP translation from inside:10.15.38.67/60379 to outside:199.17.228.252/41373 Apr 23 09:00:00 mwpix %ASA-4-313005: No matching connection for ICMP error message: icmp src inside:10.15.38.67 dst outside:199.17.241.241 (type 3, code 3) o n inside interface. Original IP payload: udp src 199.17.241.241/53 dst 10.15.38.67/60379. Apr 23 09:00:00 mwpix %ASA-6-305011: Built dynamic UDP translation from inside:10.15.35.98/50165 to outside:199.17.228.252/38489 Apr 23 09:00:00 mwpix %ASA-7-609001: Built local-host outside:74.63.145.159

39 We’re looking for…. … a line that contains: 199.17.228.252/45739 … at about 16:23:27 GMT (11:23:27 CDT)

40 grep is your friend! grep – utility to search by regular expression Some basic regular expression syntax: – Most characters “match” themselves – Exceptions: [\^$.|?*+() – \ is an escape character; meaning varies by context, always suppresses special meaning of [\^$.|?*+() –. matches any single character (except the line break character) – * repeats the previous item 0 or more times – …. So.* matches “anything” http://www.regular-expressions.info/reference.html

41 grep in action axe:/tmp # grep 199.17.228.252/45739 mwpix.log Apr 23 08:49:17 mwpix %ASA-6-305011: Built dynamic TCP translation from inside:10.15.34.45/1701 to outside:199.17.228.252/45739 Apr 23 08:49:47 mwpix %ASA-6-305012: Teardown dynamic TCP translation from inside:10.15.34.45/1701 to outside:199.17.228.252/45739 duration 0:00:30 Apr 23 11:26:23 mwpix %ASA-6-305012: Teardown dynamic TCP translation from inside:10.15.38.87/50372 to outside:199.17.228.252/45739 duration 0:03:00 Looks like we have a match - a connection of duration 3 min. which ended at 11:26:23 – i.e., was initiated at 11:23:23 (vs. 11:23:27)

42 Now check the ACS log (note: ACS timestamps are GMT, not local time) axe:/tmp # grep 10.15.38.87 acs.log Apr 23 16:19:06 172.16.210.49 CisACS_01_ PassedAuth y2gjm36g 1 0 Message-Type=Authen OK,User-Name= vvvvvv,NAS-IP- Address=172.16.212.7,Caller-ID= 10.15.38.87,NAS-Port=vvvvvv,Group- Name=Default Group,Filter Information=No Filters activated., Apr 23 16:19:06 172.16.210.49 CisACS_03_RADIUSAcc y2gjm36h 1 0 User- Name= vvvvvv,NAS-IP-Address=172.16.212.7,NAS-Port=29,Group- Name=Default Group,Framed-IP-Address=10.15.38.87,Calling-Station- Id=10.15.38.87,Acct-Status-Type= Start,Acct-Session- Id=4bd1c873 /00:24:2b:ae:79:c1/38738, Apr 23 18:05:44 172.16.210.49 CisACS_03_RADIUSAcc xjgh1n40 1 0 User- Name= vvvvvv,NAS-IP-Address=172.16.212.7,NAS-Port=29,Group- Name=Default Group,Framed-IP-Address= 10.15.38.87,Calling-Station- Id=10.15.38.87,Acct-Status-Type= Stop,Acct -Input- Octets=495945068,Acct- Output-Octets=1553689408,Acct-Session- Id=4bd1c873/00:24:2b:ae:79:c1/38738,Acct-Session-Time=6398,Acct- Input-Packets=1155133,Acct-Output-Packets=1692907, axe:/tmp #

43 Determination User vvvvvv made the connection outlined in the e-mail and downloaded 1.5GB – this case will be handled as inappropriate use Actual time to perform the log search process (including reading the e-mail, logging in to the logging server, decompressing the logs, and searching): less than 5 minutes

44 Future Directions Centralized logging for Windows workstations Use TLS connections instead of TCP – Makes snooping much more difficult – Prevents insertion of “bogus” logs Automated log monitoring/proactive notification Log to database instead of stream files? – Disk space is cheap, right?

45 Conclusions Centralized logging is not that difficult to set up Significant benefits for both normal operations and incident handling Can save large amounts of time for network/server/security administrators

46 Questions? Garrett Lanzy Through 5/11/10: Garrett.Lanzy@metrostate.edu Garrett.Lanzy@metrostate.edu 5/12/10 & After: Garrett.Lanzy@csu.mnscu.edu Access more information security training for campus technical staff and earn CEUs: its.mnscu.edu/security/training/


Download ppt "Centralized Logging: Why Would I Want to Do That? Garrett Lanzy Access more information security training for campus technical staff and earn CEUs: its.mnscu.edu/security/training/"

Similar presentations


Ads by Google