1. Network Monitoring & Troubleshooting plus Log Analysis Faculty: Scott Greene of Evidence Solutions, Inc. Scott@EvidenceSolutions.com Scott@EvidenceSolutions.com www.EvidenceSolutions.com

2. ► 10 signs that you aren't cut out for IT  1: You lack patience 10 Signs you aren’t cut out for IT

3. U of Nebraska Incident ► An Undergrad suspected in Univ. of Nebraska breach where more than 650K personal records were compromised in attack. ► The intrusion was into a university database containing personal information on more than 650,000 students, parents and employees.

4. U of Nebraska Incident ► The intrusion, which was described by university officials as a "skilled attack," exposed the Social Security Numbers (SSNs), names, addresses, course grades financial aid and other information on students who attended the university since 1985.

5. U of Nebraska Incident ► The breach also exposed personal data and financial information for parents of students who applied for financial aid at UNL, according to the university. ► A staff member in UNL's Computing Services Network discovered the breach in the Nebraska Student Information System (NeSIS) on May 23.

6. U of Nebraska Incident ► An Undergrad suspected in Univ. of Nebraska breach where more than 650K personal records were compromised in attack. ► The intrusion was into a university database containing personal information on more than 650,000 students, parents and employees.

7. U of Nebraska Incident ► The system manages student admissions, campus housing and course registration. ► It was built over a three-year period at a cost of $29.9 million, has been operational for the past two years and is based on Oracle's PeopleSoft Enterprise Campus Solution platform.

8. U of Nebraska Incident ► An FAQ on the incident posted by the university makes it clear that personal data in the breached server was not encrypted. "However, we are confident that the type of attack we experienced would have bypassed any encryption that was in place," the university said.

9. U of Nebraska Incident ► The vulnerability that enabled the intrusion has since been closed and the university is currently working with a third-party firm to review and address remaining vulnerabilities.

10. 20 Critical Security Controls ► 1) Inventory of Authorized & Unauthorized Devices ► 2) Inventory of Authorized & Unauthorized Software ► 3) Secure Configurations for Hardware & Software on Laptops, Workstations, & Servers

11. 20 Critical Security Controls ► 4) Continuous Vulnerability Assessment & Remediation ► 5) Malware Defenses

12. 20 Critical Security Controls ► 6) Application Software Security  Code Reviews  Proper Logging  Abnormal operation reporting

13. 20 Critical Security Controls ► 7) Wireless Device Control ► 8) Data Recovery Capability ► 9) Security Skills Assessment and Appropriate Training to Fill Gaps ► 10) Secure Configurations for Network Devices such as Firewalls, Routers, and Switches

14. 20 Critical Security Controls ► 11) Limitation and Control of Network Ports, Protocols, and Services  Including custom applications  Development departments need to communicate with network departments

15. 20 Critical Security Controls ► 12) Controlled Use of Administrative Privileges ► 13) Boundary Defense  See 10  Include penetration testing  Include review of firewall rules  Remote Users  Mobile Devices

16. 20 Critical Security Controls ► 14) Maintenance, Monitoring, & Analysis of Security Audit Logs ► 15) Controlled Access Based on the Need to Know

17. 20 Critical Security Controls ► 16) Account Monitoring and Control  What do users have rights to ► Why?  What do processes have rights to ► Why?

18. 20 Critical Security Controls ► 17) Data Loss Prevention ► 18) Incident Response Capability  Who responds  Test those responses  Who gets notified ► Hr ► Legal

19. 20 Critical Security Controls ► 19) Secure Network Engineering aka “Develop a Secure Infrastructure” ► 20) Penetration testing

20. #14 ► Validate audit log settings for each hardware device and the software installed on it, ensuring that logs include:  Date  Timestamp  source addresses  destination addresses  Any other useful information

21. #14 ► Normalize Logs  Syslog  Common Event Expression initiative  Use normalization tools to convert logs

22. #14 ► Reports  Security personnel and/or system administrators should run weekly reports that identify anomalies in logs.  They should then actively review the anomalies, documenting their findings. ► A log for the log events

23. #14 ► Time Synch  Use at least two synchronized time sources  All servers and network equipment should be in synch. ► Test ► Validate their synchness

24. Federal Security Standards ► NIST Special Publication (SP) 800-37  Categorize ► The information system and the information processed, stored, and transmitted by that system based on an impact analysis

25. Federal Security Standards ► NIST Special Publication (SP) 800-37  Baseline ► Select an initial set of baseline security controls for the information system based on the security categorization; tailoring and supplementing the security control baseline as needed based on an organizational assessment of risk and local conditions.

26. Federal Security Standards ► NIST Special Publication (SP) 800-37  Implement ► The security controls and describe how the controls are employed within the information system and its environment of operation.

27. Federal Security Standards ► NIST Special Publication (SP) 800-37  Assess ► Assess the security controls using appropriate assessment procedures to determine the extent to which the controls are implemented correctly, operating as intended, and producing the desired outcome with respect to meeting the security requirements for the system.

28. Federal Security Standards ► NIST Special Publication (SP) 800-37  Authorize ► Information system operation based on a determination of the risk to organizational operations and assets, individuals, other organizations, and the Nation resulting from the operation of the information system and the decision that this risk is acceptable.

29. Federal Security Standards ► NIST Special Publication (SP) 800-37  Monitor ► The security controls in the information system on an ongoing basis including assessing control effectiveness, documenting changes to the system or its environment of operation, conducting security impact analyses of the associated changes, and reporting the security state of the system to designated organizational officials.

30. Federal Security Standards ► NIST Special Publication (SP) 800-37 ► The final step in the cycle, Monitor, is of particular importance because it evaluates the effectiveness of a security control. But what if you only performed this evaluation periodically—for example, to satisfy a quarterly or annual audit for a regulation or other compliance related demand? Unfortunately, it could be months or even a year before you’d realize that the security control was not functioning as intended.

31. Federal Security Standards ► NIST Special Publication (SP) 800-37  Assess ► Assess the security controls using appropriate assessment procedures to determine the extent to which the controls are implemented correctly, operating as intended, and producing the desired outcome with respect to meeting the security requirements for the system.

33. Some Things to Monitor ► Patch management ► Network management tools ► Security tools such as:  Change management  Configuration management  Log monitoring  Vulnerability scanning solutions

34. ► 10 signs that you aren't cut out for IT  1: You lack patience  2: You have no desire to continue your education 10 Signs you aren’t cut out for IT

35. Logs? ► Syslog is the predominant standard for computer system logging ► Microsoft, in its infinite wisdom chose their own called “Windows Event Log”.  There are several converters to convert the Windows Event Log to the Syslog standard.

36. Log Log Log ► Many incidents can be readily revealed with a bit of logging and analysis those logs.

37. Logs ► Solutions  Almost everything that has a log should have the log turned on.  Logs should include: ► Date/time ► Source IP ► Destination IP ► Port ► Etc

38. Logs ► Solutions  Use standard SYSLOG entries or use software that converts logs to a common log format.  Store logs for a while – space & DVDs are cheap  Create systems & procedures for analyzing logs. ► These systems should have ‘normal’ items and ‘abnormal’ items

39. Logs ► Solutions  All remote access logging: ► should be in detail ► Should be rigorously analyzed.  All security alerts should be logged. ► Workstation ► Servers ► Devices

40. Logs ► Solutions  Use unified time ► This allows logs to be matched up across many devices and / or networks.  Border devices ► Should log verbosely ► Should log all traffic  Blocked  Allowed

41. Logs ► Solutions  Logs should be secured  Logs should be exported & saved on Write Once devices. or  Logs should be written to dedicated logging servers.  The dedicated logging servers with separate security credentials

42. Logs ► Solutions  Test the logs and review after: ► Normal / acceptable traffic ► Push the system ► Attempt to penetrate the network.  Inside  Outside ► Compare and correlate the data on all of the logs for validity.

43. Logs ► Solutions  Review ► Logs everyday ► Use automated tools to analyze large amounts of data.  Test ► Attack a system ► Test the response time.  Discovery  Action taken to attack

44. Log Review Tools ► Windows -> Syslog conversion  agent (intersectalliance.com/projects/index.html) and remote collector (sourceforge.net/projects/lassolog) are used to convert Windows Event Logs into syslog, a key component of any log management infrastructure today (at least until Visa/W7 log aggregation tools become mainstream  Snare agent (intersectalliance.com/projects/index.html) and ProjectLasso remote collector (sourceforge.net/projects/lassolog) are used to convert Windows Event Logs into syslog, a key component of any log management infrastructure today (at least until Visa/W7 log aggregation tools become mainstreamintersectalliance.com/projects/index.htmlsourceforge.net/projects/lassologintersectalliance.com/projects/index.htmlsourceforge.net/projects/lassolog

45. ► 10 signs that you aren't cut out for IT  1: You lack patience  2: You have no desire to continue your education  3: You refuse to work outside 9-to-5 10 Signs you aren’t cut out for IT

46. Database Activity Management ► Database Activity Monitoring  (DAM) is a database security technology for monitoring and analyzing database activity that operates independently of the database management system (DBMS)  It does not rely on any form of native (DBMS- resident) auditing or native logs such as trace or transaction logs.  DAM is typically performed continuously and in real-time.

47. Database Activity Management ► Add prevention and you get (DAMP)  This extension to DAM goes beyond monitoring and alerting to also block unauthorized activities.  DAM helps organizations address compliance: ► HIPAA ► PCIDSS ► Sarbanes-Oxley (SOX) ► NIST 800-53

48. Database Activity Management ► Features include:  Event aggregation  Correlation  Reporting  Auditing ► Does not require access to native database audit functions

49. Database Activity Management ► Privileged User Monitoring:  Monitoring privileged users: ► DBAs ► Sysadmins ► Developers  who typically have unfettered access to corporate databases  Protects against external and internal threats

50. Database Activity Management  Monitors all activities and transactions  Identifies anomalous activities ► Viewing sensitive data ► Creating new accounts  (with superuser privileges?) ► Adding or Deleting tables

51. Database Activity Management ► Most organizations have perimeter protection ► The next need is to monitor and protect privileged user accounts

52. Database Activity Management ► There is a high correlation between and protection from the insider. ► Privileged users are capable of:  Stored procedures  Triggers  Views

53. Database Activity Management ► Targeted attacks frequently result in attackers gaining privileged user credentials:  Monitoring of privileged activities is also an effective way to identify compromised systems.

54. Database Activity Management ► Privileged user monitoring helps ensure:  Data Privacy  Integrity

55. Implementation of Logs ► Procedures and Tools to Implement and Automate this Control  Most Everything allows logging  Evaluate what is and what is not being logged. ► compare them with the asset inventory

56. Implemenatation of Logs ► Manual inspection  analyze logs on individual devices  correlation (SIEM) tools can make audit logs far more useful

57. Implementation of Logs ► SIEM & Consolidation tools can be quite helpful in identifying subtle attacks.  These tools are not a replacement for skilled information security personnel and system administrators.  Even with automated log analysis tools, human expertise and intuition are often required to identify and understand attacks.

58. Log Measurement - Manual ► Item: Network time protocol (NTP)  Measurement: Confirm that NTP is being used to synchronize time for all devices and that all clocks are in synch. ► Pass or fail.

59. Log Measurement - Manual ► Item: Vulnerability scanner  Measurement: Run a non intrusive vulnerability scanner against random servers. ► Review logs to determine whether the information appeared in the logs. Pass or fail.

60. Log Measurement - Manual ► Item: Security Event Information Management system (SEIM / SIEM / etc)  Measurement: Correlate logs to a central source and determine that all servers are properly logging. ► Compare to inventory list ► Start @ 100% and back off 5% for each device not logging.

61. SIEM ► security information and event management ► security incident and event management

62. SIEM ► SIEM  is a computerized tool used on data networks to centralize the storage and interpretation of logs, and events.  The logs and events are generated by other hardware and software products on the network.

63. SIEM ► SIEM should include:  Gathering the logs

64. SIEM ► SIEM should collect logs from:  Syslogs  Firewall logs  IDS logs  Windows server logs  Database logs  Web server logs  Application logs.

65. SIEM ► SIEM may need to:  Handle multiple data centers  Collect data centrally

66. SIEM ► SIEM Reports:  Correlating ► Their big value is in the correlation of data from multiple sources in multiple formats.  Regulatory  Trouble-shooting  Investigating  Alerting

67. SIEM ► SIEM can also detect:  Distributed attacks  Complicated attack paths  Insider abuse ► As well as:  Normal network performance failures ► (Requires a capable analyst)

68. Commercial Software Products ► CorreLog Enterprise Server – software $5k ► ManageEngine EventLog Analyzer $350 / 10 sources ► NitroSecurity NitroView $29k ► Prism Microsystems EventTracker $30k ► Tripwire Log Center $19k

69. ► 10 signs that you aren't cut out for IT  1: You lack patience  2: You have no desire to continue your education  3: You refuse to work outside 9-to-5  4: You don’t like people 10 Signs you aren’t cut out for IT

70. NMap ► Nmap  A security scanner originally written by Gordon Lyon (aka Fyodor Vaskovich) ► 1) used to discover hosts ► 2) user to discover services on hosts ► 3) can determine;  which ports are open and closed  the operating system  names and versions of the listening services  estimated uptime  type of device  presence of a firewall.

71. Nmap

72. NMap ► DEMO

73. ► 10 signs that you aren't cut out for IT  1: You lack patience  2: You have no desire to continue your education  3: You refuse to work outside 9-to-5  4: You don’t like people  5: You give up quickly 10 Signs you aren’t cut out for IT

74. Windows Log Analysis ► Free Products  (code.google.com/p/php-syslog-ng)  LogZilla (code.google.com/p/php-syslog-ng)code.google.com/p/php-syslog-ng  Analyzes Syslogs  Is PHP-based visual front-end ► For syslog servers  Searches  Reports  etc

75. Windows Log Analysis ► Free Products  Splunk ( Free for first 500mb )

76. Windows Log Analysis ► Free Products ► (ossec.net): ► OSSEC (ossec.net):ossec.net  an open source tool  Analyzes Real-Time: ► Unix systems ► Windows servers ► Network devices  Includes: ► default alerting rules

77. Windows Log Analysis ► Free Products  agent & (intersectalliance.com/projects/index.html) and remote collector (sourceforge.net/projects/lassolog)  Snare agent & ProjectLasso (intersectalliance.com/projects/index.html) and remote collector (sourceforge.net/projects/lassolog)intersectalliance.com/projects/index.htmlsourceforge.net/projects/lassologintersectalliance.com/projects/index.htmlsourceforge.net/projects/lassolog ► Open sourc tool ► Analyzes real-time:  Windows Event Logs  Syslog  Network devices

78. Windows Log Analysis ► Free Products  (log2timeline.net/)  Log2timeline (log2timeline.net/)log2timeline.net/  Analyzes Logs ► Used as an investigation tool it can create a timeline view out of raw log data ► Runs on Linux and Mac using Perl

79. Windows Log Analysis ► Free Products  (balabit.com/network- security/syslog-ng/)  syslog-ng (balabit.com/network- security/syslog-ng/)balabit.com/network- security/syslog-ng/balabit.com/network- security/syslog-ng/  Open Source  Analyzes ► Windows Logs by installing an agent on the server ► Syslog

81. SpiceWorks ► Spiceworks headquartered in Austin, Texas. ► It was formed in early 2006 to provide a Facebook-like community integrated with a free ad-supported IT:  Systems Management  Inventory  Help desk software

82. SpiceWorks ► The product is designed for network administrators working in small- to mid-sized businesses and managing up to a few thousand network devices.

83. SpiceWorks ► SpiceWorks discovers: ► Windows ► Unix ► Linux ► Mac OS X ► Routers ► VOIP phones ► Printers ► etc.

84. SpiceWorks ► DEMO

85. ► 10 signs that you aren't cut out for IT  1: You lack patience  2: You have no desire to continue your education  3: You refuse to work outside 9-to-5  4: You don’t like people  5: You give up quickly  6: You’re easily frustrated 10 Signs you aren’t cut out for IT

86. What exactly is WireShark? ► Wireshark is a sniffer  Sniffer, Packet Analyzer, also known as a network analyzer, protocol analyzer.  A software product and / or hardware product that has the ability to intercept all network traffic and allow for the analysis of the packets of data contained in that traffic.

87. What exactly is WireShark? ► Originally named Ethereal, Wireshark is a free and open-source packet analyzer. It is used for network troubleshooting, analysis, software and communications protocol development, and education ► It runs on all popular computing platforms, including Unix, Linux, and Windows.

88. What is a sniffer? ► Protocol Analyzer  Who is talking to who  What they are saying  Header and Overhead  Payload  Problems

89. Use of a sniffer for security: ► Basic Information:  7 layer OSI model ► Application ► Presentation ► Session ► Transport ► Network ► Datalink ► Physical layers

90. Use of a sniffer for security: ► Many believe that you look at everything from the bottom up just like:  Building a building  Math  Employment, etc ► But, that is not always true….  With a house you start with the overall design…. ► In the case of sniffing  It is best to start somewhere in the middle, usually at TCP or ICMP, then move down or up based upon what you discover and what you are looking for.

91. Requirements of a Sniffer ► Hubs ► Switches  Port Mirroring  Switched Port Analyzer (SPAN)  Roving Analysis Port (RAP) ► Local Connection must operate in promiscuous mode

92. Example Sequence 1 ► DHCP request and response ► ARP for gateway ► DNS “A” request and response ► Web session setup ► Payload delivery

93. DHCP

94. DHCP: src & dst

95. DHCP: IP Component

96. DHCP: UDP Component

97. DHCP: BootStrap Parameters

98. DHCP: First Packet Results

99. DHCP: Second Packet Results

100. DHCP: Third Packet Results

101. DHCP: Fourth Packet Results

102. ► 10 signs that you aren't cut out for IT  1: You lack patience  2: You have no desire to continue your education  3: You refuse to work outside 9-to-5  4: You don’t like people  5: You give up quickly  6: You’re easily frustrated  7: You can’t multitask 10 Signs you aren’t cut out for IT

103. DNS ► DNS is the internet’s system to translate the human readable name such as EvidenceSolutions.com to the actual IP address of the desired site.

104. DNS: Request

105. DNS: Response

106. We get to the website: syn

107. We get to the website: syn & ack

108. Website final: ack

109. Website Sends Graphics

110. How to use the product…. ► Basics out of the way  Start at the bottom & work your way up ► Works great for troubleshooting  Or start at the top an work your way down ► Not so good for security- top down is best here

111. ► Real world virus example  Multiple protocols  Disguised traffic  Peel the onion to analyze

112. Real World

113. STATISTICS- PROTOCOL HIERARCHY TCP CONVERSATIONS

115. ► Compare to your list of valid IP’s ► Compare to your list of valid protocols ► Compare with what you have seen in the past ► Drill down into any anomalies or unusual instances Real World

116. FTP Traffic High…

117. FTP Traffic High… to.10

118. FTP Traffic High… between.10 &.25

119. .25 is the culprit but what is it doing

120. Conclusion for this scan: ► Normally, a port scan shows up very loudly and easily ► In this case,.2 was controlling.10 via a remote trojan on port 25 (mail)..10 was passing the instructions on to.25 via port 21 (ftp)

121. ► All of this trojan traffic passes firewall rules and doen’t get a second glance. ► You would have busted.25 for the port scan, but left.10,.25 infected and the master of it all,.2, is still at large. Conclusion for this scan:

122. ► Cool sample traffic, including security issues:  http://wiki.wireshark.org/SampleCaptures#hea d- 6c6fb4051dfbe9b992057ea1533eb8dc85c9a13a http://wiki.wireshark.org/SampleCaptures#hea d- 6c6fb4051dfbe9b992057ea1533eb8dc85c9a13a http://wiki.wireshark.org/SampleCaptures#hea d- 6c6fb4051dfbe9b992057ea1533eb8dc85c9a13a ► Filters  http://wiki.wireshark.org/DisplayFilters

123. ► 10 signs that you aren't cut out for IT  1: You lack patience  2: You have no desire to continue your education  3: You refuse to work outside 9-to-5  4: You don’t like people  5: You give up quickly  6: You’re easily frustrated  7: You can’t multitask  8: You have dreams of climbing the corporate ladder  9: You hate technology  10: You turn off your phone at night ► By Jack Wallen; February 24, 2012 10 Signs you aren’t cut out for IT

124. Evalution ► I value your comments. Please fill in your evaluation form found at the end of your packet.

125. Scott Greene: Other topics available ► Computer Forensics ► Computer Forensics for Defense Attorneys ► Personal Privacy in the Information Age ► High Technology: Just where is technology going? ► Bypassing Security: How They Steal Company Data ► Fundamentals of Digital Forensics ► Technology Forensics: Theory & Potential... is it Science or Art? ► Technology Forensics: Case Examples ► Technology Forensics: Intellectual property and identity theft ► Technology Forensics: Hardware and Software tools / Show and Tell ► Portable Devices Issues and Answers: A discussion about cell phones and the stories they can tell. ► Anti-Digital Forensics. Or is it Digital Anti-Forensics? ► Data Security and Confidentiality Issues ► E-mail: The digital Smoking Gun

126. Contact Information Scott Greene, SCFE Evidence Solutions, Inc 866-795-7166Scott@EvidenceSolutions.com