Intrusion Detection CSC 482/582: Computer Security

Topics 1. Principles 2. Models of Intrusion Detection 3. False Positives 4. Architecture of an IDS 5. IDS Deployment 6. Active Response (IPS) 7. Host-based IDS and IPS 8. IDS Evasion Techniques

CSC 482/582: Computer Security Principles of Intrusion Detection Characteristics of systems not under attack 1. User, process actions conform to statistically predictable pattern. 2. User, process actions do not include sequences of actions that subvert the security policy. 3. Process actions correspond to a set of specifications describing what the processes are allowed to do. Systems under attack do not meet at least one.

CSC 482/582: Computer Security Example Goal: insert a back door into a system Intruder will modify system configuration file or program. Requires privilege; attacker enters system as an unprivileged user and must acquire privilege. Nonprivileged user may not normally acquire privilege (violates #1). Attacker may break in using sequence of commands that violate security policy (violates #2). Attacker may cause program to act in ways that violate program’s specification (violates #3).

CSC 482/582: Computer Security Goals of IDS 1. Detect wide variety of intrusions Previously known and unknown attacks. Need to adapt to new attacks or changes in behavior. 2. Detect intrusions in timely fashion May need to be be real-time, especially when system responds to intrusion. Problem: analyzing commands may impact response time of system. May suffice to report intrusion occurred a few minutes or hours ago.

CSC 482/582: Computer Security Goals of IDS 3. Present analysis in easy-to-understand format. Ideally a binary indicator. Usually more complex, allowing analyst to examine suspected attack. User interface critical, especially when monitoring many systems. 4. Be accurate Minimize false positives, false negatives. Minimize time spent verifying attacks, looking for them.

CSC 482/582: Computer Security Deep Packet Inspection IDS requires, some firewalls do too. DPI = Analysis of Application Layer data Protocol Standard Compliance Is port 53 traffic DNS or a covert shell session? Is port 80 traffic HTTP or tunneled IM or P2P? Protocol Anomaly Detection Traffic is valid HTTP. But suspicious URL contains directory traversal.

CSC 482/582: Computer Security Models of Intrusion Detection 1. Anomaly detection What is usual, is known. What is unusual, is bad. 2. Misuse detection What is bad is known. Look for what is bad, hope it doesn’t change.

CSC 482/582: Computer Security Anomaly Detection Analyzes a set of characteristics of system, and compares their values with expected values; report when computed statistics do not match expected statistics. Threshold metrics Sequences of valid actions Statistical measures

CSC 482/582: Computer Security Threshold Metrics Counts number of events that occur Between m and n events (inclusive) expected If number falls outside this range, anomalous. Example Windows: lock user out after k failed sequential login attempts. Range is (0, k–1). k or more failed logins deemed anomalous Threshold depends on typing skill.

CSC 482/582: Computer Security Sequences of System Calls Define normal behavior in terms of sequences of system calls. Example normal trace: open read write open write close Doesn’t normally run other programs. Attack trace: open read write open exec write close

CSC 482/582: Computer Security Bayesian Filtering Calculate Probability that a word appears in spam. using training data Set of spam . Set of non-spam . For new message Combine probabilities of each word to calculate probability that message is spam. If probability > 0.9, then message is spam. Tune cutoff to adjust false positive/negative rate.

CSC 482/582: Computer Security Misuse Detection Determines whether a sequence of instructions being executed is known to violate the site security policy. Descriptions of known or potential exploits grouped into rule sets. IDS matches data against rule sets; on match, potential attack found. Cannot detect new attacks: No rules to cover them.

CSC 482/582: Computer Security Example: snort Network Intrusion Detection System Sniffs packets off wire. Checks packets for matches against rule sets. Logs detected signs of misuse. Alerts adminstrator when misuse detected.

CSC 482/582: Computer Security Snort Rules Rule Header Action: pass, log, alert Network Protocol Source Address (Host or Network) + Port Destination Address (Host or Network) + Port Rule Body Content: packet ASCII or binary content TCP/IP flags and options to match Message to log, indicating nature of misuse detected

CSC 482/582: Computer Security Snort Rule Example Example: rule for ssh shell code exploit alert tcp $EXTERNAL_NET any -> $HOME_NET 22 (msg:"EXPLOIT ssh CRC32 overflow NOOP"; flow:to_server,established; content:"| |"; reference:bugtraq,2347; reference:cve,CVE ; classtype:shellcode-detect; sid:1326; rev:3;)

CSC 482/582: Computer Security Comparison and Contrast Misuse detection: if all policy rules known, easy to construct rulesets to detect violations. Usual case is that much of policy is unspecified, so rulesets describe attacks, and are not complete. Anomaly detection: detects unusual events, but these are not necessarily security problems.

CSC 482/582: Computer Security False Positives A new test for a disease that is 95% accurate Assume 1 in 1000 people have disease. Should everyone get the test? Sample size: 1000 Expect (999 * 0.05) positives Ergo, 50 people will be told they have disease If you test positive, only 2% chance you have it.

CSC 482/582: Computer Security IDS Architecture An IDS is essentially a sophisticated audit system Agent gathers data for analysis. Director analyzes data obtained from the agents according to its internal rules. Notifier acts on director results. May simply notify security officer. May reconfigure agents, director to alter collection, analysis methods. May activate response mechanism.

CSC 482/582: Computer Security Agents Obtain information and sends to director. Preprocessing Simplifying and reformatting of data. Push vs Pull Agents may push data to Director, or Director may pull data from Agents.

CSC 482/582: Computer Security Host-Based Agents 1. Obtain information from logs May use many logs as sources. May be security-related or not. May use virtual logs if agent is part of the kernel. 2. Agent generates its information Analyzes state of system. Treats results of analysis as log data.

CSC 482/582: Computer Security Network-Based Agents Sniff traffic from network. Use hubs, SPAN ports, or taps to see traffic. Need agents on all switches to see entire network. Agent needs same view of traffic as destination TTL tricks, fragmentation may obscure this. End-to-end encryption defeats content monitoring Not traffic analysis, though.

CSC 482/582: Computer Security Aggregation of Information Agents produce information at multiple layers of abstraction. Application-monitoring agents provide one view of an event. System-monitoring agents provide a different view of an event. Network-monitoring agents provide yet another view (involving many packets) of an event.

CSC 482/582: Computer Security Director Reduces information from agents Eliminates unnecessary, redundant records. Analyzes information to detect attacks Analysis engine can use any of the modelling techniques. Usually run on separate system Does not impact performance of monitored systems. Rules, profiles not available to ordinary users.

CSC 482/582: Computer Security Example Jane logs in to perform system maintenance during the day. She logs in at night to write reports. One night she begins recompiling the kernel. Agent #1 reports logins and logouts. Agent #2 reports commands executed. Neither agent spots discrepancy. Director correlates log, spots it at once.

CSC 482/582: Computer Security Adaptive Directors Modify profiles, rulesets to adapt their analysis to changes in system Usually use machine learning or planning to determine how to do this. Example: use neural nets to analyze logs Network adapted to users’ behavior over time. Used learning techniques to improve classification of events as anomalous. Reduced number of false alarms.

CSC 482/582: Computer Security Notifier Accepts information from director Takes appropriate action Notify system security officer Respond to attack Often GUIs Use visualization to convey information.

CSC 482/582: Computer Security Example Architecture: snort

CSC 482/582: Computer Security IDS Deployment IDS deployment should reflect your threat model. Major classes of attackers: 1. External attackers intruding from Internet. 2. Internal attackers intruding from your LANs. Where should you place IDS systems? 1. Perimeter (outside firewall) 2. DMZ 3. Intranet 4. Wireless

CSC 482/582: Computer Security IDS Deployment

CSC 482/582: Computer Security Sguil NSM Console

CSC 482/582: Computer Security Intrusion Prevention Systems What else can you do with IDS alerts? Identify attack before it completes. Prevent it from completing. How to prevent attacks? Directly: IPS drops attack packets. Indirectly: IPS modifies firewall rules. Is IPS a good idea? How do you deal with false positives?

CSC 482/582: Computer Security IPS Deployment Types Inline IPS Intranet Non-Inline Intranet IPS

CSC 482/582: Computer Security Active Responses by Network Layer Data Link: Shut down a switch port. Only useful for local intrusions. Rate limit switch ports. Network: Block a particular IP address. Inline: can perform blocking itself. Non-inline: send request to firewall. Transport: Send TCP RST or ICMP messages to sender and target to tear down TCP sessions. Application: Inline IPS can modify application data to be harmless: /bin/sh -> /ben/sh

CSC 482/582: Computer Security Host IDS and IPS Anti-virus and anti-spyware AVG anti-virus, SpyBot S&D Log monitors swatch, logwatch Integrity checkers tripwire, osiris, samhain Monitor file checksums, etc. Application shims mod_security

CSC 482/582: Computer Security Evading IDS and IPS Alter appearance to prevent sig match URL encode parameters to avoid match. Use ‘ or 783>412-- for SQL injection. Alter context Change TTL so IDS sees different packets than target hosts receives. Fragment packets so that IDS and target host reassemble the packets differently.

CSC 482/582: Computer Security Fragment Evasion Techniques Use fragments Older IDS cannot handle reassembly. Flood of fragments DoS via heavy use of CPU/RAM on IDS. Tiny fragment Break attack into multiple fragments, none of which match signature. ex: frag 1:“cat /etc”, frag 2: “/shadow” Overlapping fragments Offset of later fragments overwrites earlier fragments. ex: frag 1: “cat /etc/fred”, frag 2: offset=10, “shadow” Different OSes deal differently with overlapping.

CSC 482/582: Computer Security Web Evasion Techniques URL encoding GET /%63%67%69%2d%62%69%6e/bad.cgi /./ directory insertion GET /./cgi-bin/./bad.cgi Long directory insertion GET /junklongdirectorypathstuffhereuseless/../cgi-bin/bad.cgi IDS may only read first part of URL for speed. Tab separation GET /cgi-bin/bad.cgi Tabs usually work on servers, but may not be in sig. Case sensitivity GET /CGI-BIN/bad.cgi Windows filenames are case insensitive, but signature may not be.

CSC 482/582: Computer Security Countering Evasion Keep IDS/IPS signatures up to date. On daily or weekly basis. Use both host and network IDS/IPS. Host-based harder to evade as runs on host. Fragment attacks can’t evade host IDS. Network IDS still useful as overall monitor. Like any alarm, IDS/IPS has False positives False negatives

CSC 482/582: Computer Security Key Points 1. Models of IDS: 1. Anomaly detection: unexpected events. 2. Misuse detection: violations of policy. 2. IDS Architecture: 1. Agents. 2. Director. 3. Notifiers. 3. Types of IDS 1. Host: agent on host checks files, procs to detect attacks. 2. Network: sniffs and analyzes packets to detect intrusions. 4. IDS/IPS Evasion 1. Alter appearance to avoid signature match. 2. Alter context to so IDS interprets differently than host.

CSC 482/582: Computer Security References 1. Richard Bejtlich, The Tao of Network Security Monitoring, Addison-Wesley, Matt Bishop, Computer Security: Art and Science, Addison-Wesley, Brian Caswell, et. al., Snort 2.0 Intrusion Detection, Snygress, William Cheswick, Steven Bellovin, and Avriel Rubin, Firewalls and Internet Security, 2 nd edition, The Honeynet Project, Know Your Enemy, 2 nd edition, Addison-Wesley, Richard A. Kemmerer and Giovanni Vigna, “Intrusion Detection: A Brief History and Overview,” IEEE Security & Privacy, v1 n1, Apr 2002, pp Steven Northcutt and Julie Novak, Network Intrusion Detection, 3 rd edition, New Riders, Michael Rash et. al., Intrusion Prevention and Active Response, Syngress, Rafiq Rehman, Intrusion Detection Systems with Snort: Advanced IDS Techniques Using Snort, Apache, MySQL, PHP, and ACID, Prentice Hall, Ed Skoudis, Counter Hack Reloaded 2/e, Prentice Hall, Ed Skoudis and Lenny Zeltser, Malware: Fighting Malicious Code, Prentice Hall, 2003.