1 Introduction to Honeypot, Denial-of- Service, and Rootkit Cliff C. Zou CAP6135 Spring, 2010

2 What Is a Honeypot? Abstract definition: “A honeypot is an information system resource whose value lies in unauthorized or illicit use of that resource.” (Lance Spitzner) Concrete definition: “A honeypot is a faked vulnerable system used for the purpose of being attacked, probed, exploited and compromised.”

3 Example of a Simple Honeypot Install vulnerable OS and software on a machine Install monitor or IDS software Connect to the Internet (with global IP) Wait & monitor being scanned, attacked, compromised Finish analysis, clean the machine

4 Benefit of Deploying Honeypots Risk mitigation:  Lure an attacker away from the real production systems (“easy target“). IDS-like functionality:  Since no legitimate traffic should take place to or from the honeypot, any traffic appearing is evil and can initiate further actions.

5 Benefit of Deploying Honeypots Attack analysis:  Find out reasons, and strategies why and how you are attacked.  Binary and behavior analysis of capture malicious code Evidence:  Once the attacker is identified, all data captured may be used in a legal procedure. Increased knowledge

6 Honeypot Classification High-interaction honeypots  A full and working OS is provided for being attacked  VMware virtual environment  Several VMware virtual hosts in one physical machine Low-interaction honeypots  Only emulate specific network services  No real interaction or OS  Honeyd Honeynet/honeyfarm  A network of honeypots

7 Low-Interaction Honeypots Pros:  Easy to install (simple program)  No risk (no vulnerable software to be attacked)  One machine supports hundreds of honeypots, covers hundreds of IP addresses Cons:  No real interaction to be captured  Limited logging/monitor function  Hard to detect unknown attacks; hard to generate filters  Easily detectable by attackers

8 High-Interaction Honeypots Pros:  Real OS, capture all attack traffic/actions  Can discover unknown attacks/vulnerabilites  Can capture and anlayze code behavior Cons:  Time-consuming to build/maintain  Time-consuming to analysis attack  Risk of being used as stepping stone  High computer resource requirement

9 Honeynet A network of honeypots High-interaction honeynet  A distributed network composing many honeypots Low-interaction honeynet  Emulate a virtual network in one physical machine  Example: honeyd Mixed honeynet  “Scalability, Fidelity and Containment in the Potemkin Virtual Honeyfarm”, presented next week Reference: honeypot-forensics-slides.ppthttp:// honeypot-forensics-slides.ppt

10 Honeypot-Aware Botnet [Zou’07] Honeypot is widely used by defenders  Ability to detect unknown attacks  Ability to monitor attacker actions (e.g., botnet C&C) Botnet attackers will adapt to honeypot defense  When they feel the real threat from honeypot  We need to think one step ahead

11 Honeypot Detection Principles Hardware/software specific honeypot detection  Detect virtual environment via specific code  E.g., time response, memory address  Detect faculty honeypot program  Case by case detection Detection based on fundamental difference  Honeypot defenders are liable for attacks sending out  Liability law will become mature  It’s a moral issue as well  Real attackers bear no liability  Check whether a bot can send out malicious traffic or not

12 Detection of Honeypot Bot Infection traffic  Real liability to defenders  No exposure issue: a bot needs to do this regardless Other honeypot detection traffic  Port scanning, spam, web request (DoS?) C&C bot Sensor (secret) 1 malicious traffic 2 Inform bot’s IP 3 Authorize

13 Two-stage Reconnaissance to Detect Honeypot in Constructing P2P Botnets Fully distributed  No central sensor is used  Could be fooled by double-honeypot  Counterattack is presented in our paper Lightweighted spearhead code  Infect + honeypot detection  Speedup UDP-based infection Host A spearhead Host B request main-force spearhead Host C 1 3 2

14 Defense against Honeypot-Aware Attacks Permit dedicated honeypot detection systems to send out malicious traffic  Need law and strict policy Redirect outgoing traffic to a second honeypot  Not effective for sensor-based honeypot detection Figure out what outgoing traffic is for honeypot detection, and then allow it  It could be very hard Neverthless, honeypot is still a valuable monitoring and detection/defense tool

15 Distributed Denial of Service (DDoS) Attack Send large amount of traffic to a server so that the server has no resource to serve normal users Attacking format:  Consume target memory/CPU resource  SYN flood (backscatter paper presented before)  Database query…  Congest target Internet connection  Many sources attack traffic overwhelm target link  Very hard to defend

16 Why hard to defined DDoS attack? Internet IP protocol has no built-in security  No authentication of source IP  SYN flood with faked source IP  However, IP is true after connection is setup Servers are supposed to accept unsolicited service requests Lack of collaboration ways among Internet community  How can you ask an ISP in another country to block certain traffic for you?

17 DDoS Defenses Increase servers capacity  Cluster of machine, Multi-CPUs, larger Internet access Use Internet web caching service  E.g., Akamai Defense Methods (many in research stage)  SYN cookies (  SOS  IP traceback

18 SYN Cookies SYN flood attack  Fill up server’s SYN queue  Property: attacker does not respond to SYN/ACK from victim. Defense  Fact: normal client responds to SYN/ACK  Remove initial SYN queue  Server encode info in TCP seq. number  Use it to reconstruct the initial SYN

19 DoS spoofed attack defense: IP traceback Suppose a victim can call ISPs upstream to block certain traffic SYN flood: which traffic to block? IP traceback:  Find out the real attacking host for SYN flood  Based on large amount of attacking packets  Need a little help from routers (packet marking)

20 SOS: Secure Overlay Service Central Idea:  Use many TCP connection respondent machines  Only setup connections relay to server  Identity of server is secrete

The Evolution of Malware Malware, including spyware, adware and viruses want to be hard to detect and/or hard to remove Rootkits are a fast evolving technology to achieve these goals  Cloaking technology applied to malware  Not malware by itself  Example rootkit-based viruses: Rootkit history  Appeared as stealth viruses  One of the first known PC viruses, Brain, was stealth  First “rootkit” appeared on SunOS in 1994  Replacement of core system utilities (ls, ps, etc.) to hide malware processes

Cloaking Modern rootkits can cloak:  Processes  Services  TCP/IP ports  Files  Registry keys  User accounts Several major rootkit technologies  User-mode API filtering  Kernel-mode API filtering  Kernel-mode data structure manipulation  Process hijacking Visit for tools and informationwww.rootkit.com

Attack user-mode system query APIs Con: can be bypassed by going directly to kernel- mode APIs Pro: can infect unprivileged user accounts Examples: HackerDefender, Afx Taskmgr.exe Taskmgr.exe Ntdll.dll Ntdll.dll Explorer.exe, Malware.exe, Winlogon.exe Explorer.exe, Malware.exe, Winlogon.exe Rootkit Rootkit Explorer.exe, Winlogon.exe Explorer.exe, Winlogon.exe user mode user mode kernel mode kernel mode User-Mode API Filtering

Attack kernel-mode system query APIs Cons:  Requires admin privilege to install  Difficult to write Pro: very thorough cloak Example: NT Rootkit Taskmgr.exe Taskmgr.exe Ntdll.dll Ntdll.dll user mode user mode kernel mode kernel mode Rootkit Rootkit Explorer.exe, Winlogon.exe Explorer.exe, Winlogon.exe Explorer.exe, Malware.exe, Winlogon.exe Explorer.exe, Malware.exe, Winlogon.exe Kernel-Mode API Filtering

Also called Direct Kernel Object Manipulation Attacks active process data structure  Query API doesn’t see the process  Kernel still schedules process’ threads Cons:  Requires admin privilege to install  Can cause crashes  Detection already developed Pro: more advanced variations possible Example: FU Explorer.exe Explorer.exe Malware.exe Malware.exe Winlogon.exe Winlogon.exe Active Processes Active Processes Kernel-Mode Data Structure Manipulation

Hide inside a legitimate process Con: doesn’t survive reboot Pro: extremely hard to detect Example: Code Red Explorer.exe Explorer.exe Malware Malware Process Hijacking

Detecting Rootkits All cloaks have holes  Leave some APIs unfiltered  Have detectable side effects  Can’t cloak when OS is offline Rootkit detection attacks holes  Cat-and-mouse game  Several examples  Microsoft Research Strider/Ghostbuster  RKDetect  Sysinternals RootkitRevealer  F-Secure BlackLight

Perform a directory listing online and compare with secure alternate OS boot (see )  Offline OS is Windows PE, ERD Commander, BartPE dir /s /ah * > dirscan.txt windiff dirscanon.txt dirscanoff.txt This won’t detect non-persistent rootkits that save to disk during shutdown Simple Rootkit Detection

RootkitRevealer RootkitRevealer Rootkit Rootkit Windows API Windows API Raw file system, Raw Registry hive Raw file system, Raw Registry hive Filtered Windows API Filtered Windows API omits malware files and keys omits malware files and keys Malware files and keys are visible in raw scan Malware files and keys are visible in raw scan RootkitRevealer RootkitRevealer (RKR) runs online RKR tries to bypass rootkit to uncover cloaked objects  All detectors listed do the same  RKR scans HKLM\Software, HKLM\System and the file system  Performs Windows API scan and compares with raw data structure scan

Demo HackerDefender  HackerDefender before and after view of file system  Detecting HackerDefender with RootkitRevealer

RootkitRevealer Limitations Rootkits have already attacked RKR directly by not cloaking when scanned  RKR is given true system view  Windows API scan looks like raw scan SysInternals have modified RKR to be a harder to detect by rootkits  RKR is adopting rootkit techniques itself  Rootkit authors will continue to find ways around RKR’s cloak  It’s a game nobody can win

Unless you have specific uninstall instructions from an authoritative source: Don’t rely on “rename” functionality offered by some rootkit detectors  It might not have detected all a rootkit’s components  The rename might not be effective Reformat the system and reinstall Windows! Reformat the system and reinstall Windows! Dealing with Rootkits