1. An Out-of-the-Box Approach to High Assurance Computer System Monitoring and Integrity Protection Cyber Defense Conference, Rome, NY, May 12-14, 2008 Assistant Professor Dept. of Computer Science George Mason University Xuxian Jiang Associate Professor CERIAS and Dept. of Computer Science Purdue University Dongyan Xu

2. Outline  Motivation  “Out-of-the-box” for high assurance  New VMM component: OBSERV  New capabilities enabled  High assurance system monitoring  Stealth malware detection  External run of COTS anti-virus software  OS integrity protection against kernel rootkits  Planned work  Summary

3.  Malware remains a top concern in cyber defense  Malware: viruses, worms, rootkits, spyware, bots… Motivation

4.  Rootkit attack trend Source: McAfee Avert Lab Report (April 2006) 400% growth Q1 of 2005 700% growth Viruses, worms, bots, …

5.  State-of-the-art: Running high-assurance modules (e.g., anti-virus systems) inside the monitored system  Advantage: They can see everything (e.g., files, processes…)  Disadvantage: VirusScanFirefox IE OS Kernel … Why Going “Out-of-the-Box”? They cannot see anything!

6. Why Going “Out-of-the-Box”?  Fundamental flaw in current practice  Malware and malware defense running in the same system space at the same privileged level  No clear winner in this “arms race”  Solution: Going “out-of-the-box” Firefox IE OS Kernel … VirusScan Virtual Machine Monitor (VMM)

7. Semantic Gap The “Semantic-Gap” Challenge  What we get:  Low-level states  Memory pages, disk blocks…  Low-level events  Privileged instructions,  Interrupts, I/O…  What we want:  High-level semantic states  Files, processes…  high-level semantic events  System calls, context switches… Virtual Machine Monitor (e.g., VMware, Xen) Guest OS VirusScan

8. Our Solution: OBSERV  OBSERV: “Out-of-the-Box” with SEmantically Reconstructed View  A new component missing in current VMMs Firefox IE OS Kernel … Virtual Machine Monitor (VMM) OBSERV

9. New Capabilities Capability II: Malware detection by view comparison Capability II: Malware detection by view comparison Capability I: High-assurance system logging Capability I: High-assurance system logging Firefox IE OS Kernel … Virtual Machine Monitor (VMM) OBSERV Capability III: External run of COTS anti-virus software Capability III: External run of COTS anti-virus software OBSERV View In-the-box View Diff Capability IV: OS kernel integrity protection Capability IV: OS kernel integrity protection

10. OBSERV: Bridging the Semantic Gap  Step 1: Procuring low-level VM states and events  Disk blocks, memory pages, registers…  Traps, interrupts…  Step 2: Reconstructing high-level semantic view  Files, directories, processes, and kernel modules…  System calls, context switches… VM Introspection Guest View Casting

11. Step 1: VM Introspection Raw VMM Observations Virtual Machines (VMs) VMware Academic Program VM disk image VM hardware state (e.g., registers) VM physical memory VM-related low-level events (e.g., interrupts)

12. Step 2: Guest View Casting Virtual Machine Monitor (VMM) Guest OS Key observation: The guest OS provides all semantic “templates” of data structures and functions to reconstruct VM’s semantic view OBSERV Semantic Gap

13. Guest View Casting Raw VMM Observations Casted Guest Functions & Data Structures Reconstructed Semantic View Device drivers, file system drivers Memory translation, task_struct, mm_struct CR3, MSR_SYSENTER_CS, MSR_SYSENTER_EIP/ESP Event semantics Syscalls, context switches,.... Event-specific arguments… VM disk image VM hardware state (e.g., registers) VM physical memory VM-related low-level events (e.g., interrupts)

14. Guest View Casting on Memory State Process List Process Memory Layout

15. OBSERV Capability I Capability I: High-assurance system logging Capability I: High-assurance system logging Firefox IE OS Kernel … Virtual Machine Monitor (VMM) OBSERV X. Jiang, X. Wang, "'Out-of-the-Box' Monitoring of VM-Based High-Interaction Honeypots", International Symposium on Recent Advances in Intrusion Detection (RAID 2007)

16. OBSERV Capabilities II and III Capability II: Stealth malware detection by view comparison Capability II: Stealth malware detection by view comparison Firefox IE OS Kernel … Virtual Machine Monitor (VMM) OBSERV Capability III: External run of COTS anti- virus software Capability III: External run of COTS anti- virus software OBSERV View In-the-box View Diff X. Jiang, X. Wang, D. Xu, "Stealthy Malware Detection Through VMM-Based 'Out-of-the- Box' Semantic View Reconstruction", ACM Conference on Computer and Communications Security (CCS 2007)

17. View Comparison for Malware Detection  Experiment setup  Both guest OS and host OS run Windows XP (SP2)  VMM: VMware Server 1.0.1  Running Symantec AntiVirus twice  Inside  Outside Hacker Defender NTRootkit

18. External Scanning Result Internal Scanning Result Diff

19. OBSERV Capability IV: OS Kernel Integrity Protection  High-assurance OS kernel  No malicious kernel code  No kernel rootkit attacks  Two main tasks:  Tracking run-time kernel code layout  Enforcing the following properties  Only loading authenticated kernel code  Only executing authenticated kernel code R. Riley, X. Jiang, D. Xu, "Guest-Transparent Prevention of Kernel Rootkits with VMM-based Memory Shadowing", CERIAS Technical Report TR2001-146, Purdue University, 2008

20. OBSERV NICKLE: “ No Instruction Creeping into Kernel Level Executed” NICKLE Standard memory Kernel Code Shadow memory VMM Guest OS  Step 1: Create two memory spaces  Standard memory  Shadow memory  Step 2: Authenticate and copy kernel code to shadow memory  Step 3: Memory access dispatch  Kernel code fetch -> shadow memory  All other accesses -> standard memory Kernel Code

21. Demonstration of Effectiveness Successfully preventing 23 real-world kernel rootkits!

22. Planned Work  Porting OBSERV to hardware  FPGA, multicore, PCI card…  Research problems  Software/hardware function division  Hardware primitives/policies for high assurance  Formal verification of OBSERV capabilities  Performance optimization

23. Summary  OBSERV enables “out-of-the-box” malware defense paradigm, bringing high assurance to  System logging and monitoring  Malware detection and prevention  OS kernel (against kernel rootkits)  We are looking for  Applications in Cyber Defense activities  Collaboration/deployment/funding opportunities

24. Process Coloring: An Information Flow-Preserving Approach to Malware Investigation Eugene Spafford, Dongyan Xu, Ryan Riley Department of Computer Science and Center for Education and Research in Information Assurance and Security (CERIAS) Purdue University Xuxian Jiang Department of Computer Science George Mason University Part of NICIAR Program A related project funded by IARPA through AFRL

25. Thank you! For more information: xjiang@gmu.eduxjiang@gmu.edu, dxu@cs.purdue.edudxu@cs.purdue.edu http://www.cs.gmu.edu/~xjiang http://friends.cs.purdue.edu