1. In-Band Detection of Virtual Machines Estefan Ortiz & Cory Hayes Computer Science and Engineering Graduate Operating Systems December 16, 2011 1

2. Introduction  Malicious programs (malware) need to know if they are in a virtual environment so they can modify their behavior and avoid detection  Related work  Red Pill Tests: Examine byte-level behavior of instructions for physical and emulated CPUs. If any disagreements in output, create one or more “red pills” that can avoid detection  SubVirt: Virtual machine-based rootkit installed underneath host OS that runs OS as a guest to remain nearly undetectable 2

3. Our Approach  Similar to Red Pill and SubVirt, but client-server based  Idea: Instead of monitoring system call discrepancies, analyze network data sent to/from physical and virtual machines  Goal: Determine if there are sufficient differences in network traffic to detect if a client/server is being run on a virtual machine 3

4. Goal 4 Client Native TCP/IP PacketClient Virtual Machine TCP/IP Packet Byte 0 Byte n Byte k 1 Byte k 2 Difference Found

5. General Setup 5

6. Actual Setup 6 Network output saved for analysis Functions as the “MITM”

7. Experiment Setup  Using Wireshark, capture and compare the raw info of TCP/IP packets sent back and forth between a client and a physical/virtual server running Apache  Bits 1-160: IP  Remainder: TCP  Virtual machine OS matches the OS of the host (Ubuntu- Ubuntu, Vista-Vista)  Use a small set of Matlab commands to send regular and malformed packets  Dynex 5-port 10/100/1000 Gigabit Ethernet Switch 7

8. Sample Captured Wireshark Output 8 8 th Packet sent between Client & VM running Apache 8 th Packet sent between Client & Host running Apache VM Client Host

9. Metrics 9 Bit Difference Comparison: Fractional Hamming distance between two packets

10. Metrics (cont.) 10 Round trip time: Time from SYN request sent by client to received ACK from server

11. Metrics (cont.)* 11 Pairwise Packet Length Comparison: Number of concurrent packet pairs that differ in length

12. Experiment #1  Client: Windows Vista (4GB RAM, 2.6GHz)  Server: Ubuntu 11.04 32-bit w/ Apache Web Server 2.2  Server: Host OS Ubuntu: VirtualBox w/ Ubuntu running Apache  On isolated switch network (no other traffic) 12

13. Exp. #1: Frac. Hamming Distance 13

14. Exp. #1: Round-trip Timing 14

15. Example: Packet #9 15 These bits correspond to the header length & flags in the TCP header

16. Experiment #2  Client: Mac (4GB RAM, 2.4GHz, MacOSX 10.6.8)  Server: Windows Vista 32-bit w/ Apache Web Server 2.2  Server: Host OS Windows Vista: VirtualBox w/ Windows Vista running Apache  On isolated switch network (no other traffic) 16

17. Exp. #2: Frac. Hamming Distance 17

18. Exp. #2: Round-trip Timing 18

19. Example: Packet #4 19 Destination Address in IP header Flags in TCP header

20. Experiment #3  Client: Windows Vista (4GB RAM, 2.6GHz)  Server: Ubuntu 11.04 32-bit w/ Apache Web Server 2.2  Server: Host OS Ubuntu: VirtualBox w/ Ubuntu running Apache  Both client and server on CVRL subnet (at ~3:00 am) 20

21. Exp. #3: Frac. Hamming Distance 21

22. Exp. #3: Round-trip Timing 22

23. Example: Packet #3 23 Destination Address in IP header

24. Experiment #4 24 ND/CVRL subnet

25. Experiment #4  Client: Windows Vista (4GB RAM, 2.6GHz)  Server: Ubuntu 11.04 32-bit w/ Apache Web Server 2.2  Server: Host OS Ubuntu: VirtualBox w/ Ubuntu running Apache  Could not monitor packet information; only ping tests  Varied number of bytes sent using ping  Performed 100 per fixed byte amount  Calculated avg. & std. dev  Executed at ~3:30 am 25

26. Exp. #4: Ping Timing 26

27. Conclusion  Examined packet information from a high level (packet- length) down to specific bit difference comparisons  Packet length provided no insight  Timing tests didn’t provide conclusive evidence of a connection to a virtual machine  Fractional hamming dist. provided first level of insight  Further analysis of differences at the bit level provided clues where to look for VM traces 27

28. Future Direction  Experiments 1-3 were conducted under somewhat “ideal” scenarios  More realistic approach would be packet analysis on multi-hop connections with knowledge of which sections of the TCP/IP packets to monitor 28