1. Boris Lau, Vanja Svajcer Sophoslabs, Journal in Computer Virology, 2008

2. Outline Introduction Virtual machine detection methods Methodology of our study with DSD-Tracer Results Conclusion 2

3. Introduction #1 Virtual machine technology is first implemented by IBM More attention from virus writers & computer security researchers If in VM malware will behave like a normal program If the proportion is > 0.1% developing an environment to successfully analyze VM-aware malware is important 3

4. Introduction #2 The most common security use cases with VM Software vulnerability research Malware analysis Honeypots 4

5. Virtual machine detection methods #1 If VM is detected, the malware will stop its execution or launch a specially crafted payload Zlob Trojans IRC bots Executable packers 5

6. Virtual machine detection methods #2 Detection of running under MS virtual PC using VPC communication channel Communication between guest OS & VMM Exceptions due to opcode 0x0f, 0x3f / 0x0f, 0xc7, 0xc8 Call different VMM services 0x07, 0x0B 6

7. Invalid instruction VPC communication channel detection 7

8. Virtual machine detection methods #3 Detection of running under VMware using VMWare control API VMWare backdoor communication guest host communication IN instruction port 0x5658 eax 0x564D5868 VMXh ebx function number 8

9. 9

10. Anti-VMWare prevention virtual machine initialization settings 10

11. Virtual machine detection methods #4 Redpill using SIDT, SGDT or SLDT SxxT x86 instruction Return the contests of the sensitive register IDT in VMWare is 0xffXXXXXX IDT in Virtual PC is 0xe8XXXXXX Compare with 0xd0 Invalid in multi processor system 11

12. Redpill 12

13. Virtual machine detection methods #5 SMSW VMWare detection Store Machine Specific Word instruction Return 16-bit result 32 bits register 16-bit undefined + 16-bit result In VMWare, the top 16-bits doesnt change 13

14. SMSW VMWare detection code 14

15. Methodology of our study with DSD-Tracer #1 DSD-Tracer identify obfuscation packers dynamic & static analysis 15

16. Methodology of our study with DSD-Tracer #2 16

17. Methodology of our study with DSD-Tracer #3 Dynamic component Instructions decoded before its execution All CPU registers Reads / writes to virtual / physical memory Interrupts / exceptions generated Instrumented virtual machine Low-level information 17

18. Methodology of our study with DSD-Tracer #4 Static component C++ interface Python Script Match known techniques for detecting VM Automatic replication harness Web-based automatic replication harness 18

19. Methodology of our study with DSD-Tracer #5 Case study DSD-Tracer on Themida Analyzing Themida by traditional debugger/static technique is troublesome recording memory-io dump sample in static environment 19

20. Methodology of our study with DSD-Tracer #6 Justification for using DSD-Tracer Coverage of packed samples Low-level accuracy Circumventing armour techniques Mitigating factors in using DSD-Tracer No Bochs detect techniques in any sample 4 samples/hour, 5 samples from each set of packed file 85% of Themida samples with VM-aware techniques 20

21. Methodology of our study with DSD-Tracer #7 Proof of concept experiment for DSD-Tracer on VMware Cross-verified multiple dynamic analysis Implemented on VMware Workstation 6 Invisible breakpoint GDB script for printing the assembly execution trace in user mode 21

22. Results #1 VM detection in packers 193 different packers, 400 packed samples Overall VM detection rate is 1.15% Themida accounting for 1.03% ExeCryptor accounting for 0.15% EncPk custom packers 22

23. Results #2 VM detection in malware families Static analysis rules – disassembly Dynamic analysis rules – Sophos virus engine emulation 2 million known malicious files A large set of knows clean files VM-aware samples < 1% Method breakdown Table 1. Family breakdown Table 2. Dial/FlashL 23

24. Results #3 24

25. Results #4 VMWare backdoor detection method 50% VPC illegal instruction detection method VPC illegal instruction detection method 93% VMWare backdoor detection method 25

26. Results #5 Fig. 7 VMWare backdoor detection in 2007 26

27. Results #6 Fig. 8 VPC backdoor detections in 2007 27

28. Conclusion Combination of dynamic and static analysis is better 2.13% VM-aware samples 28

29. Q & A 29

30. Appendix VMWare backdoor I/O port On the Cutting Edge:Thwarting Virtual MachineDetection Trapping worm in a virtual net Trapping worm in a virtual net VM Virtual PC Bochs http://hi.baidu.com/%CC%FA%D0%AC%B9%C3%C4%E F/blog/item/085cc609b215f3226b60fba5.html http://hi.baidu.com/%CC%FA%D0%AC%B9%C3%C4%E F/blog/item/085cc609b215f3226b60fba5.html http://www.osnews.com/story/1054 30

31. Thanks ~ 31