Boris Lau, Vanja Svajcer Sophoslabs, Journal in Computer Virology, 2008

Outline Introduction Virtual machine detection methods Methodology of our study with DSD-Tracer Results Conclusion 2

Introduction #1 Virtual machine technology is first implemented by IBM More attention from virus writers & computer security researchers If in VM malware will behave like a normal program If the proportion is > 0.1% developing an environment to successfully analyze VM-aware malware is important 3

Introduction #2 The most common security use cases with VM Software vulnerability research Malware analysis Honeypots 4

Virtual machine detection methods #1 If VM is detected, the malware will stop its execution or launch a specially crafted payload Zlob Trojans IRC bots Executable packers 5

Virtual machine detection methods #2 Detection of running under MS virtual PC using VPC communication channel Communication between guest OS & VMM Exceptions due to opcode 0x0f, 0x3f / 0x0f, 0xc7, 0xc8 Call different VMM services 0x07, 0x0B 6

Invalid instruction VPC communication channel detection 7

Virtual machine detection methods #3 Detection of running under VMware using VMWare control API VMWare backdoor communication guest host communication IN instruction port 0x5658 eax 0x564D5868 VMXh ebx function number 8

9

Anti-VMWare prevention virtual machine initialization settings 10

Virtual machine detection methods #4 Redpill using SIDT, SGDT or SLDT SxxT x86 instruction Return the contests of the sensitive register IDT in VMWare is 0xffXXXXXX IDT in Virtual PC is 0xe8XXXXXX Compare with 0xd0 Invalid in multi processor system 11

Redpill 12

Virtual machine detection methods #5 SMSW VMWare detection Store Machine Specific Word instruction Return 16-bit result 32 bits register 16-bit undefined + 16-bit result In VMWare, the top 16-bits doesnt change 13

SMSW VMWare detection code 14

Methodology of our study with DSD-Tracer #1 DSD-Tracer identify obfuscation packers dynamic & static analysis 15

Methodology of our study with DSD-Tracer #2 16

Methodology of our study with DSD-Tracer #3 Dynamic component Instructions decoded before its execution All CPU registers Reads / writes to virtual / physical memory Interrupts / exceptions generated Instrumented virtual machine Low-level information 17

Methodology of our study with DSD-Tracer #4 Static component C++ interface Python Script Match known techniques for detecting VM Automatic replication harness Web-based automatic replication harness 18

Methodology of our study with DSD-Tracer #5 Case study DSD-Tracer on Themida Analyzing Themida by traditional debugger/static technique is troublesome recording memory-io dump sample in static environment 19

Methodology of our study with DSD-Tracer #6 Justification for using DSD-Tracer Coverage of packed samples Low-level accuracy Circumventing armour techniques Mitigating factors in using DSD-Tracer No Bochs detect techniques in any sample 4 samples/hour, 5 samples from each set of packed file 85% of Themida samples with VM-aware techniques 20

Methodology of our study with DSD-Tracer #7 Proof of concept experiment for DSD-Tracer on VMware Cross-verified multiple dynamic analysis Implemented on VMware Workstation 6 Invisible breakpoint GDB script for printing the assembly execution trace in user mode 21

Results #1 VM detection in packers 193 different packers, 400 packed samples Overall VM detection rate is 1.15% Themida accounting for 1.03% ExeCryptor accounting for 0.15% EncPk custom packers 22

Results #2 VM detection in malware families Static analysis rules – disassembly Dynamic analysis rules – Sophos virus engine emulation 2 million known malicious files A large set of knows clean files VM-aware samples < 1% Method breakdown Table 1. Family breakdown Table 2. Dial/FlashL 23

Results #3 24

Results #4 VMWare backdoor detection method 50% VPC illegal instruction detection method VPC illegal instruction detection method 93% VMWare backdoor detection method 25

Results #5 Fig. 7 VMWare backdoor detection in

Results #6 Fig. 8 VPC backdoor detections in

Conclusion Combination of dynamic and static analysis is better 2.13% VM-aware samples 28

Q & A 29

Appendix VMWare backdoor I/O port On the Cutting Edge:Thwarting Virtual MachineDetection Trapping worm in a virtual net Trapping worm in a virtual net VM Virtual PC Bochs F/blog/item/085cc609b215f3226b60fba5.html F/blog/item/085cc609b215f3226b60fba5.html 30

Thanks ~ 31