Anti-Unpacking techniques Peter Ferrie Senior Anti-virus Researcher 1 May, 2008 1

Classes of unpackers Memory dumper Debugger Emulator Code-buffer W-X interceptor Peter Ferrie, Microsoft Corporation 2

Anti-dumping SizeOfImage Field in PEB When altered, prevents debugger from attaching Causes incorrect number of pages to be dumped Erasing the header Removes section information Not obvious how many pages should be dumped Used by Yoda’s Crypter Nanomites Introduced in Armadillo Replace branches with int 3 and encoded pointers Peter Ferrie, Microsoft Corporation 3

Anti-dumping (cont.) Stolen bytes Introduced in ASProtect Copy bytes elsewhere and execute from there Optionally destroy original bytes Guard pages Allows on-demand decryption or decompression Used by Armadillo (decryption) Used by Shrinker (decompression) Imports Table is often removed, encrypted or compressed Sometimes erased after first use Functions can be redirected to another location Peter Ferrie, Microsoft Corporation 4

Anti-dumping (cont.) Virtual machines Uses p-code instead of native code P-code can be polymorphic (Themida) P-code can contain anti-debugging (HyperUnpackMe2) Interpreter can be obfuscated (Virtual CPU) Peter Ferrie, Microsoft Corporation 5

Anti-debugging NtGlobalFlag Field in PEB Debugger typically causes certain flags to be set Very common method Very commonly incorrect (cmp instead of test) Very unreliable (several ways to set/clear flags) Used by ExeCryptor Heap flags Pointer is field in PEB Two sets of flags Also very unreliable (flags controllable like NtGlobalFlag) Peter Ferrie, Microsoft Corporation 6

Anti-debugging (cont.) Heap Tail markers appear if certain flag is set Debugger typically causes flag to be set Marker is constant and easy to find Used by Themida IsDebuggerPresent() Introduced in Windows 95 Reads field in PEB One of the most common methods Returns true if a debugger is present Peter Ferrie, Microsoft Corporation 7

Anti-debugging (cont.) CheckRemoteDebuggerPresent() Introduced in Windows XP SP1 Wrapper around NtQueryInformationProcess (ProcessDebugPort) Returns true if a debugger is present NtQueryInformationProcess() Can be called with ProcessDebugPort class Returns EPROCESS->DebugPort value Non-zero if a debugger is present Used by MSLRH Peter Ferrie, Microsoft Corporation 8

Anti-debugging (cont.) Debug objects Another use for NtQueryInformationProcess() Introduced in Windows XP Can be called with ProcessDebugObjectHandle class Returns non-zero if a debugger is present Used by HyperUnpackMe2 More debug objects Yet another use for NtQueryInformationProcess() Also introduced in Windows XP Can be called with ProcessDebugFlags class Returns inverse of EPROCESS->NoDebugInherit bit Returns zero if a debugger is present Peter Ferrie, Microsoft Corporation 9

Anti-debugging (cont.) Kernel debugger Final use for NtQueryInformationProcess() Can be called with SystemKernelDebuggerInformation class Not supported by any version of Windows(?) Returns non-zero in AH register if a debugger is present Used by SafeDisc NtQueryObject() Supported since Windows NT Extended in Windows XP to include debug object Request all objects Search for "DebugObject" Check associated handle count Non-zero if a debugger is present Peter Ferrie, Microsoft Corporation 10

Anti-debugging (cont.) Thread hiding NtSetInformationThread() method Introduced in Windows 2000 Can be called with HideThreadFromDebugger class Debugger receives no further events for that thread Used by HyperUnpackMe2 OpenProcess() Csrss.exe process can be opened If debug privilege is obtained first Even though it's a system process Process can be opened with full access rights Allows thread injection and system-wide DoS Peter Ferrie, Microsoft Corporation 11

Anti-debugging (cont.) CloseHandle() Different behaviour if debugger is present Kernel raises EXCEPTION_INVALID_HANDLE For invalid handle if debugger is present Exception can be intercepted by debugged process OutputDebugString() Kernel clears thread error code If debugger is present ReadFile() Indirect breakpoint removal Read data from file over location of breakpoint Peter Ferrie, Microsoft Corporation 12

Anti-debugging (cont.) WriteProcessMemory() Alternative breakpoint removal Write data from memory over location of breakpoint Used by NsAnti UnhandledExceptionFilter() Called as last resort if no other handlers succeed Not called if a debugger is present Exception is passed to debugger instead BlockInput() Blocks mouse and keyboard events Used by Yoda's Protector Peter Ferrie, Microsoft Corporation 13

Anti-debugging (cont.) SuspendThread() Can be used on remote processes Such as debuggers Used by Yoda's Protector Guard pages Certain debuggers might not support them properly Access should cause EXCEPTION_GUARD_PAGE OllyDbg uses guard pages for memory breakpoints Consumes breakpoint and allows execution Used by PC Guard Peter Ferrie, Microsoft Corporation 14

Anti-debugging (cont.) Alternative desktop Sessions support multiple desktops A new desktop can be created and made active No obvious way to switch back Used by HyperUnpackMe2 Prefetch queue Queue is not flushed when rep instruction is used Allows for some old self-overwriting tricks Two known variations of the trick One is used by Invius The other is used by Obsidium Peter Ferrie, Microsoft Corporation 15

Anti-debugging (cont.) Hardware breakpoints Exception handler receives context structure Structure contains all CPU debug registers Non-zero values suggest presence of debugger Handler can also remove breakpoints and resume Instruction counting Handler can insert own hardware breakpoints Then count number of exceptions that occur Debugger might skip some breakpoints Execution timing Single-stepping through code take a long time Delay can be detected Several time sources available Peter Ferrie, Microsoft Corporation 16

Anti-debugging (cont.) Exceptions Obfuscated method to move EIP around Infinite variety of possible implementations Header entrypoint Some debuggers don't use VirtualProtectEx() So header remains write-protected Cannot set initial breakpoint Code runs freely Parent process Should be shell process if launched by clicking Usually Explorer.exe, unless debugger is involved Used by Yoda's Protector Peter Ferrie, Microsoft Corporation 17

Anti-debugging (cont.) Self-execution Simple method to escape from debugger Synchronisation object prevents infinite executions Used by MSLRH Process name Process names can be enumerated Names can be compared to "watch list" List commonly includes anti-malware software Peter Ferrie, Microsoft Corporation 18

Anti-debugging (cont.) Threads Multiple threads are hard to trace Some anti-malware emulators don't support them One thread is often devoted to debugger detection Can also be used to detect code changes Used by PE-Crypt32 Self-debugging Effective anti-debugging method Only one debugger can attach to a process Defeated by zeroing EPROCESS->DebugPort Or call DebugActiveProcessStop() on Windows XP Used by Armadillo Peter Ferrie, Microsoft Corporation 19

Anti-debugging (cont.) Disassembly Can be used to steal instructions Safer method to detect breakpoints Can also be used to detect detours Problematic if routine is hot-patched Since that will be a detour TLS callback Relatively little-known method to execute code Code executes before main entrypoint Also executes before inserted breakpoint Debuggers typically don't handle it Used by ExeCryptor Peter Ferrie, Microsoft Corporation 20

Anti-debugging (cont.) Device names A way to communicate with drivers Also a way to detect presence of drivers List of names often target SoftICE, RegMon and FileMon SoftICE-specific Driver names can be enumerated Version information can be queried Product and copyright strings can be compared to "watch list" Harder to change than renaming the driver List of strings are usually SoftICE-related Peter Ferrie, Microsoft Corporation 21

Anti-debugging (cont.) SoftICE-specific (cont.) Interrupt 1 is normally not invokable from ring 3 SoftICE hooks interrupt 1 and allows ring 3 access So wrong exception when SoftICE is running Used by SafeDisc OllyDbg-specific Cannot handle unusual NumberOfRvaAndSizes value Some unchecked fields allow memory allocation DoS Initial ESI register value is -1 on Windows XP Looks like a detection method It's just a coincidence Peter Ferrie, Microsoft Corporation 22

Anti-debugging (cont.) OllyDbg-specific (cont.) Passes user-defined data directly to _vsprintf() Leads to DoS condition Debugger window can be found by calling FindWindow("OLLYDBG") HideDebugger-specific Plug-in for OllyDbg Detectable by far jump at OpenProcess()+6 ImmunityDebugger-specific Based on OllyDbg Shares many of the same vulnerabilities WinDbg-specific Debugger window can be found by calling FindWindow("WinDbgFrameClass") Peter Ferrie, Microsoft Corporation 23

Anti-emulating Interrupt 3 Size of instruction is not checked Using "cd 03" instead of "cc" instruction -> EIP points to "03" Prefixed "cc" instruction works, too Except perhaps in some emulators Used by TryGames Time-locks Effective anti-emulation technique Causes all iterations to be consumed before anything happens Some loops can be skipped arbitrarily Some loops cannot (results might be used for decryption keys) Used by Tibs Peter Ferrie, Microsoft Corporation 24

Anti-emulating (cont.) Invalid API parameters Some emulators do not support invalid parameters in APIs An expected error code can be used for decryption keys No error code means no decryption Used by Tibs GetProcAddress There are too many possible APIs for emulators to handle So some emulators always return a value for GetProcAddress() Sometimes GetProcAddress() should return 0 A non-zero value reveals the emulator Used by NsAnti Peter Ferrie, Microsoft Corporation 25

Anti-emulating (cont.) "Modern" CPU instructions Some emulators do not handle modern CPU instructions Some malware use FPU and MMX as do-nothing instructions Emulators might give up early as a result Used by NsAnti Undocumented instructions Some emulators do not handle undocumented CPU instructions Some instructions more undocumented than others Example: multi-byte NOP Example: fully allocated PREFETCH Peter Ferrie, Microsoft Corporation 26

Anti-emulating (cont.) Selector verification Some emulators pretend to be Windows NT-based While using Windows 9x-based selector values A CS selector value larger than 0xff reveals this Used by MSLRH Memory layout Certain in-memory structures are at predictable memory locations Example: RTL_USER_PROCESS_PARAMETERS at 0x20000 If they are missing, the program might not run Used by TryGames Peter Ferrie, Microsoft Corporation 27

Anti-emulating (cont.) File-format tricks Non-aligned SizeOfImage Windows will silently round up the value Overlapping structures Tools such as IDA have a problem with this Non-standard NumberOfRvaAndSizes SoftICE and OllyDbg have a problem with this Non-aligned SizeOfRawData Non-aligned PointerToRawData Windows will silently round down the value No section table Allowed when SectionAlignment is less than 4kb Header becomes writable and executable Peter Ferrie, Microsoft Corporation 28

Anti-intercepting Write->Exec Some interceptors watch for write-then-exec Executing dummy just-written instruction can fool them Used by ASPack, but probably for multi-processor support Write^Exec Some interceptors prevent write and exec on the same page Change can be detected indirectly Kernel functions return error when writing to read-only pages VirtualQuery() and VirtualProtect() return old page attributes Peter Ferrie, Microsoft Corporation 29

Thank you More techniques in the paper http://pferrie.tripod.com Questions? Peter Ferrie, Microsoft Corporation 30