1. Part 5: Anti-Reverse-Engineering
Chapter 15: Anti-Disassembly
Chapter 16: Anti-Debugging
Chapter 17: Anti-Virtual Machine Techniques
Chapter 18: Packing and Unpacking

2. Chapter 15: Anti-Disassembly

3. Anti-Disassembly
1. Understanding Anti-Disassembly 2. Defeating Disassembly Algorithms 3. Anti-Disassembly Techniques 4. Obscuring Flow Control 5. Thwarting Stack-Frame Analysis

4. 1. Understanding Anti-Disassembly
Special code to cause disassembly analysis to produce incorrect program listings
Goal is to delay or prevent analysis of malicious code
Thwart automated and manual analysis
Tricking disassembly at an incorrect offset
Naïve disassembly p. 328, Loc. 8288
Corrected disassembly p. 329, Loc. 8300

5. 2. Defeating Disassembly Algorithms
Two types of algorithms
Linear disassembly
Iterate over a block of code, disassembling one instruction at a time linearly
Decode blindly from start to end, ignores flow-control instructions that cause only a part of the buffer to execute
Example previously on p
Bogus, injected opcode 0xE8 assumed to be “call”
Next 4 bytes assumed to be target (can instead contain malicious code)
Example p. 330, Loc. 8344
Jump table correctly disassembled
Turns to code in a linear disassembler (p. 331, Loc. 8357)

6. Defeating Disassembly Algorithms
Two types of algorithms
Flow-oriented disassembly
Builds list of locations to assemble by examining code from entry
Example
p. 332, Loc. 8368: Decoding correctly stops after unconditional jmp
p. 332, Loc. 8388: Linear disassembly parses ‘Failed’ as code
p. 333, Loc. 8401: Call-pop to get pointer to "hello" into a register
p. 334, Loc. 8415: Instructions after call disassembled first parse string as code
p. 334, Loc 8428: Manual cleanup
Issues with flow-oriented disassembly
On conditional branch, which branch should be followed first?
On a call, do you disassemble the call or instructions after first?
Arbitrary results based on your choice!

7. 3. Anti-Disassembly Techniques
Fake conditionals
Take advantage of choice in disassembly
Jump instructions with the Same Target
Back-to-back conditional jumps with the same target
Consider Figure 15-2, p. 335, Loc. 8477
Unconditional jump that *always* skips bogus 0xE8
Confused as code if fall-through followed firstin flow-disassembler
Incorrect disassembly at p. 335, Loc. 8448
Correct disassembly at p. 335, Loc. 8462
Toggle bytes from code to data using “C” and “U”
Jump instruction with a constant condition
Figure 15-3, p. 336, Loc. 8505
XOR reg,reg followed by jz
Incorrect disassembly: p. 335, Loc. 8477
Correct disassembly: p. 336, Loc. 8491
Note: Methods use a “rogue” call/jmp byte (0xE9 or 0xE8)

8. Anti-Disassembly Techniques
Impossible disassembly
Using a single byte in two instructions
Disassembler limited to picking one interpretation, but processor can use both
Inward jump (Figure 15-4, p. 337, Loc. 8517)
More complex case (Figure 15-5, p. 338, Loc. 8533)
Incorrect disassembly (p. 338, Loc. 8541)
Correct disassembly (p. 339, Loc. 8555)
Must manually patch code or NOP out bogus instructions
IDAPython script on p. 339, Loc. 8574

9. 4. Obscuring Flow Control
Function pointers
Locations resolved at run-time
Hard to statically reverse engineer
p. 341, 2 calls to sub_4011C0 via function pointers not labeled
Return pointer abuse
Modify return value on stack at run-time (return-oriented programming)
call $+5 coupled with rogue ret instruction on p. 342
retn makes it appear sub_4011C0 is finished, but control goes just beyond retn instruction

10. Obscuring Flow Control
Misusing structured exception handlers
SEH allows program to handle error conditions intelligently
Uses a stack to manage (FS segment register)
Example on p. 346
Craft pointer to exception routine (0x401080) via top two instructions
Push address onto stack as exception handler in third instruction
Trigger exception (divide-by-zero).
Routine at 0x is not disassembled
Use IDA to manually disassemble

11. 5. Thwarting Stack-Frame Analysis
Stack-frame analysis dependent upon compiler used
Calling conventions vary
Custom management also possible such as management using esp directly
(Listing 15-1, p. 347) does not use ebp, breaking IDA Pro analysis
cmp instruction sets up a fake conditional, but IDA Pro traces incorrect branch
Assumes “add esp, 104h” is actually executed and shows esp getting into an incorrect range (at -F8)
cmp should never trigger

12. Talk

13. In-class exercise
Lab 15-01, 15-02

14. Chapter 16: Anti-Debugging

15. Anti-Debugging
Anti-analysis technique for malware to recognize when it is under the control of a debugger
Slow down analysis as much as possible to increase window of vulnerability
Hundreds of techniques

16. Anti-Debugging
1. Windows Debugger Detection 2. Identifying Debugger Behavior 3. Interfering with Debugger Functionality 4. Debugger Vulnerabilities

17. 1. Windows Debugger Detection
Using the Windows API
IsDebuggerPresent() returns 0 if no debugger attached by searching the Process Environment Block for field IsDebugged
CheckRemoteDebuggerPresent() allows one to check the IsDebugged flag on other processes
NTQueryInformationProcess using value ProcessDebugPort

18. Windows Debugger Detection
Using the Windows API
OutputDebugString() (Listing 16-1, p. 353)
Get errorValue initially
Call function
If no debugger attached when attempting to output debug string, errorValue changed to indicate
Do the malicious thing

19. Windows Debugger Detection
Direct checks of structures in memory
Bypass Windows API
Preferred by malware since API calls can be hooked by anti-virus

20. Windows Debugger Detection
Direct checks of structures in memory
BeingDebugged flag
Load PEB structure address fs:[30h]
Access BeingDebugged at offset 0x2
(Listing 16-2, Table 16-1, p. 354)

21. Windows Debugger Detection
Direct checks of structures in memory
Flags indicate if process heap was created by the debugger
Debugger heap check
Get address of process’s first heap (ProcessHeap) by loading value at 0x18 into PEB structure, then accessing flag field at 0x10 (XP) or 0x44 (Win7) (Listing 16-3, p. 355)

22. Windows Debugger Detection
Direct checks of structures in memory
Heap management different for debugged programs
NtGlobalFlag check
Specified at 0x68 offset in PEB. Set to 0x70 if debugged (Listing 16-4, p. 355)

23. Windows Debugger Detection
System artifacts
API hooks (OllyDbg detour of OpenProcess)
Registry values used by debuggers (HKLM\....\AeDebug)
Window names (e.g. OLLYDBG)
Debugging services running on system
File system artifacts
Memory artifacts (e.g. OllyDbg stores some strings at 0x004B064B)

24. 2. Identifying Debugger Behavior
INT scanning
INT 3 inserted by debugger to temporarily replace an instruction so that debug exception handler can run when software breakpoints are hit (Opcode 0xCC)
Search for 0xCC in code (Listing 16-6, p. 357)
Performing code checksums
Malware performs checksum on its code pages and exits if tampering detected
Get address
Set address to
start of code
Check 4096 bytes for 0xCC
0xCC found!

25. Identifying Debugger Behavior
Timing checks
Malware takes timestamps and exits if there is a lag
Especially effective when taken before & after an exception
Implemented via rdtsc instruction (Listing 16-7, p. 358), QueryPerformanceCounter, or GetTickCount (Listing 16-8, p. 359)

26. Identifying Debugger Behavior
Debugger artifacts
INT 1 osed to single-step program in debugger
Overwrites 6 bytes beyond current sp with return values for IP, CS, and Flags
Put canary just beyond stack and check that it survives
PUSH AX
POP AX
DEC SP
POP BX
CMP AX,BX
JNE CODE_IS_TRACED
Put value on stack
Get stack pointer to point back to it
Read it again
If not the same, then single stepping
has clobbered it

27. Identifying Debugger Behavior
Debugger artifacts
Force INT 1/INT 3 tracing to disable essential functions
Hide critical value (e.g. decryption key) on stack directly without modifying stack pointer
Debugger overwrites value if it runs
Registers/flags saved by debugger on context switches
Debug registers (DR0-DR7) saved on stack on context switch
Set handler and force exception (divide by zero)
Read and write values directly
Execute exception with Trap flag set
No debugger = SEH occurs since no one handles trap
Debugger attached = SEH will not occur (trap sent to debugger)

28. 3. Interfering with Debugger Functionality
Using TLS (thread local storage) callbacks
Debuggers typically execute until program entry point defined by the PE header
TLS implemented in an executable contains a .tls section that can be executed before program entry point
Malware can hide functionality in TLS
Must configure debugger to pause before TLS callback code

29. Interfering with Debugger Functionality
Using exceptions
Debuggers typically configured to trap exceptions and not pass them through to program
Malware probes to ensure exceptions are passed through quickly
Must configure debugger to pass them through automatically

30. Interfering with Debugger Functionality
Inserting interrupts
Inserting a long loop of INT 3 instructions or 0xCD03 (STATUS_BREAKPOINT) to generate an INT 3.
Trolls the reverse-engineer
Doing the same with INT 2D (kernel debugger breakpoint)

31. Interfering with Debugger Functionality
Putting payload in interrupt handler
Having a debugger attached results in disabling malware
Make malicious code be a part of an SEH handler
INT 3 without debugger invokes SEH handler to execute malicous code
INT 3 with debugger goes to next instruction unless debugger configured appropriately (Listing 16-9, p. 363)
Place SEH on stack (continue)
Trigger exception
Code that does nothing and exits
Malicious code

32. Interfering with Debugger Functionality
Use interrupt handling to obfuscate execution
Example: Running line of code
Hook INT 1
Decrypt next instruction, encrypt previous one
Only one instruction decrypted in memory at a time
Hard to analyze
Themida, last level of Microcorruption

33. Interfering with Debugger Functionality
Modifying expected interrupt behavior
Continually overwrite Interrupt Vector of INT 1/3 instructions to point to garbage code to crash debugger
Turn off keyboard interrupts
IN AL, 20h
OR AL, 02
OUT AL, 20
<malicious code>
IN AL, 20
AND AL, NOT 2
OUT AL,20

34. 4. Debugger Vulnerabilities
PE header vulnerabilities
OllyDbg follows specifications of PE headers more strictly than Windows. Crashes on malformed headers that will run without debugger
Code vulnerabilities
Pass malformed string to crash debugger
OutputDebugString vulnerable to format string vulnerability in OllyDbg v. 1.1.
See fuzzing lecture
Exploit instructions that OllyDbg handles differently than CPU to crash debugger
Exploit exceptions that OllyDbg handles differently than CPU to crash debugger (memory handling)

35. In-class exercise
Lab 16-01

36. Chapter 17: Anti-Virtual Machine Techniques

37. Anti-Virtual Machine Techniques
Virtual machines initially used only by malware analysts
Malware benefited from detecting VM (especially VMware) and shutting down to escape analysis
Rollback recovery easy
Portability

38. Anti-Virtual Machine Techniques
1. VMware Artifacts 2. Vulnerable Instructions 3. Tweaking Settings 4. Escaping the Virtual Machine

39. 1. VMware Artifacts
Process listing (Figure 17-1, p. 370)

40. VMware Artifacts Filesystem Registry keys (p. 371)
(e.g. C:\Program Files\VMware\VMware Tools)
Registry keys (p. 371)

41. VMware Artifacts Networking
MAC addresses assigned for use by IEEE for VMware NICs begin with 00:0C:29
Memory (invariant strings in VMware virtual machine)

42. VMware Artifacts Example code to check Must disable checks
Listing 17-1, p. 372
Must disable checks
Patch condition on branch to bypass in debugger
Use hex editor to modify VMware string
Uninstall VMware tool being checked

43. 2. Vulnerable Instructions
Descriptor Table Instructions
3 special x86 registers for pointing to machine-wide data structures
IDTR: points to Interrupt Descriptor Table Register
GDTR: points to Global Descriptor Table Register (memory lookups)
LDTR: points to Local Descriptor Table Register (unused in Windows)
Guest VM must have a different location for these tables than Host VM
VM software creates separate locations
But, Guest VM can directly execute x86 instructions that directly access underlying registers to check for inconsistency (user-mode instructions not caught by hypervisor)
Must NOP out these checks

44. 2. Vulnerable Instructions
Examples of Descriptor Table checks
Red Pill
x86 instruction sidt loads value of IDTR (Listing 17-2, p. 374)
VMware sets 5th byte of IDTR to 0xFF
No Pill
x86 instruction sldt loads value of LDTR
Zero on non-virtualized, Non-zero on virtualized
x86 instruction sgdt loads value of GDTR

45. Vulnerable Instructions
Querying the I/O communication port (Phatbot, Storm)
VMware virtualizes I/O ports
Port can be queried to detect presence of VMware
Obtaining VMware version via IO port (Listing 17-3, p. 376)
Must NOP out the check

46. Vulnerable Instructions
Common Anti-VM instructions
sidt, sgdt, sldt, smsw, str, in, cpuid
20 instructions designated by VMware as “not virtualizable”
Malware will only use them if checking for VMs

47. 3. Tweaking Settings
VMware provides options to hide itself from malware
Listing 17-5, p. 379
Protects against all checks implemented by ScoopyNG, a free VMware detection tool
Last-resort since performance will crater if used

48. 4. Escaping the Virtual Machine
Exploiting VMware bugs to crash host or run code in it
Prior exploits (now patched) include shared folder feature, drag-and-drop functionality in VMware Tools, shared clipboard, VM display function

49. Talk
Blue Pill, Don't Tell Joanna

50. In-class exercise
Lab 17-01

51. Chapter 18: Packers and Unpacking

52. Packers
Used to shrink and encrypt malware to thwart detection by antivirus
Original executable transformed to a unique self-extracting one via compression, encryption, and obfuscation
Difficult to reverse-engineer
Prevents static analysis since malware must be unpacked and decrypted before analysis
Employs anti-disassembly, anti-debugging, and anti-VM techniques to make analysis difficult

53. Packers

54. Packers and Unpacking
1. Packer Anatomy 2. Identifying Packed Programs 3. Unpacking Options 4. Tips and Tricks for Common Packers 5. Packed DLLs

55. 1. Packer Anatomy Unpacking Stub
Small piece of code loaded by the operating system just as a normal program
Unpacking stub then loads original program
Step 1: Unpacking original executable into memory
Loader reads PE header and copies packed sections into allocated memory normally
Unpacking code called and does the same for packed code

56. Packer Anatomy Step 1 (cont.) Within unpacking routine, unpacking loop
Save flags and all registers (PUSHFD, PUSHAD), call unpacking routine
Within unpacking routine, unpacking loop
0040AC44 FFFF INVALID
0040AC4C 9C PUSHFD
0040AC4D PUSHAD
0040AC4E E CALL AC55
**If you step over this CALL using F10, the program will run.
Thus, reload the program and step into this CALL using F8 next time.
aaaaaaaa
...
wwwwwwww
xxxxxxxx JNZ zzzzzzzz <-- Loop back to aaaaaaaa
yyyyyyyy JMP aaaaaaaa
zzzzzzzz Rest of unpacking code

57. Packer Anatomy Step 2: Resolving imports of original executable
On normal binaries, loader reads PE header to find library functions to import and their addresses
Unless packed code's imports included in unpacking code's import section, unpacker must resolve imports manually using LoadLibrary and GetProcAddress (will revisit in Chapter 19)

58. Packer Anatomy
Step 3: Transfer of execution to original execution point (OEP)
POPAD followed by tail jmp to entry point (Listing 18-1, p. 392)
Note: empty bytes after JMP and huge offset
IDA Pro can identify JMP goes to garbage and flags it red (Figure 18-5, p. 393)

59. 2. Identifying Packed Programs
Simple indicators
Program with few imports and imports are LoadLibrary and GetProcAddress
IDA Pro recognizes a small amount of code
Presence of UPX0 section (a specific packer)
Abnormal section sizes
Entropy calculation
Disorder in a program much larger in encrypted and compressed payloads

60. 3. Unpacking Options Automated static unpacking
Specific to a packer (i.e. you must know which packer was used)
Apply decompression and decryption based on packer type
PE Explorer
Supports NSPack, UPack, and UPX
Automated dynamic unpacking
Program is run and unpacker stub is allowed to unpack original executable
Tool attempts to find when tail jump is reached, then memory is dumped and original program written to disk
Fails if the end of unpacking stub is not identified properly
Not many publicly available tools for this

61. Unpacking Options Manual dynamic unpacking
Option #1: Discover packing algorithm and write a program to run it in reverse
Option #2: Run unpacking stub
Find OEP via stack trace
Registers pushed before entry to unpacking stub
Set breakpoint on %esp accessing those stack locations to catch their restoration (indicates a jump to OEP forthcoming)
Find OEP via iteration
Break at end of main unpacking loop, step to tail jump
Then, break and dump the process out of memory (Listing 18-2 and 18-3, p )

62. Unpacking Options

63. Unpacking Options Issues in dumping unpacked binaries
Import table of function names and addresses not reconstructed
Listing 18-4, p. 396 functions not labeled
Addressed via manual import table patching
Patch PE header to reconstruct import tables
Cross-reference between OllyDbg and IDA Pro to patch import table with function names
OllyDump plug-in for OllyDbg (performs OEP identification, import table reconstruction, entry point patching)
ImpRec (Import Reconstructor tool) when OllyDump fails to build a proper import table

64. 4. Tips and Tricks for Common Packers
UPX (Ultimate Packer for eXecutables)
Open-source
Designed for compression not for security
OllyDump finds easily using heuristics previously described
PECompact
Similar to UPX, uses a tail jump of jmp *eax
ASPack
Uses self-modifying code to thwart analysis

65. Tips and Tricks for Common Packers
Petite
Uses single-step exceptions to break into debugger
Must pass single-step exceptions back to Petite or employ hardware breakpoints to find OEP
WinUpack
Uses PUSH followed by RET for tail jump
Placed in the middle of stub (Listing 18-5, p. 399)

66. Tips and Tricks for Common Packers
Themida
Secure packer employing anti-debugging, anti-analysis, and anti-VM techniques
Contains a kernel component making it difficult to follow
Runs code continuously
Use ProcDump to dump memory without attaching debugger
Blacklisted and identified as malware whenever found

67. 5. Packed DLLs Similar to executables
Unpacking stub contained in DllMain
DllMain unpacks original DLL
Some debuggers execute DllMain before breaking
Need to set IMAGE_FILE_HEADER values to cause DLL to be interpreted as executable to prevent

68. In-class exercise
Lab 18-1