1. 1 Malware Analysis and Instrumentation Andrew Bernat and Kevin Roundy Paradyn Project Paradyn / Dyninst Week Madison, Wisconsin May 2-4, 2011

2. Forensic analysts need help Malware Analysis and Instrumentation 2 90% of malware resists analysis [1]  Malware attacks cost billions of dollars annually [2]  65% of users feel effect of cyber crime [3]  69% cybercrimes are resolved [3]  28 days on average to resolve a cybercrime [3] [1] McAfee. 2008 [2] Computer Economics. 2007 [3] Norton. 2010 7a 77 0e 20 e9 3d e0 09 e8 68 c0 45 be 79 5e 80 89 08 27 c0 73 1c 88 48 6a d8 6a d0 56 4b fe 92 57 af 40 0c b6 f2 64 32 f5 07 b6 66 21 0c 85 a5 94 2b 20 fd 5b 95 Malware Binary

3. Malware Analysis and Instrumentation 3 7a 77 0e 20 e9 3d e0 09 e8 68 c0 45 be 79 5e 80 89 08 27 c0 73 1c 88 48 6a d8 6a d0 56 4b fe 92 57 af 40 0c b6 f2 64 32 f5 07 b6 66 21 0c 85 a5 94 2b 20 fd 5b 95 Malware Binary Binary code identification Control- and data-flow analysis Instrumentation Effectiveness on malware The needed toolbox Forensic analysts need help

4. Malware Analysis and Instrumentation Dyninst Dyninst is a toolbox for analysts 4 program binary 7a 77 0e 20 e9 3d e0 09 e8 68 c0 45 be 79 5e 80 89 08 27 c0 73 1c 88 48 6a d8 6a d0 56 4b fe 92 57 af 40 0c b6 f2 64 32 f5 07 b6 66 21 Dyninst CFG loop, block, function, instruction instrument- ation function replace- ment call stack walking forward & backward slices loop analysis process control library injection symbol table reading, writing binary rewriting machine language parsing Control flow analyzer Instrumenter Data flow analyzer

5. Analysis tool Dyninst Dyninst is a toolbox for analysts Malware Analysis and Instrumentation Mutator  Specifies instrumentation  Gets callbacks for runtime events  Builds high-level analysis program binary 7a 77 0e 20 e9 3d e0 09 e8 68 c0 45 be 79 5e 80 89 08 27 c0 73 1c 88 48 6a d8 6a d0 56 4b fe 92 57 af 40 0c b6 f2 64 32 f5 07 b6 66 21 Dyninst Control flow analyzer Instrumenter Data flow analyzer CFG 5 loop, block, function, instruction instrument- ation function replace- ment call stack walking forward & backward slices loop analysis process control library injection symbol table reading, writing binary rewriting machine language parsing

6. Analysis tool Dyninst is a toolbox for analysts Malware Analysis and Instrumentation 6 Analysis of network communications Code visualizations Time bomb detection and analysis Identification of stolen data Reports on anti- analysis techniques printf(…) counter++ if (pred) callback(…) getTarget(insn) Code snippets Mutator  Specifies instrumentation  Gets callbacks for runtime events  Builds high-level analysis program binary 7a 77 0e 20 e9 3d e0 09 e8 68 c0 45 be 79 5e 80 89 08 27 c0 73 1c 88 48 6a d8 6a d0 56 4b fe 92 57 af 40 0c b6 f2 64 32 f5 07 b6 66 21 CFG Dyninst Control flow analyzer Instrumenter Data flow analyzer

7. Analysis tool Dyninst Dyninst on malware Malware Analysis and Instrumentation 7 printf(…) counter++ if (pred) callback(…) getTarget(insn) Code snippets Mutator  Specifies instrumentation  Gets callbacks for runtime events  Builds high-level analysis Malware defeats static analysis & is sensitive to instrument- ation malware binary 7a 77 0e 20 e9 3d e0 09 e8 68 c0 45 be 79 5e 80 89 08 27 c0 73 1c 88 48 6a d8 6a d0 56 4b fe 92 57 af 40 0c b6 f2 64 32 f5 07 b6 66 21 CFG Analysis of network communications Code visualizations Time bomb detection and analysis Identification of stolen data Reports on anti- analysis techniques Analysis of network communications Code visualizations Time bomb detection and analysis Identification of stolen data Reports on anti- analysis techniques Control flow analyzer Instrumenter Data flow analyzer

8. Analysis tool Dyninst Control flow analyzer Instrument- er Data flow analyzer Dyninst on malware Malware Analysis and Instrumentation 8 printf(…) counter++ if (pred) callback(…) getTarget(insn) Code snippets Mutator  Specifies instrumentation  Gets callbacks for runtime events  Builds high-level analysis Malware defeats static analysis & is sensitive to instrument- ation malware binary 7a 77 0e 20 e9 3d e0 09 e8 68 c0 45 be 79 5e 80 89 08 27 c0 73 1c 88 48 6a d8 6a d0 56 4b fe 92 57 af 40 0c b6 f2 64 32 f5 07 b6 66 21 CFG SR-Dyninst static-dynamic analysis Analysis of network communications Code visualizations Time bomb detection and analysis Identification of stolen data Reports on anti- analysis techniques Control flow analyzer Sensitivity Resistant Instrumenter Data flow analyzer

9. Outline Malware Analysis and Instrumentation 9 Anti-analysis tricks Hybrid static-dynamic analysis Sensitivity resistance Results H.A. Anti S.R. Res. 9

10. PC-sensitive code Obfuscated control flow Unpacked code Overwritten code Anti-patching Address-space probing PC-sensitive code call-pop pairs, return-address manipulation, call-stack tampering & probing Anti-analysis tricks Malware Analysis and Instrumentation 10 Obfuscated control flow indirect control flow, stack tampering, overlapping code, signal-based ctrl flow Unpacked code all-at-once, block-, loop-, function-at-a-time, to empty or allocated space Overwritten code single operand or opcode, whole instruction, function, code section, buffer Anti-patching checksum whole regions, probe for patches, use code as data, move stack ptr Anti Address-space probing scans & probes of locations that should be un-allocated Anti-analysis Anti-instrumentation

11. 030405060708090a0b0c0d e80300 e9eb045d4555c3 CALLJMP 40d00a459dd4f7 JMPPOPINCPUSHRET 40d00eebp anti-patching storm worm Obfuscated control flow Malware Analysis and Instrumentation 11 obfuscated control flow 40d002 address-space probing unpacked code overwritten code obfuscated control flow Entry Point pc-sensitive code Anti

12. storm worm Unpacked code Malware Analysis and Instrumentation 12 Entry Point 7a 77 0e 20 e9 3d e0 09 e8 68 c0 45 be 79 5e 80 89 08 27 c0 73 1c 88 48 6a d8 6a d0 56 4b fe 92 57 af 40 0c b6 f2 64 32 f5 07 b6 66 21 0c 85 a5 94 2b 20 fd 5b 95 e7 c2 16 90 14 8a 14 26 60 d9 83 a1 37 1b 2f b9 51 84 02 1c 22 8e 63 01 obfuscated control flow unpacked code obfuscated control flow Anti 12 anti-patching address-space probing overwritten code pc-sensitive code

13. Overwritten code Malware Analysis and Instrumentation 13 Upack packer obfuscated control flow overwritten code obfuscated control flow Anti 13 anti-patching address-space probing pc-sensitive code unpacked code 7a 77 0e 20 e9 3d e0 09 e8 68 c0 45 be 79 5e 80 89 08 27 c0 73 1c 88 48 6a d8 6a d0 56 4b fe 92 57 af 40 0c b6 f2 64 32 f5 07 b6 66 21 0c 85 a5 94 2b 20 fd 79 5e 80 89 08 27 c0 73 1c 88 48 6a d8 5b 95 e7 c2 16 90 14 8a 14 26 60 d9 83 a1 37 1b 2f b9 51 84 02 1c 22 8e 63 01 Entry Point

14. PC Sensitive code Malware Analysis and Instrumentation 14 obfuscated control flow overwritten code obfuscated control flow Anti 14 anti-patching address-space probing pc-sensitive code unpacked code Local Data Access call pop esi add esi, eax mov ebx, ptr[esi] data Use call to get current PC Pop PC into register Construct pointer and dereference e.g., ASProtect

15. anti-patching obfuscated control flow Anti-patching Malware Analysis and Instrumentation 15 checksum routine protected code xor eax, eax cmp eax,.chksum jne.fail e.g., PECompact Checksumming detects instrumentation [Aucsmith 96] add eax, ptr[ebx] add 4, ebx cmp ebx, 0x41000 jne.loop jmp instrument- ation is detected passfail calculate checksum of protected region compare to expected value Anti 15 address-space probing unpacked code overwritten code pc-sensitive code

16. Address-space probing Malware Analysis and Instrumentation 16 obfuscated control flow overwritten code obfuscated control flow Anti 16 anti-patching address-space probing pc-sensitive code unpacked code segv_handler() { ptr += PAGESIZE; goto RESTART: } int *ptr = 0; sigaction(SIGSEGV, segv_handler); while(1) { RESTART: *ptr; ptr += PAGESIZE; } data code instrumentation Memory Scan

17. Malware Analysis and Instrumentation 17 Code discovery algorithm Hybrid algorithm: ? ? Parse from known entry points Instrument control flow that may lead to new code Resume execution H.A. instrumentexceptionoverwrite CALL ptr[eax] DIV eax, 0

18. Malware Analysis and Instrumentation 18 Code discovery algorithm ? Parse from known entry points Instrument control flow that may lead to new code Resume execution ? Hybrid algorithm: H.A. instrumentexceptionoverwrite CALL ptr[eax] DIV eax, 0

19. Malware Analysis and Instrumentation 19 Code discovery algorithm ? Parse from known entry points Instrument control flow that may lead to new code Resume execution ? Hybrid algorithm: H.A. instrumentexceptionoverwrite CALL ptr[eax] DIV eax, 0

20. Malware Analysis and Instrumentation 20 Code discovery algorithm ? Parse from known entry points Instrument control flow that may lead to new code Resume execution ? Hybrid algorithm: H.A. instrumentexceptionoverwrite CALL ptr[eax] DIV eax, 0

21. Malware Analysis and Instrumentation 21 Code discovery algorithm Parse from known entry points Instrument control flow that may lead to new code Resume execution ? Hybrid algorithm: H.A. instrumentexceptionoverwrite CALL ptr[eax] DIV eax, 0

22. Malware Analysis and Instrumentation 22 Instrumentation-based discovery H.A. Invalid control transfers Indirect control transfers Exception-based control transfers push eax ret call 401000 Invalid Region call ptr[eax] ? jmp eax ? xor eax, eax mov ebx, ptr[eax] Exception Handler

23. Overwritten code discovery Malware Analysis and Instrumentation 23 Dyninst write RWX 23 H.A. RWX

24. Update after overwrite 1.Handle overwrite signal a)instrument write loop exits b)copy overwritten page c)restore write permissions d)resume execution 2.Update CFG when writes end a)remove overwritten and unreachable blocks b)parse at entry points to overwritten regions c)remove write permissions d)resume execution Overwritten code discovery Malware Analysis and Instrumentation 24 Dyninst R-X code write handler CFG update routine write Update after overwrite 1.Handle overwrite signal a)instrument write loop exits b)copy overwritten page c)restore write permissions d)resume execution 2.Update CFG when writes end a)remove overwritten and unreachable blocks b)parse at entry points to overwritten regions c)remove write permissions d)resume execution cb RWX cb R-X 24 H.A.

25. Dyninst Overwritten code discovery Malware Analysis and Instrumentation 25 Update after overwrite 1.Handle overwrite signal a)instrument write loop exits b)copy overwritten page c)restore write permissions d)resume execution 2.Update CFG when writes end a)remove overwritten and unreachable blocks b)parse at entry points to overwritten regions c)remove write permissions d)resume execution R-X RWX code write handler CFG update routine cb write cb 25 H.A.

26. PC-sensitivity analysis Malware Analysis and Instrumentation 26 SR-Dyninst S.R. call... data... pop esi add esi, eax mov ebx, ptr[esi]... process main: reloc_main: push jmp 0 pop esi add esi, eax mov ebx, ptr[esi]... Relocate Analyze

27. Anti-anti patching Malware Analysis and Instrumentation 27 S.R. checksum routine xor eax, eax cmp eax,.chksum jne.fail add eax, ptr[ebx] add 4, ebx cmp ebx, 0x41000 jne.loop passfail data code instrumentation patch add 4, ebx cmp ebx, 0x41000 jne.loop emulate (add eax, ptr[ebx]) restore state save state jmp 863828 shadow memory

28. Address-space scanning Malware Analysis and Instrumentation 28 S.R. scan routine xor eax, eax call chk_mem mov ptr[eax], ebx add 4, eax cmp eax, 0x0 jne.loop passfail data code instrumentation patch add 4, eax cmp ebx, 0x0 jne.loop emulate (mov ptr[eax], ebx) restore state save state jmp 863828 segv_handler... dyn_segv_handler...

29. Dyninst SR- Dyninst x x √ √ √ x √ √ √ √ √ √ yes Malware Analysis and Instrumentation 29 The packers we’re studying [1] Packer (r)evolution. Panda Research, 2008. Two-month average Feb-March 2008. Packer Malware market share [1] 0.13%MEW 0.17%WinUPack 0.33%Yoda's Protector 0.37%Armadillo 0.43%Asprotect 1.26%FSG 1.29%Aspack 1.74%nPack 2.08%Upack 2.59%PECompact 2.95%Themida 4.06%EXECryptor 6.21%PolyEnE 9.45%UPX 0.89%Nspack Res. Self- modifying yes Anti instru- mentation yes Obfuscated yes √ √ √ anti-debugging techniques

30.  Reduced relocation overhead despite emulation  Better handling of program features  Exceptions  Indirect control flow Malware Analysis and Instrumentation 30 Improved Dyninst overhead Res.

31. Malware Analysis and Instrumentation 31 Conclusion SR-Dyninst gives you  All the benefits of Dyninst on malware  Safer instrumentation on normal binaries Ongoing work  Anti-debugger techniques  More descriptive CFGs  Automated defensive-mode activation  SR-Dyninst in next Dyninst release