1. Model Checking x86 Executables with CodeSurfer/x86 and WPDS++ G. Balakrishnan 1, T. Reps 1,2, N. Kidd 1, A. Lal 1, J. Lim 1, D. Melski 2, R. Gruian 2, S. Yong 2, C.-H. Chen 2, and T. Teitelbaum 2,3 1 University of Wisconsin 2 GrammaTech, Inc. 3 Cornell University

2. 2 Model checker State machine OK Error report Source code Front end CFG + call graph + other info IR Construction IR Exploration Static Bug-Detection Tools

3. 3 Model checker State machine OK Error report Executable Memory-access analyzer CFG + call graph + memory-access info Static Bug-Detection Tools IR Recovery IR Exploration

4. 4 Reveals platform-specific choices made by compiler –What you see is what you get Some source-level issues go away Better platform for finding security vulnerabilities –Source-code tools: Lack of fidelity can allow vulnerabilities to escape detection Why Executables?

5. 5 Minimizing Data Lifetime? Windows –Login process keeps a user’s password in the heap after a successful login Should minimize data lifetime by –clearing memory –calling free() But... –the compiler might optimize away the memory-clearing code (“useless-code” elimination) free(buffer); memset(buffer, ‘\0’, len); free(buffer);

6. 6 Puzzle int callee(int a, int b) { int local; if (local == 5) return 1; else return 2; } int main() { int c = 5; int d = 7; int v = callee(c,d); // What is the value of v here? return 0; } Answer: 1 (for the Microsoft compiler)

7. 7 Tutorial on x86 (Intel Syntax) p = q; p = *q; *p = q; p = &a[2];

8. 8 mov ecx, edx mov ecx, [edx] mov [ecx], edx lea ecx, [esp+8] Tutorial on x86 (Intel Syntax) ecx = edx; ecx = *edx; *ecx = edx; ecx = &a[2];

9. 9 Puzzle int callee(int a, int b) { int local; if (local == 5) return 1; else return 2; } int main() { int c = 5; int d = 7; int v = callee(c,d); // What is the value of v here? return 0; } Answer: 1 (for the Microsoft compiler) mov [ebp+var_8], 5 mov [ebp+var_C], 7 mov eax, [ebp+var_C] push eax mov ecx, [ebp+var_8] push ecx call _callee... Standard prolog Prolog for 1 local push ebp mov ebp, esp sub esp, 4 push ecx

10. 10 The Vision Code-inspection tools for security analysts Analyses for identifying –security vulnerabilities and bugs –malicious behavior (code vs. memory snapshots) –commonalities and differences Platform for –de-compilation –code obfuscation –installation of protection mechanisms –remediation of security vulnerabilities –de-obfuscation (w/ assistance from dyn. tools)

11. 11 What Should a Tool Provide? IR recovery –control-flow graph (w/ indirect jumps resolved) –call graph (w/ indirect calls resolved) –identification of variables –values of pointers –used, killed, and possibly-killed variables for CFG nodes –data dependences –identification of types: base types, pointer types, structs, and classes GUI for code browsing and navigation Scripting language –API for accessing the IR –API for modifying the IR IR exploration –API for traversal/searching/pattern matching –API for defining static-analyzers/model-checkers –GUI to investigate warnings Cooperation with dynamic tools No use of symbol-table or debugging information!!!

12. 12 CodeSurfer Build SDG Browse Binary Connector Value-set Analysis Initial estimate of code vs. data procedures call sites malloc sites IDA Pro Build CFGs Parse Binary fleshed-out CFGs fleshed-out call graph used, killed, may-killed variables for CFG nodes points-to sets reports of violations Binary Rewriter Decompiler Security Analyzers User Scripts CodeSurfer/x86 Architecture

13. 13 CodeSurfer Build SDG Browse Binary Connector Value-set Analysis Initial estimate of code vs. data procedures call sites malloc sites CodeSurfer/x86 Architecture IDA Pro Build CFGs Parse Binary fleshed-out CFGs fleshed-out call graph used, killed, may-killed variables for CFG nodes points-to sets reports of violations Binary Rewriter Decompiler Security Analyzers User Scripts

14. 14 CodeSurfer Build SDG Browse Binary Connector Value-set Analysis Initial estimate of code vs. data procedures call sites malloc sites CodeSurfer/x86 Architecture IDA Pro Build CFGs Parse Binary fleshed-out CFGs fleshed-out call graph used, killed, may-killed variables for CFG nodes points-to sets reports of violations Binary Rewriter Decompiler Security Analyzers User Scripts

15. 15 CodeSurfer Build SDG Browse Binary Connector Value-set Analysis Initial estimate of code vs. data procedures call sites malloc sites CodeSurfer/x86 Architecture IDA Pro Build CFGs Parse Binary fleshed-out CFGs fleshed-out call graph used, killed, may-killed variables for CFG nodes points-to sets reports of violations Binary Rewriter Decompiler Security Analyzers User Scripts

16. 16 CodeSurfer Build SDG Browse Binary Connector Value-set Analysis Initial estimate of code vs. data procedures call sites malloc sites CodeSurfer/x86 Architecture IDA Pro Build CFGs Parse Binary fleshed-out CFGs fleshed-out call graph used, killed, may-killed variables for CFG nodes points-to sets reports of violations Binary Rewriter Decompiler Security Analyzers User Scripts

17. 17 CodeSurfer Build SDG Browse Binary Connector Value-set Analysis Initial estimate of code vs. data procedures call sites malloc sites CodeSurfer/x86 Architecture IDA Pro Build CFGs Parse Binary fleshed-out CFGs fleshed-out call graph used, killed, may-killed variables for CFG nodes points-to sets reports of violations Binary Rewriter Decompiler Security Analyzers User Scripts

18. 18 IR Recovery: Scope of our Ambitions Programs that conform to a “standard compilation model” –procedures –activation records –global data region –heap-allocated structs/objects (malloc/new) –virtual functions –dynamically linked libraries Report violations –violations of stack protocol –return address modified within procedure Memory-safety violations!

19. 19 Static Analysis of Executables: State of the Art Prior to CS/x86 Relies on symbol-table/debugging info –Atom, EEL, Vulcan, Rival Able to track only data movements via registers –EEL, Cifuentes, Debbabi, Debray Poor treatment of memory operations –Overly conservative treatment  many false positives –Non-conservative treatment  many false negatives Limited usefulness for security analysis

20. 20 An Application of CodeSurfer/x86 Project at MIT Lincoln Labs (originally classified) –Adopted CodeSurfer/x86 (replacing IDA Pro) –DARPA funding under “Dynamic quarantine of worms” –PI: Rob Cunningham; PM: Anup Ghosh Given a worm... –What are its target-discovery, propagation, and activation mechanisms? –What is its payload? Use of CodeSurfer/x86’s analysis mechanisms –Find system calls –Find their arguments –Follow dependences backwards to find where their values come from –...

21. 21 Demo CodeSurfer/C CodeSurfer/x86

22. 22 Reveals platform-specific choices made by compiler –memory layout padding between fields of a struct which variables are adjacent? –register usage –execution order –optimizations performed –compiler bugs Some source-level issues go away –analyze the actual library code, not hand-written stubs –in-line assembly code –use of multiple source languages Better platform for finding security vulnerabilities –A source-code tool would have to duplicate all choices made by the compiler & optimizer Why Executables?

23. 23 IR Exploration API for traversal/searching/pattern matching API for defining static-analyzers/model-checkers –Use a script to traverse IR –Create a model of the program as Weighted PDS –Invoke analyzer (WPDS++) Path Explorer tool –Software-assurance plug-in to CodeSurfer/x86 –Performs security-related analyses on the IR –Uses the GUI to investigate warnings

24. 24 Related Work Debray et al., “Alias analysis of executable code” [POPL 98] Cifuentes et al., “Assembly to high-level language translation” [ICSM 98] A. Mycroft, “Type-based decompilation” [ESOP 99] Linn et al., “Stack analysis of x86 executables” [Unpublished] Guo et al., Practical and accurate low-level pointer analysis” Amme et al., “Data dependence analysis of assembly code” [PACT 98] Balakrishnan and Reps, “Analyzing memory accesses in x86 executables” [CC04] http://www.cs.wisc.edu/~reps/#cc04

25. 25 Questions & Discussion

26. 26