1. Environment Selection Application  Firefox 1.0 or 2.0  Apache 2.0.36 Operating System  Linux  Windows XP Instrumentation Package  JIT (DynamoRio, Pin)  Trampoline (Dyninst, Detours, Pin, etc)

2. Application Firefox 1.0 (Phase 1)  Complex app with embedded interpreter  39 to 46 applicable vulnerabilities Firefox 2.0  Similar vulnerabilities as 1.0 Apache 2.0.36  Less complex application  6-8 applicable vulnerabilities Proposal: Firefox 1.0  Many interesting vulnerabilities  Leverages Phase 1 experience

3. Operating System Linux  Open source  Open source tools (gcc, Xnee, etc)  Instrumentation tools are supported better Windows XP  Closed source  More marketable results Proposal: Windows XP  No show stoppers for Windows  Shows program is more generally applicable

4. Instrumentation Tools Instrumentation tool approaches  JIT  Probe based Call interception  System call  Library call

5. JIT Binary Translation PIN & DynamoRIO Allows us, at runtime, to manipulate every instruction, with:  Minimal performance overhead  Full transparency Exports interface for building custom tools No modifications to hardware, operating system, or application

6. How does it work? (conceptually) fetchdecodeexecute Start

7. In more detail 120% to 200%

8. JIT-mode Summary Powerful instruction-level instrumentation  Supports shadow stack  Supports arbitrary repairs  Stack-walk Direct access to system call gateway

9. Probe based instrumentation: PIN probe, Dyninst, Detours

10. Probe-based Repair

11. Probe-mode Summary Considerably faster than JIT-mode  No constant performance overhead Potential issues  x86: need at least 5 bytes for trampolines  Can be expensive for fine-grained instrumentation  Limited to function-level instrumentation  Does not support shadow stack

12. Direct System Call Interception Application System call gateway Operating System Interception

13. Library System Call Interception Application System call gateway Operating System Win32 API Win32 DLLs Interception

14. Issue With Library Interception Can only catch system calls made through API (libc, win32API) Malicious attacker could inject a different version of the library we are intercepting  But that would require code-injection

15. Library Interception Can only catch system calls made through API (libc, win32API) Malicious attacker could inject a different version of the library we are intercepting  But that would require code-injection Stable, coherent interface

16. Monitor/Repair Matrix ToolTypeOSStack Replace Args Change or drop syscall Syscall return value Perform ance PINJIT Win, Linux SS SW YYY 500% 240% PINProbe Win, Linux SWLLL180% DRJIT Win, Linux SS SW YLL 400% 220% DetoursProbeWinSWLLL~180% DyninstProbeLinuxSWLLL~180%

17. PIN Automatically in-lines instrumentation code  Uses callouts ‣ More expensive but easy to write  No restrictions on library usage Simple, easy-to-use API Works on Linux and Windows Two modes of operation: JIT and Probe  Cover both models we want to use Only slightly slower than DynamoRIO

18. DynamoRIO Lower level interface Library calls are constrained  Must use DR version of calls (e.g., malloc)  Some calls (e.g., sockets) not supported Does not allow direct manipulation of system calls Just released as open source Phase 1 code (shadow stack, HeapGuard) now available

19. Plan Use Pin to develop prototype  Supports both JIT and Probe  Easy to use Implement final approach later  Evaluate numerous exploits  Understand what our needs are Options  Use probe mode if possible  Consider DynamoRio if necessary for speed and/or flexibility

20. Conclusion Application: Firefox 1.0 Operating System: Windows XP Instrumentation: Pin for now

21. Windows-Linux: Development Windows is closed source  Forced to reverse engineer Windows (and its tools) to debug problems  Visual Studio compiler is closed source ‣ Difficult to debug ‣ Cygwin environment has issues

22. Accomplishments Monitoring framework Analysis framework Reproduction framework

23. Monitoring Framework PIN-based monitoring tool Two modes of operation  JIT  Probe Analysis  Shadow stack  Stack walk (after we disable FP optimization)

24. Analysis Framework Tools for analyzing callstack information  Finite state automata data-structure  Visual representation  Suffix-tree fast lookup comparisons Implemented in Python  Using networkx libraries  Cross platform

25. Reproduction Framework Automate training  Record & replay user interactions with FF ‣ Record mouse & keyboard events  Works in Linux ‣ Using Xnee

26. Windows - Linux Pin Performance Windows  Shadow Stack: ~4.5x  Stack Walk: ~0.40x Linux  Shadow Stack: ~4.0x  Stack Walk: ~0.30x