1. Software & Services Group PinADX: Customizable Debugging with Dynamic Instrumentation Gregory Lueck, Harish Patil, Cristiano Pereira Intel Corporation CGO 2012, San Jose, USA 1

2. Software & Services Group Hypothetical Problem 1 2 (gdb) run Program received signal SIGSEGV, Segmentation fault. 0x0000000000000000 in ?? () (gdb) bt #0 0x0000000000000000 in ?? () #1 0x0000000000000000 in ?? () (gdb) run Program received signal SIGSEGV, Segmentation fault. 0x0000000000000000 in ?? () (gdb) bt #0 0x0000000000000000 in ?? () #1 0x0000000000000000 in ?? () Crash with bad PC and no stack trace Corrupted return address someplace... Want to stop BEFORE bad “ret” instruction

3. Software & Services Group Hypothetical Problem 2 3 thread stack ? ? ? ?... Massively threaded application How much stack space needed? At what point does each thread use its max stack?

4. Software & Services Group Traditional Debugger Breakpoints? 4 Original Application … 1.Debugger catches trap 2.Check if “ret” is to good PC 3.If yes, resume How can debugger find all “ret” instructions? Horribly slow to trap on each return Foo: ret Bar: … ret Application In Debugger … Foo: trap Bar: … trap Overwrite each “ret” with trap ret

5. Software & Services Group Dynamic Binary Instrumenation (DBI) 5 Application if (return to bad PC) Breakpoint() Application if (stack too big) Breakpoint() Much faster – avoids trap overhead DBI can find all “ret” instructions reliably General approach – solves stack problem (and others) BUT difficult to integrate with debugger sub 0x60, %sp … Foo: ret Bar: … ret Foo: Bar: sub 0x10, %sp … … Instrumentation

6. Software & Services Group Pin Overview Tool JIT compiler Application Code Cache Traces Tool controls instrumentation (e.g. “if return to bad PC”) JIT compiler fetches application instructions, calls tool to instrument Instrumented instructions stored in code cache for efficiency Fetch Instrument Optimize Store & execute 6

7. Software & Services Group Original CodeCode Cache JIT Compiler Overview 1’ 2 1 6 3 5 2’ 6’ Pin 4 3’ 5’ Tool inserts instrumentation (e.g. check if return to bad PC) Dynamic recompilation makes debugging hard 7

8. Software & Services Group Process running under Pin PinADX Architecture 8 Tool Pin Application Debugger PinADX presents “pure” view of application. Hides effect of instrumentation and recompilation. Tool extends debugger via instrumentation. PinADX core GDB or Microsoft Visual Studio 11 Supports Linux & Windows

9. Software & Services Group Rest of the Talk Introduction / Motivation Example: Using “Stack Debugger” extension Example: Authoring “Stack Debugger” extension Implementing PinADX 9

10. Software & Services Group $ gdb./my-application (gdb) target remote :1234 (gdb) break PrintHello Breakpoint 1 at 0x4004dd: file hw.c, line 13 (gdb) cont Breakpoint 1, PrintHello () at hw.c:13 (gdb) backtrace #0 PrintHello () at hw.c:13 #1 main () at hw.c:7 (gdb) x/2i $pc => 0x4004dd : mov $0x4005e8,%edi 0x4004e2 : callq 0x4003b8 Example – Stack Debugger 10 $ pin –appdebug –t stack-debugger.so --./my-application Application stopped until continued from debugger. Start GDB, then issue this command at the (gdb) prompt: target remote :1234 Run application under Pin Debugger connected to Pin (gdb) cont Breakpoint 1, PrintHello () at hw.c:13 (gdb) backtrace #0 PrintHello () at hw.c:13 #1 main () at hw.c:7 (gdb) x/2i $pc => 0x4004dd : mov $0x4005e8,%edi 0x4004e2 : callq 0x4003b8 (gdb) break PrintHello Breakpoint 1 at 0x4004dd: file hw.c, line 13

11. Software & Services Group (gdb) monitor stackbreak 4000 Break when thread uses 4000 stack bytes (gdb) cont Stopped: Thread uses 4004 bytes of stack (gdb) backtrace #0 0x3f07214445 in _dl_runtime_resolve () #1 0x00004004e7 in PrintHello () at hw.c:13 #2 0x00004004d2 in main () at hw.c:7 (gdb) monitor stackbreak 10000 Break when thread uses 10000 stack bytes (gdb) break exit Breakpoint 2 at 0x7fffe60f9650 (gdb) cont Breakpoint 2, 0x7fffe60f9650 in exit () (gdb) monitor stats Maximum stack usage: 8560 bytes. Example – Stack Debugger 11 Stop when application uses too much stack (gdb) monitor stackbreak 4000 Break when thread uses 4000 stack bytes (gdb) cont Stopped: Thread uses 4004 bytes of stack (gdb) monitor stackbreak 10000 Break when thread uses 10000 stack bytes (gdb) break exit Breakpoint 2 at 0x7fffe60f9650 (gdb) cont Breakpoint 2, 0x7fffe60f9650 in exit () (gdb) backtrace #0 0x3f07214445 in _dl_runtime_resolve () #1 0x00004004e7 in PrintHello () at hw.c:13 #2 0x00004004d2 in main () at hw.c:7 (gdb) monitor stats Maximum stack usage: 8560 bytes.

12. Software & Services Group Rest of the Talk Introduction / Motivation Example: Using “Stack Debugger” extension Example: Authoring “Stack Debugger” extension Implementing PinADX 12

13. Software & Services Group Stack Debugger – Instrumentation 13 Thread Start: […] sub$0x60, %esp cmp%esi, %edx jle size = StackBase - %esp; if (size > MaxStack) MaxStack = size; if (size > StackLimit) TriggerBreakpoint(); StackBase = %esp; MaxStack = 0; After each stack-changing instruction Record initial stack

14. Software & Services Group VOID Instruction(INS ins, VOID *) { if (INS_RegWContain(ins, REG_STACK_PTR)) { IPOINT where = (INS_HasFallThrough(ins)) ? IPOINT_AFTER : IPOINT_TAKEN_BRANCH; INS_InsertCall(ins, where, (AFUNPTR)OnStackChange, IARG_REG_VALUE, REG_STACK_PTR, IARG_THREAD_ID, IARG_CONTEXT, IARG_END); } } VOID OnStackChange(ADDRINT sp, THREADID tid, CONTEXT *ctxt) { size_t size = StackBase - sp; if (size > StackMax) StackMax = size; if (size > StackLimit) { ostringstream os; os << "Stopped: Thread uses " << size << " stack bytes."; PIN_ApplicationBreakpoint(ctxt, tid, FALSE, os.str()); } } Stack Debugger – Implementation 14 Instrumentation Analysis Instrument only instructions that change $SP Call after each instruction

15. Software & Services Group int main() { […] PIN_AddDebugInterpreter(HandleDebugCommand, 0); } BOOL HandleDebugCommand(const string &cmd, string *result) { if (cmd == "stats") { ostringstream os; os << "Maximum stack usage: " << StackMax << " bytes.\n"; *result = os.str(); return TRUE; } else if (cmd.find("stackbreak ") == 0) { StackLimit = /* parse limit */; ostringstream os; os << "Break when thread uses " << limit << " stack bytes."; *result = os.str(); return TRUE; } return FALSE; // Unknown command } Stack Debugger – Implementation 15

16. Software & Services Group Visual Studio IDE Extension 16

17. Software & Services Group Other Debugger Extensions Intel Inspector XE Product –Memory Checker –Thread Checker Intel SDE: Instruction emulation Debug from log file (PinPlay, CGO 2010) Dynamic slicing (Rajiv Gupta, UC Riverside) Cmp$im: Cache simulator Write your own! 17

18. Software & Services Group Rest of the Talk Introduction / Motivation Example: Using “Stack Debugger” extension Example: Authoring “Stack Debugger” extension Implementing PinADX 18

19. Software & Services Group Process running under Pin PinADX Architecture 19 Tool Pin Application Debugger PinADX presents “pure” view of application. Hides effect of instrumentation and recompilation. Tool extends debugger via instrumentation. PinADX core GDB or Microsoft Visual Studio 11

20. Software & Services Group Communication Details 20 Very low level Symbol processing in debugger Expression evaluation in debugger Extension of GDB’s remote debugging protocol Commands Read / write registers, memory Set breakpoints Continue, single-step, stop Notifications Breakpoint triggered Caught signal Application exited Debugger PinADX core Pin

21. Software & Services Group Communication Details 21 Breakpoint alternatives Insert real INT3 trap instruction Virtualize inside Pin VM See paper for details Commands Read / write registers, memory Set breakpoints Continue, single-step, stop Notifications Breakpoint triggered Caught signal Application exited Debugger PinADX core Pin

22. Software & Services Group set breakpoint at 4 continue Code CacheOriginal Code Breakpoint 1’ 2 3 1 4 5 6 breakpoint notification Execution stops in Pin Waits for GDB to continue BP 2’ 3’ PinADX core Debugger 22

23. Software & Services Group Code CacheOriginal Code Single Step 1’ 2 3 1 4 5 6 step complete notification Debugger Execution stops in Pin Waits for GDB to continue do single-step PinADX core 23

24. Software & Services Group Thanks Mark Charney – SDE software emulator Andria Pazarloglou – Created VS11 GUI plugin Gregg Miskelly – Microsoft VS11 debugger architect Robert Cohn – Father of Pin 24

25. Software & Services Group Summary DBI can implement powerful debugger features API allows Pin tools to extend debugger easily Multi-platform –Linux: GDB –Windows: Microsoft Visual Studio 11 (soon) Works with off-the-shelf debuggers 25 http://pintool.org