1. 1 Debugging with the TotalView Source Code Debugger Ed Hinkel Sales Engineer TotalView Technologies MIT March 6, 2008

2. 2 Agenda TotalView Technologies Intro Source Code Debugging - Setup - Navigation - Data View and Analysis Memory Debugging Parallel Debugging Debugging Large Apps Questions / Comments

3. 3 TotalView Technologies Corporate Overview The Most Experienced Technologists in Parallel Debugging Technology originally developed at BBN in late 80’s Developed from scratch specifically for debugging parallel applications TotalView is recognized worldwide as the gold standard for debugging in multi-core, data intensive, high-performance, distributed, and clustered computing environments The debugging leader in the HPC, EDU, and Commercial sectors Founded as Etnus, Inc. in 1999, Renamed TotalView Technologies in 2007 50 employees (heavily engineering influenced) Over 1,400 customers in 55 countries Over 10K developers with over 2 million cores under license Award winning product line (Supercomputing Online's Product of the Year)

4. 4 What Is TotalView?

5. 5 What is TotalView? A comprehensive debugging solution for demanding multi-core applications C, C++, Fortran 77 & 90, UPC Wide compiler & platform support Multi-threaded Debugging Parallel Debugging MPI, PVM, Others Remote Debugging Memory Debugging Capabilities Integrated into the Debugger Powerful and Easy GUI Visualization CLI for Scripting

6. 6 Supported Compilers and Architectures Platform Support Linux x86, x86-64, ia64, Power Mac Power and Intel Solaris Sparc and AMD64 AIX, Tru64, IRIX, HP-UX ia64 Cray X1, XT3, XT4, IBM BGL, BGP, SiCortex Languages / Compilers C/C++, Fortran, UPC, Assembly Many Commercial & Open Source Compilers Parallel Environments MPI (MPICH1 & 2, LAM, Open MPI, poe, MPT, Quadrics, MVAPICH, & many others )‏ UPC

7. 7 Architecture for Cluster Debugging Single Front End (TotalView)‏ GUI debug engine Debugger Agents (tvdsvr)‏ Low overhead, 1 per node Traces multiple rank processes TotalView communicates directly with tvdsvrs Not using MPI Protocol optimization Compute Nodes Provides Robust, Scalable and efficient operation with Minimal Program Impact

8. 8 TotalView Basics _________________ Startup, Process Control & Navigation

9. 9 Starting TotalView Normal totalview [ tv_args ] prog_name [–a prog_args ] Attach to running program totalview [ tv_args ] prog_name –pid PID# [–a prog_args ] Attach to remote process totalview [ tv_args ] prog_name –remote name [–a prog_args ] Attach to a core file totalview [ tv_args ] prog_name corefile_name [ –a prog_args ] Command LineGUI

10. 10 Interface Concepts Root Window State of all processes being debugged Process Window Detailed state of a single process Thread within a process Point of control Control the process and possibly other related processes

11. 11 TotalView Root Window Host name Action Point ID number Expand - Collapse Toggle Process Status TotalView Thread ID # Rank # (if MPI program) Hierarchical/ Linear Toggle

12. 12 Process Window Overview Stack Trace PaneStack Frame Pane Source Pane Tabbed Area Toolbar

13. 13 Stack Trace and Stack Frame Panes LanguageNameFunction Pointer

14. 14 Source Code Pane

15. 15 Viewing Source Code TV always tries to display source code If it cannot you will see assembly -g puts ‘symbol table’ and ‘source code + line number’ info into your application These are references, usually by relative path from the object file to source file TV takes the basename and the path TotalView will first try to use this info to find the source file Then it will search a TV search path for the basename Paths can be set via $tree function CLI variables provides for setting source search paths - see documentation for details

16. 16 Debugging Assembly Code Display/Debug Source, Assembly or Both

17. 17 Process Status

18. 18 Stepping Commands Based on PC location

19. 19 Basic Process Control Control Group –All the processes created or attached together Share Group –All the processes that share the same image Workers Group –All the processes & threads that are not recognized as manager or service processes or threads Lockstep Group –All threads at the same PC Automatic Grouping

20. 20 Finding Functions, Variables, and Source Files Menu: View > Lookup ---------- Accelerator Keys: f, v ---------- “Closest Match” Search Results

21. 21 Action Points Breakpoints ---------- Barrier Points ---------- Conditional Breakpoints ---------- Evaluation Points ---------- Watchpoints

22. 22 Setting Breakpoints Setting action points Single left-click outlined source code line numbers Action Points Tab Lists all action points Dive on an action point to focus it in source pane Action point properties Context menu when right-clicking the action point Deleting action points Left-click in Source Pane Context menu in Source Pane / Action Points Tab Disabling action points Context menu Left-click in Action Points Tab Saving all action points Action Point > Save All

23. 23 Setting Breakpoints

24. 24 Conditional Breakpoint

25. 25 Evaluation Points Generalization of Conditional Breakpoints C/C++ or Fortran Call functions Set variables Test conditions Test small source code patches Help set up program circumstances

26. 26 Test Fixes on the Fly

27. 27 Watchpoints Use Tools > Watchpoint from a Variable Window. Watchpoints are set on a fixed memory region. When the contents of watched memory change, the watch- point is triggered and TotalView stops the program. Watchpoints are not set on a variable. You you need to be aware of the variable scope. Watchpoints can be conditional or unconditional Use intrinsic variables $newval and $oldval in the conditional expression

28. 28 Using Set PC to Replay Code

29. 29 Help System Context sensitive buttons on many dialog windows Help menu in the main windows Launches an html browser Navigate or search the full content Also available in pdf and hard copy Check out the tip of the week archive

30. 30 TotalView Documentation

31. 31 TotalView Basics _________________ Viewing and Editing Data

32. 32 Diving on Variables You can use Diving to: … get more information … to open a variable in a Variable Window. … to chase pointers in complex data structures. You can Dive on: … variable names to open a variable window … function names to open the source in the Process Window. … processes and threads in the Root Window. How do I Dive? Double-click the left mouse button on selection Single-click the middle mouse button on selection. Select Dive from context menu opened with the right mouse button

33. 33 Diving on Variables

34. 34 Undiving In a Process Window:retrace the path that has been explored with multiple dives. In a Variable Window: replace contents with the previous contents. You can also remove changes in the variable window with Edit > Reset Default.

35. 35 Dive in All Dive in All displays an element in an array of structures as if it were a simple array.

36. 36 The Variable Window Editing Variables Click once on the value Cursor switches into edit more Esc key cancels editing Enter key commits a change Editing values changes the memory of the program Window contents are updated automatically Changed values are highlighted “Last Value” column is available

37. 37 Expression List Window Add to the expression list using contextual menu with right-click on a variable, or by typing an expression directly in the window

38. 38 Expression List Window Reorder, delete, add Sort the expressions Edit expressions in place Dive to get more info Updated automatically Expression-based Simple values/expressions View just the values you want to monitor

39. 39 Four Ways to Look at Variables Glance Stack frame Hover Source pane Dive to data window Source, Stack or Variable Window Arrays, structures, explore Monitor via expression list Source, Stack or Variable window Keep an eye on scalars and expressions

40. 40 Viewing Arrays

41. 41 Slicing Arrays Slice notation is [start:end:stride]

42. 42 Filtering Arrays

43. 43 Visualizing Arrays Visualize array data using Tools > Visualize from the Variable Window Large arrays can be sliced down to a reasonable size first Visualize is a standalone program Data can be piped out to other visualization tools Visualize allows to spin, zoom, etc. Data is not updated with Variable Window; You must re-visualize $visualize() is a directive in the expression system, and can be used in evaluation point expressions.

44. 44 Typecasting Variables Edit the type of a variable Changes the way TotalView interprets the data in your program Does not change the data in your program Often used with pointers Type cast to a void or code type to snoop around in memory Give TotalView a starting memory address and TotalView will interpret and display your memory from that location.

45. 45 Type Casts Read from Right to Left Examples: int[10]* Pointer to an array of 10 int int*[10] Array of 10 pointers to int int*[10]* Pointer to an array of 10 pointers to int int[5]*[10] Array of 10 pointers to arrays of 5 int

46. 46 Typecasting Examples Castfloat * to float [100]* to see a dynamic array’s values Castto built-in types like $string to view a variable as a null-terminated string (automatic cast for char *) Cast to $void for no type interpretation or for displaying regions of memory Cast to $code[100] to see 100 instructions of disassembly Castto your own structs, objects, Fortran user defined types, common block definitions, etc.

47. 47 STLView STLView transforms templates into readable and understandable information –STLView supports std::vector, std::list, std::map, std::string –See doc for which STL implementations are supported

48. 48 TotalView understands your C++ templates and gives you a choice... Boxes with solid lines around line numbers indicate locations with replicated code C++ Templates

49. 49 Managing Signals File > Signals ErrorStop the process and flag as error StopStop the process ResendPass the signal to the target and do nothing: use with signal handlers IgnoreDiscard the signal

50. 50 TotalView Basics _________________ Memory Debugging

51. 51 What is a Memory Bug? A Memory Bug is a mistake in the management of heap memory Failure to check for error conditions Leaking: Failure to free memory Dangling references: Failure to clear pointers Memory Corruption Writing to memory not allocated Over running array bounds

52. 52 The Agent and Interposition Heap Interposition Agent (HIA)‏ Malloc API User Code and Libraries Allocation Table Deallocatio n Table Process TotalView

53. 53 TotalView HIA Technology Advantages of TotalView HIA Technology Use it with your existing builds No Source Code or Binary Instrumentation Programs run nearly full speed Low performance overhead Efficient memory usage Low memory overhead Support wide range of platforms and compilers

54. 54 Memory Debugger Features Automatically detect allocation problems View the heap Leak detection Block painting Dangling pointers Deallocation/reallocation notification Guard Blocks Memory Comparisons between processes Collaboration features

55. 55 Heap Graphical View

56. 56 Heap Graphical View

57. 57 Leak Detection Based on Conservative Garbage Collection Can be performed at any point in runtime Helps localize leaks in time Multiple Reports Backtrace Report Source Code Structure Graphically Memory Location

58. 58 Leak Detection

59. 59 Guard Blocks & Memory Corruption

60. 60 Guard Blocks & Memory Corruption

61. 61 Memory Comparisons “Diff” live processes Compare processes across cluster Compare with baseline See changes between point A and point B Compare with saved session Provides memory usage change from last run

62. 62 Memory Status

63. 63 What is MemoryScape? Streamlined Lightweight Intuitive Collaborative Memory Debugging Features Shows Memory Errors Memory Status Memory Leaks Bounds Violations MPI Memory Debugging Remote Memory Debugging Tech Low Overhead No Instrumentation Interface Inductive Collaboration Multi-process MemoryScape

64. 64 Script Mode Automation Support MemoryScape lets users run tests and check programs for memory leaks without having to be in front of the program Simple command line program called memscript Doesn’t start up the GUI Can be run from within a script or test harness The user defines What configuration options are active What things MemoryScape is looking for What actions MemoryScape should take for each type of event that may occur

65. 65 TotalView Basics _________________ Parallel Application Debugging

66. 66 Challenges of Debugging in a Multi-Core Age Concurrency Stochastic errors are often many times harder to solve than others Achieving reproducibility of race conditions, deadlocks, live-locks, and other concurrent bugs is the key Precise thread-level control of all the processes in the distributed application If you can’t control the threads then you are simply hoping that the problem happens If you can’t reproduce the problem you can’t easily pose questions about why Constructs to enable problem reproduction Scripting Expression Evaluation Visibility into all the relevant data Thread specific stacks and thread-private variables Easy ways to view complex data

67. 67 Debugging Multithreaded Programs When debugging multithreaded programs, you want to: Know where to look to get thread status. Be able to switch the focus from one thread to another quickly and easily. Understand how asynchronous thread control commands (step, go, halt) and breakpoints are used. A parallel program has a lot more states than ‘Running or stopped’ There are more degrees of freedom for program control. TotalView gives you a full set of features to manage this complexity. It is important to understand how the different classes of commands and features work so as to avoid confusion.

68. 68 TotalView Provides MPI-Aware Easy launch mechanisms Seamless Parallel and Remote Debugging Powerful Process Control Features MPI Message Queue Display High Degree of Scalability Scriptability for unattended operation

69. 69 Preparing for debugging Compiling Your Application Provide TotalView with debug symbols Add '-g' to your compile line Turn off optimizations Remove any optimization flags ‘-o’

70. 70 Starting an MPI Job Within TotalView Indirect launch Choose MPI implementation Set parameters Enable Memory Debugging Indicate your MPI Start from command line mpirun -tv -np 4 my_program (mpich) totalview poe -a -np 4 my_program

71. 71 Running TotalView with SiCortex Applications The TotalView Debugger runs as a cross-debugger within the SiCortex-MIPS Linux environment. The SiCortex version is a 64-bit application runs on a x86- 64 system running a 64-bit kernel. Debugging on SiCortex uses the remote features of TotalView

72. 72 Running TotalView with SiCortex Applications TVD needs to execute a command on the target system from the development host. By default, this version uses the ssh -x command. It is suggested to use ssh set so that allows password-less commands. The program’s executable file must be visible from both the development host and the target system. Place the executables in a directory that is visible on both machines through the same path. Having the executable visible in separate directories that are accessed through the same path on both machines will also work.

73. 73 Running TotalView with SiCortex Applications The SiCortex version of TotalView uses a different set of naming conventions, using an ”sc” prefix sctv8 instead of tv8 sctototalview instead of totalview sctv8cli vs. tv8cli for the Command Line I/F

74. 74 Running TotalView with SiCortex Applications TVD must debug the MIPS version of srun, not the x86-64 version of srun. TotalView can be Invoked as follows: sctv8 -r SiCortex_node./srun -a srun_arguments Via the GUI: Use the File > New Program dialog box. Within the Parallel tab, select SiCortex from the pull- down list. (This is the preferred way to start MPI programs from within TVD.)

75. 75 Root Window with MPI Status Info T = stopped B = Breakpoint E = Error W = Watchpoint R = Running M = Mixed Navigation Dive to refocus Dive anew to get a second process window

76. 76 Call Graph Quick view of program state Each call stack is a path Functions are nodes Calls are edges Labeld with the MPI rank Construct process groups Look for outliers

77. 77 Process Control Concepts Each process window is always focused on a specific process. Process focus can be easily switched Processes can be ‘held’ - they will not run till unheld. Breakpoints can be set to stop the process or the group Breakpoint and command scope can be simply controlled

78. 78 Switching Processes You can switch the focus of the process window by using the P+ and P- buttons on the toolbar. –P+ takes you to the next process. –P- takes you to the previous process. You can also navigate directly to processes by diving on process entries in the root window. –The next slide describes using the root window in more detail.

79. 79 Process control commands have a 'scope’. For MPI debugging the following scopes are interesting: Control Group Scope: the entire MPI job, including starter (if there is a separate starter) Share Group Scope: all the rank processes (if you are focused on a rank process) Process Scope: the process that the Process Window is focused on. Arbitrary sets of processes can be controlled by defining a process group with the command line interface. See later slides and Chapter 11 of the users manual Nothing MPI specific about process control Process Control with MPI

80. 80 Looking at Variables across Processes TotalView allows you to look at the value of a variable in all MPI processes Right Click on the variable Select the View > View Across TotalView creates an array indexed by process You can filter and visualize

81. 81 View MPI Message Queues Information visible whenever MPI rank processes are halted Provides information from the MPI layer Unexpected messages Pending Sends Pending Receives Use this info to debug Deadlock situations Load balancing

82. 82 Message Queue Graph Hangs & Deadlocks Pending Messages Receives Sends Unexpected Inspect Individual entries Patterns

83. 83 Message Queue Debugging Filtering Tags MPI Communicators Cycle detection Find deadlocks

84. 84 Large Jobs: Subset Attach You can be attached to different subsets at different times through the run You can attach to a subset, run till you see trouble and then 'fan out' to look at more processes if necessary. This greatly reduces overhead TotalView does not need to be attached to the entire job

85. 85 Large Jobs: Strategies Reduce N Problem: Each process added requires overhead Strategy: Reduce the number of processes TotalView is attached to Simply reducing N is best, however data or algorithm may require large N Technique: subset attach mechanism Focus Effort Problem: Some debugger operations are much more intensive than others, when multiplied by N this is a big deal Strategy: Reduce the interaction between the debugger and the processes Technique: Use TotalView's process control features to Avoid single stepping Focus on one or a small set of processes

86. 86 Large Jobs: Focus on One Process If you want to single step through a section of code Perhaps do it on one process only… Set a process-width breakpoint at the beginning Once everything you want/expect is lined up, hold one process Do a group-width ‘go’ to get all the other processes running Unhold that one process and change the width of the commands in the tool bar to process Next and step that one process The other processes are running so they will participate in communication with your process of interest

87. 87 Thanks! QUESTIONS?

88. 88