1. Matt Wolfe LC Development Environment Group Lawrence Livermore National Laboratory Lawrence Livermore National Laboratory, P. O. Box 808, Livermore, CA 94551 This work performed under the auspices of the U.S. Department of Energy by Lawrence Livermore National Laboratory under Contract DE-AC52-07NA27344 TotalView Multiprocess Debugger LLNL-PRES-426093

2. Lawrence Livermore National Laboratory TotalView Uses  Code correction  Code exploration and learning  Code statement runtime operation study  Memory utilization and debugging  Languages: C, C++, Fortran, UPC  Paradigms: serial, MPI, OpenMP, pthreads, CUDA (coming)

3. Lawrence Livermore National Laboratory TotalView Technologies' Tools  Workbench: tvworkbench  MemoryScape: memscape, memscript  TotalView: totalview, totalviewcli, tvscript, visualize  ReplayEngine: totalview -replay  Remote Display Client: remote_display_client.sh  contributed: colortv, mtv, mxterm, sxterm

4. Lawrence Livermore National Laboratory When To Use TotalView (TV)  Start code under TV  Attach to running code with TV  Post mortem core file inspection

5. Lawrence Livermore National Laboratory Preparing Code and Invoking TotalView  Compile: -g -O0  IBM: -qsaveopt OpenMP: -qsmp=omp:noopt  totalview -args  totalview  totalview code corefile

6. Lawrence Livermore National Laboratory TotalView Basic Tools  Visual and one-click access to greater detail  Breakpoints, Barrier points, TV intrinsics  Program control, Src/Asm single-stepping, restart, detach  Source, stack trace, stack frame (args, locals, registers)  View/change data, tooltips, Last Value, change highlighting  File/text/symbol search  Thread control, signal control

7. Lawrence Livermore National Laboratory TotalView Power Tools  Watchpoints, Conditional Breakpoints, Code Patching  Statistics, Filtering, Sorting, Visualization, Dump Format  Call Graph, Show Across, Custom Groups, Disassembler  Expression List, Evaluate, Type Transformation  Message Queue Display and Graph  Program Browser, Fortran Modules, Thread Objects  Memory Debugger, Replay Engine, Command Line Interface  tvscript

8. Lawrence Livermore National Laboratory Working at Scale  Reduce your process count if possible  Try STATGUI  Subset attach  Tools > Event Log  Work up to your target scale and be patient  Processes and Threads tabs  Sort by process state and collapse for counts  Filter out uninteresting libraries' symbols

9. Lawrence Livermore National Laboratory TVTech Resources  www.totalviewtech.com/support/videos.html  Help menu  F1 key  CLI help command  www.totalviewtech.com/support-other/release_notes.html  forum.totalviewtech.com (FAQs, Q&A wisdom)  Your local development tools support team, Tri-Labs issue tracker

10. Lawrence Livermore National Laboratory Afternoon Session  Hear use cases  How to see shared object dependencies: file, ldd, cat /proc/ /maps  How to trace system calls: man strace (x86-64); man truss (rs6000)  How to drop a useful core file: limit coredumpsize unlimited, ulimit -c unlimited, ^\  How to determine your TotalView version: -V  How to start with a clean slate: rm -f –r \ ~/.{memscape,preferences.tvd,totalview,tvworkbench} \ {~,.}/.tvdrc  How to get TV to display your source: $tree()

11. Lawrence Livermore National Laboratory Afternoon Session (Continued)  How to stop TV before the stack is unwound: ifort: for__issue_diagnostic  How to catch a Nan: CDWP(C, $double, 8 bytes): if ($is_nan($newval)) $stop  How to catch heap corruption  How to quickly return to a location of interest: $count  How to trace a variable (x86-64) printf("#"); fflush( (void *) 0);  How to display an array using an array of indices Ask Jeff about UPDD.

12. Lawrence Livermore National Laboratory Afternoon Session (Continued)  How to present memory leaks to a library provider  How to check instantaneous memory changes (x86-64) printf("\nTV node ID: %d, PID: %d\n",$nid,$pid); malloc_stats()  How to reconstruct an incomplete stack trace  How to find missing subroutine arguments and locals  Beginning to use the CLI: help  Discovering how to get TV to accept an expression