1. Linux Development Tools
Kevin Dankwardt, Ph.D.
K Computing

2. Overview
Tools helpful to developers using C when writing a network based application ...
other languages may sometimes benefit, too
Free and Open Source tools only
Quick look at a number of tools plus info on how to dig deeper
We demo and dig deeper with a few tools
Objectives:
let you know about the existence and use of a wide variety of tools
save you time in finding out more

3. Tutorial Files and Notes
To get a copy of these tutorial notes, the examples used, updated slides, and more info:

4. gcc C compiler and “front-end” to the tool chain
Be sure to look at the man (about 185 pages) and info (about 527 pages) pages
Has about 1911 command line options man gcc |col -b |tr “ “ “\n” |tr “\t” “\n”|sort|uniq|grep -e “^-”|wc -l
Some tools (profiling, debugging, code coverage, ...) require special options when compiling and/or linking.
Many tools provide inaccurate results if optimization is used
Version matters. Read the README with software.

5. Toolchains Matrix from www.kegel.com/crosstool
Toolchains don't all work

6. gdb the GNU debugger. ubiquitous info page is about 444 pages long
debug live (step through, breakpoints, ...) and dead (core dump) processes
debug remote targets with gdbserver
attach to running processes (gdb -p <pid>)
compile with the -g flag
multithreaded debugging
ulimit -c unlimited # to turn on core dumps

7. debuginfo RPMs Often needed for stack traces (backtrace) in gdb
yum -y --enablerepo=core-debuginfo install glibc-debuginfo.i686
Often needed for stack traces (backtrace) in gdb
yum install $(ldd /usr/bin/whatever | cut -d ' ' -f 3 | sort | uniq | xargs rpm -qf --qf '%{SOURCERPM}\n' | sort | uniq | sed -e 's/-[0-9].*$/-debuginfo/g')
See:
from:
Program received signal SIGSEGV, Segmentation fault.
0x004db27a in _IO_vfscanf_internal () from /lib/libc.so.6
(gdb) bt
#0 0x004db27a in _IO_vfscanf_internal () from /lib/libc.so.6
#1 0x004e01fb in scanf () from /lib/libc.so.6
#2 0x in get_num () at error.c:7
#3 0x b in main () at error.c:22

8. DDD GNU's Data Display Debugger
Front-end to other debuggers, such as GDB
Click on variables in code to display, click on fields in display to draw a linked list ...

9. insight Red Hat/Cygnus project – GUI debugger integrated into GDB
Install from tarball

10. gprof GNU preformance profiler Build with -pg and -g flags
run the program
gprof -l <program>
Line Number

11. Profiling Profiling is finding "hotspots" in one's code
Code often exhibits the 80/20 rule - 80% of the time is spent in 20% of the code
Profiling helps one to find which 20%
Profiling is normally done after a program is performing correctly
Profiling often times incurs significant overhead

12. oprofile Need vmlinux file (install kernel source and build)
Profiles kernel and user space
Can use hardware performance counters
commands: opcontrol, opreport, opannotate, opgprof

13. LTTng Linux Trace Tool – Next Generation
Trace kernel and user space tracing
Low overhead: usable in ISR and even in NMIs
Precise enough for real-time measuring
GUI viewer – LTTV
Need to patch your kernel

14. systemtap
Write scripts that extract data from the kernel to help diagnose performance or functional problems.
Script is translated into a kernel loadable module
Can trace kernel functions by name, system calls, instructions.
Can set timers
probe for function entry
probe for function exits.

15. gcov GNU tool Provides code coverage reports
Don't optimize code when compiling
Annotates source with the count for the number of times each line was executed.

16. Mpatrol
Dynamically linked library that can be dynamically reconfigured to log dynamic memory operations.
Memory allocations listed by size
The function that allocated them.
List of memory leaks
Redefines a large set of memory related functions, not just malloc/free.

17. valgrind Program for debugging and profiling programs. Works on X86
Language independent - runs on the executable. Has been used with C, C++, Java, Perl, Python, x86 assembly, Fortan and others.
Can start GDB and attach to the program when it detects a problem.
GUI front-ends are available.
Helps with memory related bugs.
==15308== Invalid read of size 4
==15308== at 0x : main (lotsa_errs.c:15)
==15308== Address 0x3C01F024 is 0 bytes inside a block of size 4000 free'd
==15308== at 0x3C01B739: free (vg_replace_malloc.c:127)
==15308== by 0x804846D: main (lotsa_errs.c:14)

18. Dmalloc library
debug library to replace malloc(), realloc(), calloc(), free() and other memory allocation functions.
Helps to detect memory leaks and other problems.
No man page, use info page.
Include dmalloc.h
link -ldmalloc
Set environment variable DMALLOC_OPTIONS before running the program

19. Using Dmalloc
Put the following function definition in ~/.bashrc and “source” it.
function dmalloc { eval `command dmalloc -b $*`; }
Run dmalloc function with options like “-l logfile -i 100 high”. This sets the environment variable.
Run the program, the file “logfile” will contain the report.
$ dmalloc -l logfile -i 100 high
$ gcc -g bad_stuff.c -ldmalloc -o bad_stuff
$ ./bad_stuff
$ cat logfile

20. Electric Fence Provides support for finding code that
references beyond the bounds of malloc'd buffers
or references within freed buffers
Provides both read and write checking
Makes use of VM hardware support
Ordinarily a part of Fedora Core
Uses hardware (MMU) to detect problems so it can be fast.
May allocate many extra pages so can consume a lot of memory.

21. Using ElectricFence
Compile program with debugging and ElectricFence library
cc -g prog.c libefence.a -o prog -lpthread
Run program
If program segfaults - i.e., ends with "Segmentation Fault" you can use a debugger to find the line that caused the fault
Alternatively, you can set up a signal hander for SIGSEGV or run the program from within a debugger
There may also be an “ef” program.

22. strace Trace system calls
Shows parameter list and return value (with errno code when there's an error)
Can show timing information
Great for helping to debug applications that do a poor job with error messages.
Can trace a process that is already running

23. strace output stace -e open date 2>&1 | grep \lib
open("/lib/librt.so.1", O_RDONLY) = 3
open("/lib/libc.so.6", O_RDONLY) = 3
open("/lib/libpthread.so.0", O_RDONLY) = 3
open("/usr/lib/locale/locale-archive", O_RDONLY|O_LARGEFILE) = 3

24. ltrace Trace library calls Can also trace system calls
Show the system calls that a library call makes
Trace a script to trace shell builtins

25. vmstat Report system information about:
processes, memory, paging, block IO, traps, and cpu activity
procs memory swap io system--
r b swpd free buff cache si so bi bo in cs
-----cpu------
us sy id wa st

26. /proc/diskstats An example proc file #reads
#reads/writes merged with adjacent
#sectors read
#writes completed
#sectors written
#milliseconds spent writing
#I/Os currently in progress
and more ...

27. iptraf Network statistics
TCP connections, packet/byte counts, traffic breakdown
Activity of other hosts

28. Wireshark (Ethereal) Packet Sniffer
yum install wireshark and gnome-wireshark

29. netstat Network statistics Connections Routing Interface statistics
Ports being listened on

30. free Amount of free and used memory in the system
total used free shared buffers cached
Mem:
-/+ buffers/cache:
Swap:

31. frysk
fcatch
fcore
fhpd
fparser
fstack
fstep
ftrace

32. pstack pstack output of top command process
Print stack trace of running process
pstack output of top command process
#0 0x00bb2402 in __kernel_vsyscall ()
#1 0x00ce909d in ___newselect_nocancel () from /lib/libc.so.6
#2 0x in ?? ()
#3 0x00c38f2c in __libc_start_main () from /lib/libc.so.6
#4 0x in ?? ()

33. ctags/etags
Provides a mechanism to locate the definition of C function declarations
ctags file_list
etags file_list

34. vim with tags Syntax Highlighting :syntax on :syntax off Tags support
vim -t tag_name - switch to a tag
:tag tag_name - switch to a tag
Control-] - switch to tag under cursor
Control-T - return to previous location
:tn - switch to next occurrence of a tag
:ts - list all occurrences of a tag

35. cscope Interactive tool for browsing C sourcecode
Locate functions, function calls, macros, variables, preprocessor symbols
Functions called and called by
Search for extended regular expressions

36. splint Secure Programming Lint Static C program checker A better Lint
Unused declarations, type inconsistencies, etc.
Can get greater checking by adding annotations to the source.
Annotations about: whether a pointer can be null, mutability, abstract types and many more

37. doxygen Documentation generator
Extracts information from source file comments
Output can be HTML, RTF, PDF, LaTex, Postscript or man pages

38. Summary