1. Ladebug Kernel Debugging Tutorial Bob Lidral

2. Introduction Kinds of kernel debugging How to use Ladebug for kernel debugging Not how to debug a kernel

3. Agenda Preparing a Kernel for Debugging Local Kernel Debugging – Standalone Ladebug – Ladebug With kdbx Crash Dump Analysis – Standalone Ladebug – Ladebug with kdbx Remote Kernel Debugging – Without Gateway System – With Gateway System

4. Overview Who – System Engineers – System Administrators When and Why – Kernel Development and Debugging – Crash Analysis and Troubleshooting – System Configuration and Tuning

5. Preparing a Kernel for Debugging (if possible) Not always possible if debugging a running kernel or a crash dump Compile without full optimization Do not strip symbol table information

6. Local Kernel Debugging (Standalone) Debug running kernel on same node as Ladebug session Must have superuser (root) permissions Enter command: # ladebug -k /vmunix /dev/mem Some Ladebug commands are not available in this mode (e.g., cont, next, rerun, run, step, and stop )

7. Local Kernel Debugging (with kdbx) Debug running kernel on same node as kdbx session Same restrictions as for standalone Ladebug Enter command: # kdbx –dbx /bin/ladebug \ –k /vmunix /dev/mem Supports all valid Ladebug commands plus some kdbx command extensions

8. Crash Dump Analysis (Standalone) Determine cause of a crash or panic May not need superuser (root) permissions Crash dump files are created in /var/adm/crash Named vmunix.n and vmcore.n where n is the version number and must match Enter command: # ladebug -k vmunix.1 vmcore.1

9. Crash Dump Analysis (with kdbx) Determine cause of a crash or panic Same restrictions as for standalone Ladebug Enter command: # kdbx –dbx /bin/ladebug \ -k vmunix.1 vmcore.1 Supports all valid Ladebug commands plus some kdbx extensions

10. Remote Kernel Debugging Run kernel on a remote node under control of Ladebug running on local node Supports breakpoints in kernel code Requires serial line physical connection between Test System and either Build or Gateway System – Specific communication port (usually COM1) on Test System (see documentation) – Communication port may be configurable on Build or Gateway System

11. Remote Kernel Debugging Terminology Build System – system on which Ladebug executes Test System – system on which kernel being debugged executes Gateway System (optional) – between Build System and Test System, connected to – Test System by physical serial line – Build System over network

12. Remote Kernel Debugging System Requirements Test System – Tru64 UNIX Version 2.0 or higher Verify: % uname –a – Kernel Debugging Tools subset Verify: % setld -i | grep -i kernel | \ grep installed – Kernel Breakpoint Debugger kernel option Verify: % /sbin/sysconfig -s kdebug

13. Remote Kernel Debugging System Requirements Build System – Tru64 UNIX Version 3.2 or higher – Copy of Test System kernel executable image – Copy of kernel source code (optional, recommended) Gateway System – Tru64 UNIX Version 2.0 or higher – Kernel Debugging Tools subset Verify by same methods as for Test System

14. Serial Line BC16E Cable H8571-J DECconnect Passive Adapter

15. Remote Kernel Debugging Without Gateway System Specify Build System communication port to use for kdebug by appropriate entry in /etc/remote : – Default (COM1): kdebug:dv=/dev/tty00:br#9600:pa=none: – Override in order to use as any of Build, Gateway, or Test System (COM2): kdebug:dv=/dev/tty01:br#9600:pa=none: Note: the label “ kdebug ” must match the value of the Ladebug debugger variable $kdebug_line

16. kernel COM1 COM2 kdebug Test System Remote Kernel Debugging Without Gateway System kdebugd COM1 COM2 Ladebug Build System Serial Line

17. Remote Kernel Debugging With Console Serial Line Connect second serial line – Must be connected to COM1 communication port on Test System – Configurable communication port on Build or Gateway System Test System: redirect console input and output at console prompt >>> set console serial

18. Remote Kernel Debugging With Console Serial Line Build (or Gateway) System: add entry to /etc/remote for console serial line tstsys:dv=/dev/tty00:br#9600:pa=none: Build (or Gateway) System: create a new window for the Test System console and establish connection % tip tstsys

19. kernel COM1 COM2 kdebug Test System Remote Kernel Debugging With Console Serial Line kdebugd COM1 COM2 Ladebug Build System kdebug Serial Line Console Serial Line

20. Remote Kernel Debugging With Console Serial Line Restore Test System console communication to its console: >>> set console graphics Exit from tip session: ~.

21. Remote Kernel Debugging With Gateway System Gateway System: specify communication port for kdebug by appropriate entry in /etc/remote : – Default (COM1): kdebug:dv=/dev/tty00:br#9600:pa=none: – Override in order to use as any of Build, Gateway, or Test System (COM2): kdebug:dv=/dev/tty01:br#9600:pa=none:

22. Remote Kernel Debugging With Gateway System Gateway System: open window to display kdebugd/kdebug communication traffic: – execute tty command to get pathname of device corresponding to this window – Use this pathname for the value of the Ladebug debugger variable $kdebug_dbgtty on the Build System

23. Remote Kernel Debugging With Gateway System Build System: set Ladebug debugger variables describing connections: – $kdebug_host – node name of Gateway System (default: localhost => no Gateway System) – $kdebug_line – label of /etc/remote entry on Gateway System for kdebug communication port (default: kdebug ) – $kdebug_dbgtty – terminal window on Gateway System used to display kdebug communication traffic (output of tty command in that window) (default: null => no Gateway System)

24. kernel COM1 COM2 kdebug Test System Remote Kernel Debugging With Gateway System kdebugd Ladebug Build System kdebugd COM1 COM2 Gateway System Serial Line Network

25. Remote Kernel Debugging Startup On Build System % ladebug –remote./vmunix.test (ladebug) stop in ttyretype (ladebug) stop in panic On Test System – If not at console prompt ( >>> ) # shutdown –h now

26. Remote Kernel Debugging Startup Then, in either order – On Build System (ladebug) run – On Test System >>> boot –flags k Once started, typing ^R on the Test System will trigger the ttyretype breakpoint. There is no way for Ladebug to gain control except at a breakpoint or other interruption of the kernel.

27. Where to Get More Information Ladebug Debugger Manual – Kernel Debugging Local Kernel Debugging Crash Dump Analysis – Remote Kernel Debugging with the kdebug Debugger Compaq Tru64 Unix Reference Page – ladebug(1)

28. Other Compaq Tru64 Unix Manuals Kernel Debugging – crashdc – kdbx – kdebug System Administration System Configuration and Tuning

29. Other Compaq Tru64 Unix Reference Pages crashdc(8) ikdebug(8) kdbx(8)