1. © 2013 MontaVista Software, LLC. MontaVista Confidential and Proprietary. CGE7 Flight Recoder Demo Nawneet Anand

2. © 2013 MontaVista Software, LLC. MontaVista Confidential and Proprietary. Agenda Introduction to Flight Recorder Purpose of Flight Recorder Execution flow Diagram Flight recorder set up requirement Trace analysis DR7 Lttng kernel trace Viewer Advantage References 2

3. © 2013 MontaVista Software, LLC. MontaVista Confidential and Proprietary. Introduction to flight recoder Flight recorder –Flight Recorder is a tool which allows tracing of LTT events on running system to a circular buffer. – It has ability to extract that buffer from a kernel core dump in addition to normal LTT operation. The default set of events that are logged in flight recorder mode are: – syscall –entry/exit, –trap entry/exit, –Irq entry/exit, –schedule change, –kernel timer, Soft Irq and processes etc 3

4. © 2013 MontaVista Software, LLC. MontaVista Confidential and Proprietary. Purpose of Feature The feature is used to help diagnose the cause of a system crash. What was happening that lead up to the crash. It is referred to as a "flight recorder" because it needs to be able to survive the kernel crash. Record or retain history of scheduling and other activity (known as "parameters") for some finite amount of time prior to a crash. Lttng kernel probes are used to define which records will be saved. Default probes are located in kernel/ltt/probes/ directory. 4

5. © 2013 MontaVista Software, LLC. MontaVista Confidential and Proprietary. High Level Design Flight Recorder uses standard LTT to record traces. The crash utility modified to support extracting Flight Recorder data from a crash dump. It stores the data in a standard LTT file format for analysis by the standard LTT tools or Devrocket. 5

6. © 2013 MontaVista Software, LLC. MontaVista Confidential and Proprietary. Execution Flow Diagram 6 Initial kernel Loaded with crash parameter=128M@32M to reserve space for capture kernel Lttng trace started in snapshot mode with e trace all trace points enabled Preload the capture kernel with Kexec tool Is kernel crashed Continuue with Initial kernal NO YES Load Capture Kernel Copy crash dump in Vcore Reboot the system Start Lttng- Gdb tool View Trace in trace Viewer

7. © 2013 MontaVista Software, LLC. MontaVista Confidential and Proprietary. Flight Recorder Setup requirement Initial kernel should be booted with crashkernel space reserved. Ideally it should be crashkernal=128M@32M. The dump kernel should be loaded by kexec so that after the crash, capture kernel should boot with the crash kernel content. LTTng trace should be started on snapshot mode with all the trace events enabled. After the crash copy the crash dump from the booted captured kernel and reboot the system and start the LTTng GDB tool. 7

8. © 2013 MontaVista Software, LLC. MontaVista Confidential and Proprietary. Trace Analysis Example: kernel is trying to access Null pointer and kernel crashed. 1. Reproduce the problem with crash kernel space reserved and capture kernel is loaded. Note :for demo we are crashing the kernel by./crash_kernel script.This script also includes the starting of trace on snapshot mode. In Normal execution we don’t need to crash the kernel manually. 2. Once kernel will crash execute and capture kernel is booted,execute the script./copy_dump./lttng-gdb Enter following commands on gdb shell: source /usr/share/gdb/macros/ltt-kdump/ltt.py dumpltt /lib/modules/version/kernel/lttng-module/lttng-modules-2.3.4 trace-out. 3.Babletarce trace-out >trace 8

9. © 2013 MontaVista Software, LLC. MontaVista Confidential and Proprietary. Contd… 9

10. © 2013 MontaVista Software, LLC. MontaVista Confidential and Proprietary. Contd… Check the swapper function in trace.it gives the information what was the last executed process and what next running process with priority Check the kernel API/open file/system call executed.it will give the information on what all are the API’s executed for a process. Check the log messages and find out the crash point. In example we have NULL pointer deference message. After this we have sysrq_handle_crash message. Conclusion: From the shell through system call sys_open we opened a file sysrq-trigger that has NULL pointer dereference which leads to system crash. 10

11. © 2013 MontaVista Software, LLC. MontaVista Confidential and Proprietary. DR7 Lttng kernel trace Viewer Start the Devrocket 7 and create new tracing project for ex:trace. Right click the project trace on left hand menu and open batch import. Import the generated trace-out folder and start the trace viewing. 11

12. © 2013 MontaVista Software, LLC. MontaVista Confidential and Proprietary. Contd.. 12

13. © 2013 MontaVista Software, LLC. MontaVista Confidential and Proprietary. Contd.. 13

14. © 2013 MontaVista Software, LLC. MontaVista Confidential and Proprietary. Contd.. It’s a pictorial representation of trace with enhanced functionality. Between two time stamps we can see how many process executed. It gives the information about the amount of time process executed i.e which process taken how much time. We can see the details of process as well as process switching. It gives the detail information about a specific time stamp what process and with event is running and what are the contents with that event type 14

15. © 2013 MontaVista Software, LLC. MontaVista Confidential and Proprietary. DR7 Lttng kernel trace Viewer Advantage Easy to Use. More clarity because of GUI based trace. Easy to track problems such as watchdog, system lock, long delay of some process execution System performance issue analysis in specific case. 15

16. © 2013 MontaVista Software, LLC. MontaVista Confidential and Proprietary. Advantage of Flight Recorder Engineer don’t have required the target Need only two things Vmlinux and vcore file with these files problem can be analyzed. Easy to use in the field. Save the crash dump in case of crash with the use of capture kernel on the fly. Easy to go back on normal kernel execution. 16

17. © 2013 MontaVista Software, LLC. MontaVista Confidential and Proprietary. References http://lse.sourceforge.net/kdump/docu mentation/ols2oo5-kdump-paper.pdfhttp://lse.sourceforge.net/kdump/docu mentation/ols2oo5-kdump-paper.pdf http://quadyen.caveonetworks.com/foswiki /bin/view/Main/SDK3TechSpecFlightRecor derhttp://quadyen.caveonetworks.com/foswiki /bin/view/Main/SDK3TechSpecFlightRecor der 17

18. © 2013 MontaVista Software, LLC. MontaVista Confidential and Proprietary. 18 Demo

19. © 2013 MontaVista Software, LLC. MontaVista Confidential and Proprietary. 19 Thank you