1. Embedded systems debugging 25 February 2015 Lecture a.kurek@samsung.com

2. Agenda 1.Makefiles 1.What is a makefile? 2.Syntax basics 3.Variables, automatic variables, dependency chains, phony targets 2.GDB 1.What is GDB? 2.Basic commands 3.Coredump analysis 4.GDBServer 5.GDB in Eclipse 3.Cross compilation

3. Makefiles

4. What is a makefile Basic compilation: g++ -c app.cpp -o app.o What if we have many files? g++ -c file1.cpp file2.cpp file3.cpp -o app.o Cons: Compiling everything, everytime; Doing everything „by hand”; No fast cleaning procedure; Makefiles

5. What is a makefile Makefile – text file written in a certain prescribed syntax, helps in building software, organizing code, compilation, linking and clearing the workspace. It can be understood as a recipe to acquiring predefined targets. Standard targets often contain: Build, rebuild, clean, configure, install; Makefiles

6. Syntax basics Makefiles

7. Variables CC=g++ main: main.o $(CC) main.o -o main main.o: main.cpp $(CC) -c main.cpp -o main.o Makefiles

8. Automatic variables Makefiles Available automatic variables: $@ - current target, everything on the left of the colon; $^ - all of the dependencies, everything on the right of the colon; $< - first file from the list of dependencies (on the right of the colon); $? - all of the dependencies which are out of date; $* - a call to the matched pattern, selected using %. Example: CC=g++ CFLAGS=-c -Wall main: main.o file1.o $(CC) $^ -o $@ %.o: %.cpp $(CC) $(CFLAGS) $< -o $@

9. Dependency chains Makefiles CC=g++ CFLAGS=-c -Wall all: hello hello: main.o factorial.o hello.o $(CC) main.o factorial.o hello.o -o hello main.o: main.cpp $(CC) $(CFLAGS) main.cpp factorial.o: factorial.cpp $(CC) $(CFLAGS) factorial.cpp hello.o: hello.cpp $(CC) $(CFLAGS) hello.cpp clean: rm -f *o hello

10. Phony targets Makefiles CC=g++ CFLAGS=-c -Wall all: hello hello: main.o factorial.o hello.o $(CC) main.o factorial.o hello.o -o hello main.o: main.cpp $(CC) $(CFLAGS) main.cpp factorial.o: factorial.cpp $(CC) $(CFLAGS) factorial.cpp hello.o: hello.cpp $(CC) $(CFLAGS) hello.cpp.PHONY:clean clean: rm -f *o hello

11. GDB

12. What is GDB? GDB GNU Debugger – standard debugger for GNU operating systems, works with Ada, C, C++, Fortran, Java, etc... Enables running application in a controlled environment, variable viewing/substitution, backtracing, core dump analysis, and more.

13. Debug symbols GDB gcc -g produces debugging information, so that GDB can work with it. You can also strip this information with gcc –s, or strip command, in order to reduce binary size. This, along with optimization flags, are often the two main differences between release and debug software versions, as seen from a toolchain level.

14. Optimization GDB Debugging a program built with –g and –O may produce unexpected results. Shortcuts taken by optimized code may contain: Some variables not existing at all; Flow of control moving to unexpected places; Some statements computation ommited, due to their constant state or values already at hand; Statements executing in other places, as they may have been moved out of loops;

15. Basic commands GDB gdb./app – run application under gdb control; Example application: #include int main(int argc, char* argv[]) { int test=5; void* buf = malloc(16); free(buf); printf("test %d", test); return 0; }

16. GDB Example application: #include int main(int argc, char* argv[]) { int test=5; void* buf = malloc(16); free(buf); printf("test %d", test); return 0; } [build@FedoraYouSee2 programming]$ gdb./main GNU gdb (GDB) Fedora (7.2-51.fc14) Copyright (C) 2010 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Type "show copying" and "show warranty" for details. This GDB was configured as "i686-redhat-linux-gnu". For bug reporting instructions, please see:... Reading symbols from /home/build/programming/main...done. (gdb) run Starting program: /home/build/programming/main *** glibc detected *** /home/build/programming/main: double free or corruption (fasttop): 0x0804a008 *** ======= Backtrace: ========= /lib/libc.so.6[0xaf7fb6] /home/build/programming/main[0x804845d] /lib/libc.so.6(__libc_start_main+0xe6)[0xa9fe36] /home/build/programming/main[0x8048391] ======= Memory map: ======== 00110000-00111000 r-xp 00000000 00:00 0 [vdso] 00a68000-00a85000 r-xp 00000000 fd:00 75 /lib/ld-2.13.so 00a85000-00a86000 r--p 0001c000 fd:00 75 /lib/ld-2.13.so 00a86000-00a87000 rw-p 0001d000 fd:00 75 /lib/ld-2.13.so 00a89000-00c0c000 r-xp 00000000 fd:00 151 /lib/libc-2.13.so 00c0c000-00c0d000 ---p 00183000 fd:00 151 /lib/libc-2.13.so 00c0d000-00c0f000 r--p 00183000 fd:00 151 /lib/libc-2.13.so 00c0f000-00c10000 rw-p 00185000 fd:00 151 /lib/libc-2.13.so 00c10000-00c13000 rw-p 00000000 00:00 0 00c70000-00c8c000 r-xp 00000000 fd:00 168 /lib/libgcc_s-4.5.1- 20100924.so.1 00c8c000-00c8d000 rw-p 0001b000 fd:00 168 /lib/libgcc_s-4.5.1- 20100924.so.1 08048000-08049000 r-xp 00000000 fd:00 271797 /home/build/programming/main 08049000-0804a000 rw-p 00000000 fd:00 271797 /home/build/programming/main 0804a000-0806b000 rw-p 00000000 00:00 0 [heap] b7fed000-b7fee000 rw-p 00000000 00:00 0 b7fff000-b8000000 rw-p 00000000 00:00 0 bffdf000-c0000000 rw-p 00000000 00:00 0 [stack] Program received signal SIGABRT, Aborted.

17. GDB Example application: #include int main(int argc, char* argv[]) { int test=5; void* buf = malloc(16); free(buf); printf("test %d", test); return 0; } Program received signal SIGABRT, Aborted. (gdb) backtrace #0 0x00110416 in __kernel_vsyscall () #1 0x00ab42f1 in raise () from /lib/libc.so.6 #2 0x00ab5d5e in abort () from /lib/libc.so.6 #3 0x00af051d in __libc_message () from /lib/libc.so.6 #4 0x00af7fb6 in _int_free () from /lib/libc.so.6 #5 0x0804845d in main (argc=1, argv=0xbffff464) at./main.cpp:9 (gdb) frame #0 0x00110416 in __kernel_vsyscall () (gdb) frame 5 #5 0x0804845d in main (argc=1, argv=0xbffff464) at./main.cpp:9 9free(buf); (gdb) list 4int main(int argc, char* argv[]) 5{ 6int test=5; 7void* buf = malloc(16); 8free(buf); 9free(buf); 10printf("test %d", test); 11return 0; 12} (gdb) print test $1 = 5 (gdb)

18. GDB l = list; b [L] = breakpoint [line]; r = run; bt = backtrace; p= print; n= next; s= step into; c= continue; q= quit; f= frame; gdb./main GNU gdb (GDB) Fedora (7.2-51.fc14) Copyright (C) 2010 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Type "show copying" and "show warranty" for details. This GDB was configured as "i686-redhat-linux-gnu". For bug reporting instructions, please see:... Reading symbols from /home/build/programming/main...done. (gdb) l 1#include 2#include 3 4int main(int argc, char* argv[]) 5{ 6int test=5; 7void* buf = malloc(16); 8free(buf); 9free(buf); 10printf("test %d", test); (gdb) b 7 Breakpoint 1 at 0x8048435: file./main.cpp, line 7. (gdb) r Starting program: /home/build/programming/main Breakpoint 1, main (argc=1, argv=0xbffff464) at./main.cpp:7 7void* buf = malloc(16); Missing separate debuginfos, use: debuginfo-install glibc-2.13-2.i686 (gdb) bt #0 main (argc=1, argv=0xbffff464) at./main.cpp:7 (gdb) p test $1 = 5 (gdb) n 8free(buf); (gdb) c What happens next?

19. Core dump analysis GDB What is it? Core dump/memory dump/system dump – recorded state of the working memory of a computer program at a speciffic time, usually when the program has abnormally terminated. Used to assist error diagnosis in programs, and analyze their specific state.

20. Core dump analysis GDB When is it useful? Hard to predict, occasional crashes; Inability to reproduce a bug with gdb running; Offline debugging – application/host can restart providing (relatively) continuous runtime; Inability to run gdb or connect via gdbserver; Dynamic memory allocation bugs tracked at the exact moment of faulty memory freeing.

21. Core dump analysis ulimit – user limits for system- wide resources. -S Change and report the soft limit associated with a resource. -H Change and report the hard limit associated with a resource. -a All current limits are reported. -c The maximum size of core files created. -d The maximum size of a process's data segment. -f The maximum size of files created by the shell(default option) -l The maximum size that may be locked into memory. -m The maximum resident set size. -n The maximum number of open file descriptors. -p The pipe buffer size. -s The maximum stack size. -t The maximum amount of cpu time in seconds. -u The maximum number of processes available to a single user. -v The maximum amount of virtual memory available to the process. [root@FedoraYouSee2 programming]#./main *** glibc detected ***./main: double free or corruption (fasttop): 0x08cc7008 *** ======= Backtrace: ========= /lib/libc.so.6[0xaf7fb6]./main[0x804845d] /lib/libc.so.6(__libc_start_main+0xe6)[0xa9fe36]./main[0x8048391] [...] Aborted (core dumped) [root@FedoraYouSee2 programming]# ls main main.cpp [root@FedoraYouSee2 programming]# ulimit -a core file size (blocks, -c) 0 data seg size (kbytes, -d) unlimited scheduling priority (-e) 0 file size (blocks, -f) unlimited pending signals (-i) 28134 max locked memory (kbytes, -l) 64 max memory size (kbytes, -m) unlimited open files (-n) 1024 pipe size (512 bytes, -p) 8 POSIX message queues (bytes, -q) 819200 real-time priority (-r) 0 stack size (kbytes, -s) 8192 cpu time (seconds, -t) unlimited max user processes (-u) 1024 virtual memory (kbytes, -v) unlimited file locks (-x) unlimited [root@FedoraYouSee2 programming]# ulimit -c unlimited [root@FedoraYouSee2 programming]#./main *** glibc detected ***./main: double free or corruption (fasttop): 0x08cc7008 *** ======= Backtrace: ========= /lib/libc.so.6[0xaf7fb6]./main[0x804845d] /lib/libc.so.6(__libc_start_main+0xe6)[0xa9fe36]./main[0x8048391] [...] Aborted (core dumped) [root@FedoraYouSee2 programming]# ls core.2877 main main.cpp

22. Core dump analysis gdb./app./core – analyze a core dump of a given application. [root@FedoraYouSee2 programming]# gdb./main./core.2877 [...] Reading symbols from /home/build/programming/main...done. [New Thread 2877] Reading symbols from /lib/libc.so.6...(no debugging symbols found)...done. Loaded symbols for /lib/libc.so.6 Reading symbols from /lib/ld-linux.so.2...(no debugging symbols found)...done. Loaded symbols for /lib/ld-linux.so.2 Reading symbols from /lib/libgcc_s.so.1...(no debugging symbols found)...done. Loaded symbols for /lib/libgcc_s.so.1 Core was generated by `./main'. Program terminated with signal 6, Aborted. #0 0x00dc0416 in __kernel_vsyscall () Missing separate debuginfos, use: debuginfo-install glibc-2.13-2.i686 libgcc- 4.5.1-4.fc14.i686 (gdb) bt #0 0x00dc0416 in __kernel_vsyscall () #1 0x00ab42f1 in raise () from /lib/libc.so.6 #2 0x00ab5d5e in abort () from /lib/libc.so.6 #3 0x00af051d in __libc_message () from /lib/libc.so.6 #4 0x00af7fb6 in _int_free () from /lib/libc.so.6 #5 0x0804845d in main (argc=1, argv=0xbfb32674) at./main.cpp:9 (gdb) f #0 0x00dc0416 in __kernel_vsyscall () (gdb) f 5 #5 0x0804845d in main (argc=1, argv=0xbfb32674) at./main.cpp:9 9free(buf); (gdb) l 4int main(int argc, char* argv[]) 5{ 6int test=5; 7void* buf = malloc(16); 8free(buf); 9free(buf); 10printf("test %d", test); 11return 0; 12} (gdb) p test $1 = 5

23. GDBServer GDB Makes it possible to remotely debug programs – no need to have sources and gdb on the target machine – be it a STB, TV, or a mobile phone. Connection can be established via serial lane/TCP. GDBServer has to be run on target machine: gdbserver :2159 hello_world GDB example, on other machine: gdb hello_world target remote 192.168.0.11:2159 Rest of the debugging process proceeds in an ordinary way.

24. GDB in Eclipse GDB GDB can be fully wrapped into GUI (simmilar to Visual Studio) using Eclipse IDE.

25. GDB in Eclipse GDB

26. GDB in Eclipse GDB

27. GDB in Eclipse GDB

28. Cross compilation

29. Cross compilation – what is it? Creating executable code for a platform other than one on which the cross compiler is running. Example: Compiling executable code for an Android smartphone from a Windows OS, using a cross compiler. Cross compilation

30. When is it useful? Embedded computers development – limited resources, often unable to run a compiler, have a file system, or a development environment – for example a microwave oven; Debugging and testing may require more resources than available on the target; Multiple target compilation – when creating an application for several OS, all of the versions can be build in one environment; New platform development – creating the minimal ammount of necessary tools, for example bootstrapping – writing a self-hosting compiler. Cross compilation

31. Other approaches Virtual machines (such as Java’s JVM) – allows the same compiler output to be used across multiple target systems. They are however often slower. Cross compilation

33. Thank You! a.kurek@samsung.com