Presentation is loading. Please wait.

Presentation is loading. Please wait.

© 2010 InHand Electronics, Inc. – www.inhand.com Troubleshooting Real-Time Software Issues Using a Logic Analyzer Dave Stewart Director of Software Engineering.

Similar presentations


Presentation on theme: "© 2010 InHand Electronics, Inc. – www.inhand.com Troubleshooting Real-Time Software Issues Using a Logic Analyzer Dave Stewart Director of Software Engineering."— Presentation transcript:

1 © 2010 InHand Electronics, Inc. – Troubleshooting Real-Time Software Issues Using a Logic Analyzer Dave Stewart Director of Software Engineering InHand Electronics Embedded Systems Conference Silicon Valley 2010 Class ESC 364/366

2 Dave Stewart, ESC Silicon Valley – April 2010 Troubleshooting Real-Time Issues using a Logic Analyzer © 2010 InHand Electronics, Inc. – 2 Troubleshooting Real Time Software Issues using a Logic Analyzer Introduction Print Debug Macros Logic Analyzer Debug Macros Tricks and Techniques for Real-Time Debugging Measuring Execution Time and Real-Time Performance Summary

3 Dave Stewart, ESC Silicon Valley – April 2010 Troubleshooting Real-Time Issues using a Logic Analyzer © 2010 InHand Electronics, Inc. – 3 What are The Tough Bugs to Debug in Real-Time Systems? Glitches Timing and Synchronization Problems Driver Errors Misbehaving Interrupts Memory Corruption Priority Inversion Performance Issues Hardware Errors

4 Dave Stewart, ESC Silicon Valley – April 2010 Troubleshooting Real-Time Issues using a Logic Analyzer © 2010 InHand Electronics, Inc. – 4 Limitations of Traditional Debugging Print Statements There is no console, so Print Statements dont work Print Statements are too slow thus provide insufficient information Print function significantly affects real-time performance Writing debug output to a serial port changes the timing too much Adding print statements changes program behavior Cant measure performance at a fine granularity Max 50 to 100 print statements per second The code crashes, but there is insufficient feedback as to where Symbolic Debuggers (e.g. IDEs, via JTAG, ActiveSync, or other comm link) A symbolic debugger or emulator is not available Stepping through the code makes the program behave differently Breakpoints will break real-time performance There is real I/O, it doesnt work Debugger doesnt deal with interrupts properly There might be a race condition or other synchronization problem

5 Dave Stewart, ESC Silicon Valley – April 2010 Troubleshooting Real-Time Issues using a Logic Analyzer © 2010 InHand Electronics, Inc. – 5 Solution: Use a Logic Analyzer for Real-Time Issues Most detailed view of your code Microsecond view of software Easily 50,000 debug data points per second Info is time-stamped for timing assessment Real-Time Can use it for interrupts and I/O drivers Impact on real-time execution is negligible Identify temporal relationships among tasks Monitor interrupts and how they may affect execution Watch variables without changing execution time Anomalies Spot anomalies and different patterns of execution Obtain sufficient proof that a problem is hardware, not software Fine-grain timing measurements to identify performance culprits

6 Dave Stewart, ESC Silicon Valley – April 2010 Troubleshooting Real-Time Issues using a Logic Analyzer © 2010 InHand Electronics, Inc. – 6 Logic Analyzer for Debugging Should I use it all the time? Using a Logic Analyzer is NOT easy Use Print Statements and Symbolic Debuggers to solve easy and non-real-time problems first Add to your repertoire of available tools to solve hard problems Solve functional problems using print statements. If necessary, run the functions on the desktop, and debug them there. Only move to embedded environment when it is working well. Use Symbolic Debuggers mainly for Tracing through code that fails in a consistent manner Post-Mortem debugging of crashes, to view all variables

7 Dave Stewart, ESC Silicon Valley – April 2010 Troubleshooting Real-Time Issues using a Logic Analyzer © 2010 InHand Electronics, Inc. – 7 Logic Analyzer Features Preferably at least 8 channels 16 to 24 channels is very nice to have 2 to 7 channels still useable, but it prevents use of some techniques Large memory depth in M-Samples, not K-Samples Multiple Views Timing Diagrams State Listings Decoding of Serial Protocols Useful features: Search, Filtering, and Triggering High-Speed Interface to a PC Built-in PC, USB Memory Stick, Ethernet all OK Common logic analyzer features that are NOT needed: High Speed. Dont need GHz. Very slow analyzers, e.g. 10MHz, are OK. Many channels. Dont need 64 or 128 channel analyzers. Low cost USB Logic Analyzer Pods are OK A $25,000 logic analyzer is nice to have and will have many extras that could be useful Examples in this presentation use the Tektronix TLA700 What is discussed in this talk can be accomplished with a $400 USB pod. Examples in this presentation use the Intronix LogicPort

8 Dave Stewart, ESC Silicon Valley – April 2010 Troubleshooting Real-Time Issues using a Logic Analyzer © 2010 InHand Electronics, Inc. – 8 Sample Logic Analyzer Output Timing Diagram This output format will be used for most examples in this class. More details interpreting this diagram to follow.

9 Dave Stewart, ESC Silicon Valley – April 2010 Troubleshooting Real-Time Issues using a Logic Analyzer © 2010 InHand Electronics, Inc. – 9 Sample Logic Analyzer Output State Listing counttrigmode16modedatatime-relativetime-absolute

10 Dave Stewart, ESC Silicon Valley – April 2010 Troubleshooting Real-Time Issues using a Logic Analyzer © 2010 InHand Electronics, Inc. – 10 Target Setup: Digital Outputs Why so many bits? 8 bits Can output full 8-bit number 9 to 15 bits: Output 8 bits + additional bits to specify mode, or module For software triggers 16 bits: Can output full 16-bit number 17 or more bits: Output 16 bits + additional bits to specify mode, or module For software triggers

11 Dave Stewart, ESC Silicon Valley – April 2010 Troubleshooting Real-Time Issues using a Logic Analyzer © 2010 InHand Electronics, Inc. – 11 Additional Signals Access to communication signals, such as UART, I 2 C, SSP, SPI, I 2 S, etc. Access to trigger keys or keypad signals Specify debug ports when a system is designed, to ensure they are available when it is time to debug and test the system. If a software engineer doesnt ask for it, dont expect a hardware engineer to include it. Easy to hook-up connectors for the logic analyzer probes Target Setup: Desirable

12 Dave Stewart, ESC Silicon Valley – April 2010 Troubleshooting Real-Time Issues using a Logic Analyzer © 2010 InHand Electronics, Inc. – 12 Target Setup: Interconnections Logic Analyzer (Assuming 8-bit channels) System Under Test Px.0-Px.7Ch.1 Px.8-Px.15Ch.2 RX/TX Ch.3 I2C SPI Keys This is an example setup given 24 bits. With fewer bits, connect whatever you can.

13 Dave Stewart, ESC Silicon Valley – April 2010 Troubleshooting Real-Time Issues using a Logic Analyzer © 2010 InHand Electronics, Inc. – 13 An Aside Print Statement Debugging Forms the basis for logic analyzer methods

14 Dave Stewart, ESC Silicon Valley – April 2010 Troubleshooting Real-Time Issues using a Logic Analyzer © 2010 InHand Electronics, Inc. – 14 An Aside Print Statement Debugging myfunc() { code here printf(I got here\n); more code here printf(Going to call yourfunc()\n); result = yourfunc(); printf(My result is %d\n,result); etc } I got here Going to call yourfunc() My result is 384

15 Dave Stewart, ESC Silicon Valley – April 2010 Troubleshooting Real-Time Issues using a Logic Analyzer © 2010 InHand Electronics, Inc. – 15 Print Statement Debugging: Problems A Lot of typing Minimal information per statement Hard to separate between debug and normal print statements Prone to errors Cannot easily disable them Ultimately very inefficient

16 Dave Stewart, ESC Silicon Valley – April 2010 Troubleshooting Real-Time Issues using a Logic Analyzer © 2010 InHand Electronics, Inc. – 16 Debug Macros instead of Print Statements DEBUG_WHERE() #define DEBUG_PRINT fprintf(stderr, #define DEBUG_WHERE() \ DEBUG_PRINT \ [%s:%u-%s]\n,\ __FILE__,__LINE__,__FUNCTION__) Complete.h file for all macros given in this class are in the paper that accompanies this presentation on the conference website or CD. Note: Not all compilers support __FUNCTION__.

17 Dave Stewart, ESC Silicon Valley – April 2010 Troubleshooting Real-Time Issues using a Logic Analyzer © 2010 InHand Electronics, Inc. – 17 Print Statement Debugging myfunc() { code here DEBUG_WHERE(); more code here DEBUG_WHERE(); result = yourfunc(); DEBUG_INT(result); etc } [myfile.c:3-myfunc] [myfile.c:5-myfunc] [myfile.c:7-myfunc] result=384

18 Dave Stewart, ESC Silicon Valley – April 2010 Troubleshooting Real-Time Issues using a Logic Analyzer © 2010 InHand Electronics, Inc. – 18 Debug Macros instead of Print Statements DEBUG_INT() #define DEBUG_INT(_var) \ DEBUG_PRINT \ [%s:%u-%s] %s=%d\n,\ __FILE__,__LINE__,__FUNCTION__ \ #_var,_var) result = 384 DEBUG_INT(result) [myfile.c:7-myfunc] result=384

19 Dave Stewart, ESC Silicon Valley – April 2010 Troubleshooting Real-Time Issues using a Logic Analyzer © 2010 InHand Electronics, Inc. – 19 Debug Macros instead of Print Statements DEBUG_STRING() #define DEBUG_STRING(_var) \ DEBUG_PRINT \ [%s:%u-%s] %s=%s\n, \ __FILE__,__LINE__,__FUNCTION__ \ #_var,_var) char hello[] = Hello World; DEBUG_STRING(hello); [myfile.c:7-myfunc] hello=Hello World

20 Dave Stewart, ESC Silicon Valley – April 2010 Troubleshooting Real-Time Issues using a Logic Analyzer © 2010 InHand Electronics, Inc. – 20 Debug Macros instead of Print Statements DEBUG_MESSAGE() #define DEBUG_MESSAGE(_var) \ DEBUG_PRINT \ [%s:%u-%s] %s\n, \ __FILE__,__LINE__,__FUNCTION__,\ _var) DEBUG_MESSAGE(Start of Data Dump); [myfile.c:7-myfunc] Start of Data Dump

21 Dave Stewart, ESC Silicon Valley – April 2010 Troubleshooting Real-Time Issues using a Logic Analyzer © 2010 InHand Electronics, Inc. – 21 Debug Macros instead of Print Statements DEBUG_HEX8() #define DEBUG_HEX8(_var) \ DEBUG_PRINT \ [%s:%u-%s] %s=0x%02X\n,\ __FILE__,__LINE__,__FUNCTION__ \ #_var,_var) result = 0xA4 DEBUG_HEX8(result) [myfile.c:7-myfunc] result=0xA4

22 Dave Stewart, ESC Silicon Valley – April 2010 Troubleshooting Real-Time Issues using a Logic Analyzer © 2010 InHand Electronics, Inc. – 22 Debug Macros instead of Print Statements DEBUG_HEX16() #define DEBUG_HEX16(_var) \ DEBUG_PRINT \ [%s:%u-%s] %s=0x%04X\n,\ __FILE__,__LINE__,__FUNCTION__ \ #_var,_var) result = 0x52A4 DEBUG_HEX16(result) [myfile.c:7-myfunc] result=0x52A4

23 Dave Stewart, ESC Silicon Valley – April 2010 Troubleshooting Real-Time Issues using a Logic Analyzer © 2010 InHand Electronics, Inc. – 23 Debug Macros instead of Print Statements DEBUG_HEX32() #define DEBUG_HEX32(_var) \ DEBUG_PRINT \ [%s:%u-%s] %s=0x%08X\n,\ __FILE__,__LINE__,__FUNCTION__ \ #_var,_var) result = 0x52A4419C DEBUG_HEX32(result) [myfile.c:7-myfunc] result=0x52A4419C

24 Dave Stewart, ESC Silicon Valley – April 2010 Troubleshooting Real-Time Issues using a Logic Analyzer © 2010 InHand Electronics, Inc. – 24 Debug Macros instead of Print Statements Optionally add Time Stamp #define DEBUG_HEX32(_var) \ DEBUG_PRINT \ %s=0x%08X\n,\ __FILE__,__LINE__,__FUNCTION__,clock() \ #_var,_var) result = 0x52A4419C DEBUG_HEX32(result) result=0x52A4419C

25 Dave Stewart, ESC Silicon Valley – April 2010 Troubleshooting Real-Time Issues using a Logic Analyzer © 2010 InHand Electronics, Inc. – 25 Debug Macros instead of Print Statements Optionally add Process ID #define DEBUG_HEX32(_var) \ DEBUG_PRINT \ %s=0x%08X\n,\ getpid(),\ __FILE__,__LINE__,__FUNCTION__,clock() \ #_var,_var) result = 0x52A4419C DEBUG_HEX32(result) result=0x52A4419C

26 Dave Stewart, ESC Silicon Valley – April 2010 Troubleshooting Real-Time Issues using a Logic Analyzer © 2010 InHand Electronics, Inc. – 26 Debug Macros for Logic Analyzers Similar to debugging with Print Statements, Except: (disadvantage) Much more difficult to use Learning curve could be days, not minutes (but its worth it for the hard bugs that could otherwise take weeks to debug) (advantage) 1000 times more information Print Statement: 50 Lines/Data Points per second Logic Analyzer: Data Points per second is easy (advantage) Real-time view of the system Data points are time-stamped with microsecond resolution

27 Dave Stewart, ESC Silicon Valley – April 2010 Troubleshooting Real-Time Issues using a Logic Analyzer © 2010 InHand Electronics, Inc. – 27 Basics of Logic Analyzer Debugging bart = 0x0063; lisa = 0x1018; homer= 0xF04E; marge= 0x00E5; PDEBUG_HEX8(0x40); PDEBUG_HEX16(bart); PDEBUG_HEX8(0x41); PDEBUG_HEX16(lisa); PDEBUG_HEX8(0x42); PDEBUG_HEX16(homer); PDEBUG_HEX8(0x42); PDEBUG_HEX16(marge); See next pages for zoom of this timing diagram. This example assumes only a single 8-bit port is available.

28 Dave Stewart, ESC Silicon Valley – April 2010 Troubleshooting Real-Time Issues using a Logic Analyzer © 2010 InHand Electronics, Inc. – 28 Basics of Logic Analyzer Debugging Rectangle shows portion of image shown in next slide.

29 Dave Stewart, ESC Silicon Valley – April 2010 Troubleshooting Real-Time Issues using a Logic Analyzer © 2010 InHand Electronics, Inc. – 29 Basics of Logic Analyzer Debugging PDEBUG_HEX8(0x40);bart = 0x0063; PDEBUG_HEX16(bart); Timing Diagram of each bit. Well see later the value of these for creating code patterns.

30 Dave Stewart, ESC Silicon Valley – April 2010 Troubleshooting Real-Time Issues using a Logic Analyzer © 2010 InHand Electronics, Inc. – 30 Basics of Logic Analyzer Debugging Each code represents a different variable. Correlate with source code. PDEBUG_HEX8(0x40); PDEBUG_HEX16(bart); PDEBUG_HEX8(0x41); PDEBUG_HEX16(lisa); PDEBUG_HEX8(0x42); PDEBUG_HEX16(homer); PDEBUG_HEX8(0x42); PDEBUG_HEX16(marge);

31 Dave Stewart, ESC Silicon Valley – April 2010 Troubleshooting Real-Time Issues using a Logic Analyzer © 2010 InHand Electronics, Inc. – 31 Basics of Logic Analyzer Debugging Variable data displayed in sequence. bartlisahomermarge PDEBUG_HEX8(0x40); PDEBUG_HEX16(bart); PDEBUG_HEX8(0x41); PDEBUG_HEX16(lisa); PDEBUG_HEX8(0x42); PDEBUG_HEX16(homer); PDEBUG_HEX8(0x42); PDEBUG_HEX16(marge);

32 Dave Stewart, ESC Silicon Valley – April 2010 Troubleshooting Real-Time Issues using a Logic Analyzer © 2010 InHand Electronics, Inc. – 32 Basics of Logic Analyzer Debugging Timescale is Microseconds. Easily add debug statements with minimal intrusion on real-time code. In contrast, each print statement takes 2+ milliseconds. 1 usec

33 Dave Stewart, ESC Silicon Valley – April 2010 Troubleshooting Real-Time Issues using a Logic Analyzer © 2010 InHand Electronics, Inc. – 33 PDEBUG_HEX8() Macro Initialization (e.g. MSP430) // Example, init MSP430 Port 4 // Simple memory-mapped port uint8_t *pdebug_dout8 = (uint8_t *) 0x001D; uint8_t *pdebug_dir8 = (uint8_t *) 0x001E; *pdebug_dir8 = 0xFF;// Initialize to output

34 Dave Stewart, ESC Silicon Valley – April 2010 Troubleshooting Real-Time Issues using a Logic Analyzer © 2010 InHand Electronics, Inc. – 34 PDEBUG_HEX8() Macro (E.g. MSP430) #define PDEBUG_HEX8(_val) \ *pdebug_dout8 = (uint8_t)(_val) More details and more macros are provided in the paper that accompanies this presentation on the conference website or CD. bart = 0x0063; PDEBUG_HEX8(0x40); PDEBUG_HEX16(bart);

35 Dave Stewart, ESC Silicon Valley – April 2010 Troubleshooting Real-Time Issues using a Logic Analyzer © 2010 InHand Electronics, Inc. – 35 PDEBUG_HEX16() Macro If only 8 bits available #define PDEBUG_HEX16(_val) \ PDEBUG_HEX8( (_val) >> 8);\ PDEBUG_HEX8( (_val) ) bart = 0x0063; PDEBUG_HEX8(0x40); PDEBUG_HEX16(bart);

36 Dave Stewart, ESC Silicon Valley – April 2010 Troubleshooting Real-Time Issues using a Logic Analyzer © 2010 InHand Electronics, Inc. – 36 PDEBUG_VAR16() Shortcut bart = 0x0063; PDEBUG_HEX8(0x40); PDEBUG_HEX16(bart); bart = 0x0063; PDEBUG_VAR16(0x40,bart); #define PDEBUG_VAR16(_code,_val) \ PDEBUG_HEX8( _code);\ PDEBUG_HEX16(_val )

37 Dave Stewart, ESC Silicon Valley – April 2010 Troubleshooting Real-Time Issues using a Logic Analyzer © 2010 InHand Electronics, Inc. – 37 Test your Macros Example to test PDEBUG_HEX8() Always test your macros before using them. PDEBUG_HEX8(0xFF); PDEBUG_HEX8(0x01); PDEBUG_HEX8(0x02); PDEBUG_HEX8(0x04); PDEBUG_HEX8(0x08); PDEBUG_HEX8(0x10); PDEBUG_HEX8(0x20); PDEBUG_HEX8(0x40); PDEBUG_HEX8(0x80); PDEBUG_HEX8(0x55); PDEBUG_HEX8(0xAA);

38 Dave Stewart, ESC Silicon Valley – April 2010 Troubleshooting Real-Time Issues using a Logic Analyzer © 2010 InHand Electronics, Inc. – 38 Test your Macros Example to test PDEBUG_HEX8() Allows you to test logic analyzer setup too. If color analyzer, adjust colors to your liking too.

39 Dave Stewart, ESC Silicon Valley – April 2010 Troubleshooting Real-Time Issues using a Logic Analyzer © 2010 InHand Electronics, Inc. – 39 PDEBUG_HEX32() Macro If only 8 bits available #define PDEBUG_HEX32(_val) \ PDEBUG_HEX8( (_val) >> 24);\ PDEBUG_HEX8( (_val) >> 16);\ PDEBUG_HEX8( (_val) >> 8);\ PDEBUG_HEX8( (_val) )

40 Dave Stewart, ESC Silicon Valley – April 2010 Troubleshooting Real-Time Issues using a Logic Analyzer © 2010 InHand Electronics, Inc. – 40 PDEBUG_HEX32() Macro If only 8 bits available mickey = 0x0009B900; donald = 0x0000CFB8; PDEBUG_HEX8(0x44); PDEBUG_HEX32(mickey); PDEBUG_HEX8(0x45); PDEBUG_HEX32(donald);

41 Dave Stewart, ESC Silicon Valley – April 2010 Troubleshooting Real-Time Issues using a Logic Analyzer © 2010 InHand Electronics, Inc. – 41 PDEBUG_HEX8() Macro (PXA270) // Example, PXA270, use LCD data bits LDD6..LDD13 // 32-bit architecture, uses GPIO2 // Separate registers to set/clear bits uint32_t *pdebug_set8 = (uint32_t *) 0x40E00020; uint32_t *pdebug_clr8 = (uint32_t *) 0x40E0002C; uint32_t *pdebug_dir8 = (uint32_t *) 0x40E00014; *pdebug_dir8 |= 0x000000FF;// Init to output

42 Dave Stewart, ESC Silicon Valley – April 2010 Troubleshooting Real-Time Issues using a Logic Analyzer © 2010 InHand Electronics, Inc. – 42 PDEBUG_HEX8() Macro Initialization (PXA270) #define PDEBUG_HEX8(_hex8) \ { uint32_t h32 = (uint32_t) hex8;\ uint32_t sg2,cg2;\ \ sg2= ( h32 & 0x000000FF); // set bits\ cg2= ((~h32) & 0x000000FF); // clear bits\ *_gpsr2 = sg2;\ *_gpcr2 = cg2;}

43 Dave Stewart, ESC Silicon Valley – April 2010 Troubleshooting Real-Time Issues using a Logic Analyzer © 2010 InHand Electronics, Inc. – 43 Logic Analyzer Debug Macros There are many possibilities for defining macros Use macros to enable a common interface across platforms For each new platform, define the macros in a hardware-dependent manner, and place in a.h file Simply include a different.h file for each platform Additional macro examples Paper that accompanies this presentation includes additional macro examples not covered here.

44 Dave Stewart, ESC Silicon Valley – April 2010 Troubleshooting Real-Time Issues using a Logic Analyzer © 2010 InHand Electronics, Inc. – 44 Tricks and Techniques for Real-Time Debugging Tracing Drivers, Low-Level Code, and Real-time Code Conditional Debugging Timing and Synchronization Fine-Grained Code Optimization Performance Anomalies Memory and Data Corruption Interrupts and OS Overhead Less-than-ideal Setup Setup Extras

45 Dave Stewart, ESC Silicon Valley – April 2010 Troubleshooting Real-Time Issues using a Logic Analyzer © 2010 InHand Electronics, Inc. – 45 Tracing Drivers, Low-Level Code, and Real-time Code Recall prior example using Print Statements myfunc() { code here DEBUG_WHERE(); more code here DEBUG_WHERE(); result = yourfunc(); DEBUG_INT(result); etc } [myfile.c:3-myfunc] [myfile.c:5-myfunc] [myfile.c:7-myfunc] result = 384

46 Dave Stewart, ESC Silicon Valley – April 2010 Troubleshooting Real-Time Issues using a Logic Analyzer © 2010 InHand Electronics, Inc. – 46 Tracing Drivers, Low-Level Code, and Real-time Code #define PDEBUG_WHERE() \ PDEBUG_HEX8(0x11);\ PDEBUG_HEX8( (__LINE__) >> 8);\ PDEBUG_HEX8( (__LINE__) ) #define PDEBUG_INT(code,_val) \ PDEBUG_WHERE();\ PDEBUG_HEX16(_val);

47 Dave Stewart, ESC Silicon Valley – April 2010 Troubleshooting Real-Time Issues using a Logic Analyzer © 2010 InHand Electronics, Inc. – 47 Tracing Drivers, Low-Level Code, and Real-time Code myfunc() { code here PDEBUG_WHERE(); more code here PDEBUG_WHERE(); result = yourfunc(); PDEBUG_INT(result); etc } 384 = 0x180 Line Numbers in Hex just a code to spot line numbers The output is cryptic compared to using print statements. Hence, continue using print statements if you can. But use the logic analyzer to trace code when print statements dont work. (Some analyzers will print decimal!)

48 Dave Stewart, ESC Silicon Valley – April 2010 Troubleshooting Real-Time Issues using a Logic Analyzer © 2010 InHand Electronics, Inc. – 48 Memory and Data Corruption Watch a memory location that is changing unexpectedly uint8_t *pdebug_memptr;// Global uint8_t pdebug_memval;// Global #define PDEBUG_WATCH(addr) \ *pdebug_memptr = (uint8_t *)(addr);\ pdebug_memval = *(pdebug_memptr) #define PDEBUG_CHECK() \ if (*pdebug_memptr != pdebug_memval) {\ PDEBUG_STOP(); }

49 Dave Stewart, ESC Silicon Valley – April 2010 Troubleshooting Real-Time Issues using a Logic Analyzer © 2010 InHand Electronics, Inc. – 49 Memory and Data Corruption Watch a memory location that is changing unexpectedly Scatter PDEBUG_CHECK throughout code. First, at beginning and end of function to narrow down which function is changing it. Then between each line of code or function call to determine which line is changing it. If the line is in another function call, repeat. This method will pinpoint the line of code causing the memory corruption.

50 Dave Stewart, ESC Silicon Valley – April 2010 Troubleshooting Real-Time Issues using a Logic Analyzer © 2010 InHand Electronics, Inc. – 50 PDEBUG_HALT() Macro Use to trigger the logic analyzer. If 9 th bit is available, use that. Define PDEBUG_TRIG() similar to PDEBUG_HEX8(). Otherwise, setup logic analyzer to trigger after a specific sequence, like #define PDEBUG_HALT(_val) \ PDEBUG_WHERE();\ PDEBUG_HEX8( 0xEE);\ PDEBUG_HEX8( 0xEF);\ while (1); // clearly show on analyzer the end Set logic analyzer to trigger at 100%. This means the trigger is the end-point of data collection. Data up to the end-point is saved.

51 Dave Stewart, ESC Silicon Valley – April 2010 Troubleshooting Real-Time Issues using a Logic Analyzer © 2010 InHand Electronics, Inc. – 51 Fine-Grained Code Optimization ALWAYS measure execution time of a code segment before and after optimization. If an optimization breaks a pattern that the compiler recognizes, it could in fact make things take longer. Not all optimizations are obvious. What is faster, pointers or index arrays? What is faster, case statements or if-then else? What is faster, for loops or while loops? Measure to find out. Use PDEBUG Macros to measure code segments PDEBUG_START() before the code segment PDEBUG_STOP() after the code segment

52 Dave Stewart, ESC Silicon Valley – April 2010 Troubleshooting Real-Time Issues using a Logic Analyzer © 2010 InHand Electronics, Inc. – 52 Performance Anomalies Quite often, performance is not as expected Why Not? Scatter debug statements throughout code Start and stop of each function and thread is a good starting point. Often, just keep all the prior debug statements in place, they may provide the data needed. Run the code, and monitor on the logic analyzer. Look for patterns of behavior. Identify patterns that use too much CPU. Verify what is happening when analyzer shows periods of inactivity. Monitor execution time of interrupts. Some examples on next few slides.

53 Dave Stewart, ESC Silicon Valley – April 2010 Troubleshooting Real-Time Issues using a Logic Analyzer © 2010 InHand Electronics, Inc. – 53 Performance Anomalies Green is commmunication. We can clearly see communication stopped for a period of time. Why? Is that normal?

54 Dave Stewart, ESC Silicon Valley – April 2010 Troubleshooting Real-Time Issues using a Logic Analyzer © 2010 InHand Electronics, Inc. – 54 Performance Anomalies See the Pattern? Four short communication packets, one long one, then none for what seems like two or three system cycles.

55 Dave Stewart, ESC Silicon Valley – April 2010 Troubleshooting Real-Time Issues using a Logic Analyzer © 2010 InHand Electronics, Inc. – 55 Performance Anomalies What line of code executes immediately after PDEBUG_HEX8(0x9F)? Whatever it is, it takes a LONG time. If there is a performance issue, focus on that code first.

56 Dave Stewart, ESC Silicon Valley – April 2010 Troubleshooting Real-Time Issues using a Logic Analyzer © 2010 InHand Electronics, Inc. – 56 Performance Anomalies This is an example of code executed with PDEBUG_WHERE() plus a few other PDEBUG statements between every two lines of code. What is happening on Line 51 (0x33)?

57 Dave Stewart, ESC Silicon Valley – April 2010 Troubleshooting Real-Time Issues using a Logic Analyzer © 2010 InHand Electronics, Inc. – 57 Interrupts and OS Overhead Even without source code for the OS, these can be measured Create a high-priority task that simply does the following: while (1) { PDEBUG_HEX8(0x00); PDEBUG_HEX8(0x01); } This will toggle one bit continuously. Whenever this task gets interrupted, the toggling will stop.

58 Dave Stewart, ESC Silicon Valley – April 2010 Troubleshooting Real-Time Issues using a Logic Analyzer © 2010 InHand Electronics, Inc. – 58 Interrupts and OS Overhead Device Driver Calls If OS source code or device driver is available (e.g. stream driver, aka open/read/write/close): Add debug macros inside these wrappers. E.g. int read(int fd, void *buffer, int nbytes) { PDEBUG_HEX8(0x30 | fd); dev_driver->read(xxx) PDEBUG_HEX8(0x40 | fd); } Use codes 0x10 and 0x20 for open Use codes 0x50 and 0x60 for write etc.

59 Dave Stewart, ESC Silicon Valley – April 2010 Troubleshooting Real-Time Issues using a Logic Analyzer © 2010 InHand Electronics, Inc. – 59 Interrupts and OS Overhead If OS source code is NOT available: Create your own functions for any OS function. E.g.: int myread(int fd, void *buf, int nbytes) { PDEBUG_HEX8(0x30 | fd); rv = read(fd,buf,nbytes); PDEBUG_HEX8(0x40 | fd); return rv; } Replace read with myread for any driver you want to measure.

60 Dave Stewart, ESC Silicon Valley – April 2010 Troubleshooting Real-Time Issues using a Logic Analyzer © 2010 InHand Electronics, Inc. – 60 Less-than Ideal Setup Scrounging bits from anywhere PXA270, we had following bits available, by leveraging the Camera Interface bits: GPIO Port2: 84, 85, 93, 94 GPIO Port3: 116,106,107,108 Remember PXA270 has separate set/clear registers. Code became too complex for macro, hence: #define PDEBUG_HEX8(val) \ pdebugHex8(val) Code takes much longer to execute 5 usec instead of 0.5 usec Still very useable with minimal intrusion on system

61 Dave Stewart, ESC Silicon Valley – April 2010 Troubleshooting Real-Time Issues using a Logic Analyzer © 2010 InHand Electronics, Inc. – 61 PXA270 Bit Scrounging pdebugHex8(uint8_t hex8) { uint32_t h32 = (uint32_t) hex8; uint32_t gr2,gr3,ngr2,ngr3; // don't write to registers yet; we want to write to registers // as quickly as possible, to minimize race conditions. gr2=(((h32 & 0x0C)<<(29-2)|((h32 & 0x03)<<(20-0)))); // D3.D2 | D1.D0 gr3=(((h32 & 0xE0)<<(10-5)|((h32 & 0x10)<<(20-4)))); // D7.D6.D5 | D4 h32 = ~h32; ngr2=(((h32 & 0x0C)<<(29-2)|((h32 & 0x03)<<(20-0)))); // D3.D2 | D1.D0 ngr3=(((h32 & 0xE0)<<(10-5)|((h32 & 0x10)<<(20-4)))); // D7.D6.D5 | D4 *_gpcr2 = ngr2; *_gpsr2 = gr2; *_gpcr3 = ngr3; *_gpsr3 = gr3; }

62 Dave Stewart, ESC Silicon Valley – April 2010 Troubleshooting Real-Time Issues using a Logic Analyzer © 2010 InHand Electronics, Inc. – 62 pdebugHex8 Initialization static volatile uint32_t *_gpdr2 = (uint32_t *)0x40E00014;// GPIO 2 Direction static volatile uint32_t *_gpsr2 = (uint32_t *)0x40E00020;// GPIO 2 Set static volatile uint32_t *_gpcr2 = (uint32_t *)0x40E0002C;// GPIO 2 Clear static volatile uint32_t *_gpdr3 = (uint32_t *)0x40E0010C;// GPIO 3 Direction static volatile uint32_t *_gpsr3 = (uint32_t *)0x40E00118;// GPIO 3 Set static volatile uint32_t *_gpcr3 = (uint32_t *)0x40E00124;// GPIO 3 Clear // set directions #define _MASK2 0x #define _MASK3 0x00101C00 *_gpdr2 |= _MASK2; *_gpdr3 |= _MASK3;

63 Dave Stewart, ESC Silicon Valley – April 2010 Troubleshooting Real-Time Issues using a Logic Analyzer © 2010 InHand Electronics, Inc. – 63 Setup Extras Example More than 16 Bits PDEBUG_HEX16() macro becomes a single operation No need to split as two 8-bit signals. Create two separate PDEBUG_HEX8 E.g. call second one PDEBUG_MODULE() Upon entry or exit to any function in a module, call this one. Use PDEBUG_HEX8 to trace and print variable data. Makes it easier to correlate logic analyzer output to source code Reserve some bits as triggers A lot easier to setup logic analyzer to trigger on an individual bit, rather than a sequence of bytes.

64 Dave Stewart, ESC Silicon Valley – April 2010 Troubleshooting Real-Time Issues using a Logic Analyzer © 2010 InHand Electronics, Inc. – 64 Setup Extras Example Keys and Scope Monitor key presses They often serve as trigger points Ensure that if debouncing is needed, it is properly handled Oscilloscope Analog measurements of power consumption Correlate power usage with specific code segments Aids in software design for low power systems

65 Dave Stewart, ESC Silicon Valley – April 2010 Troubleshooting Real-Time Issues using a Logic Analyzer © 2010 InHand Electronics, Inc. – 65 Setup Extras Example Logic Analyzer Screen Captures Follow Next few pages show a logic analyzer setup with: PDEBUG_HEX8(): 8-bit debug port (yellow) PDEBUG_MODE(): 3-bit mode (green) PDEBUG_TRIG(): 1-bit trigger (magenta) 2 keys on keypad (cyan) serial TX/RX (grey) instantaneous current consumption (green analog) input voltage (red). Each page is a zoom of the prior page. The zoom area is shown by the magenta rectangle.

66 Dave Stewart, ESC Silicon Valley – April 2010 Troubleshooting Real-Time Issues using a Logic Analyzer © 2010 InHand Electronics, Inc. – 66 Setup Extras Example zoom

67 Dave Stewart, ESC Silicon Valley – April 2010 Troubleshooting Real-Time Issues using a Logic Analyzer © 2010 InHand Electronics, Inc. – 67 Setup Extras Example zoom

68 Dave Stewart, ESC Silicon Valley – April 2010 Troubleshooting Real-Time Issues using a Logic Analyzer © 2010 InHand Electronics, Inc. – 68 Setup Extras Example zoom

69 Dave Stewart, ESC Silicon Valley – April 2010 Troubleshooting Real-Time Issues using a Logic Analyzer © 2010 InHand Electronics, Inc. – 69 Setup Extras Example

70 Dave Stewart, ESC Silicon Valley – April 2010 Troubleshooting Real-Time Issues using a Logic Analyzer © 2010 InHand Electronics, Inc. – 70 Setup Extras Example Information shown in prior slides is simply unavailable when using other debug methods Order of magnitude faster to pinpoint problems Problems that would take days to find now take an hour Glitches that would take months to pinpoint now found in a day Problems found by observing anomalies often represent bugs before even testing can catch it times more data per second collected Data is visually presented in timing graphs Export listings to create your own scripts to search or analyze data

71 Dave Stewart, ESC Silicon Valley – April 2010 Troubleshooting Real-Time Issues using a Logic Analyzer © 2010 InHand Electronics, Inc. – 71 Setup Extras Monitoring Serial Transmissions Serial Communication Protocol Analyzers exist to monitor integrity of data. Built-in to some logic analyzers In many cases, you dont need anything that complex. Rather, you just want to know how many bytes, and when. Use PDEBUG_HEX8() macros to print the bytes being transmitted.

72 Dave Stewart, ESC Silicon Valley – April 2010 Troubleshooting Real-Time Issues using a Logic Analyzer © 2010 InHand Electronics, Inc. – 72 Serial Communication E.g. I 2 C CLK Data The code that immediately follows the I 2 C transfer can then be traced line by line. Some Analyzers will convert clock/data to HEX values and validate bus protocols. (Example later)

73 Dave Stewart, ESC Silicon Valley – April 2010 Troubleshooting Real-Time Issues using a Logic Analyzer © 2010 InHand Electronics, Inc. – 73 Serial Communication E.g. I2C

74 Dave Stewart, ESC Silicon Valley – April 2010 Troubleshooting Real-Time Issues using a Logic Analyzer © 2010 InHand Electronics, Inc. – 74 Serial Communication E.g. I2C Actual Signals. Verify: - Clock Rates - Glitches and Noise - Sequence - Timing between edges of different signals Protocol. Verify: Number of Bytes Transmitted – Acks and NAcks – Start and Stop Bits – Value of each byte –

75 Dave Stewart, ESC Silicon Valley – April 2010 Troubleshooting Real-Time Issues using a Logic Analyzer © 2010 InHand Electronics, Inc. – 75 Serial Communication Real-Time Gotchas Code will write register, but serial transmission is usually NOT done when code continues, as operations continue in parallel. For many hardware platforms, then the code in fact is pre-loading the NEXT byte, while previous byte is being transmitted, but then it waits on a status flag of the current byte to be completed. This could get confusing when creating the code or debugging the low-level read and write functions. Use logic analyzer coupled with PDEBUG to view precisely what is happening in the code.

76 Dave Stewart, ESC Silicon Valley – April 2010 Troubleshooting Real-Time Issues using a Logic Analyzer © 2010 InHand Electronics, Inc. – 76 Serial Communication Miscellaneous Notes Use built-in interpreters provided by logic analyzer software to analyze the protocol I2C, SPI, RS232 For more complex protocols, use a protocol analyzer, not a logic analyzer. E.g. USB or Ethernet Use a oscilloscope to verify signal integrity Use a protocol analyzer to validate the protocol Collect data in parallel with PDEBUG information, to see relationship between code execution and serial transmission

77 Dave Stewart, ESC Silicon Valley – April 2010 Troubleshooting Real-Time Issues using a Logic Analyzer © 2010 InHand Electronics, Inc. – 77 Using Logic Analyzer to Measure Thread Execution Time Resolution: measuring limitation of hardware Accuracy: depends on method, x y x=measurement, y=accuracy Granularity: what can be measured: program, function, or line of code Difficulty: effort (and time) needed to make a measurement Techniques available to measure execution time:

78 © 2010 InHand Electronics, Inc. – Troubleshooting Real-Time Software Issues Using a Logic Analyzer Dave Stewart Director of Software Engineering InHand Electronics Embedded Systems Conference Silicon Valley 2010 Class ESC 364/366

79 Dave Stewart, ESC Silicon Valley – April 2010 Troubleshooting Real-Time Issues using a Logic Analyzer © 2010 InHand Electronics, Inc. – 79 Using Logic Analyzer to Measure Thread Execution Time Resolution: measuring limitation of hardware Accuracy: depends on method, x y x=measurement, y=accuracy Granularity: what can be measured: program, function, or line of code Difficulty: effort (and time) needed to make a measurement Techniques available to measure execution time:

80 Dave Stewart, ESC Silicon Valley – April 2010 Troubleshooting Real-Time Issues using a Logic Analyzer © 2010 InHand Electronics, Inc. – 80 Using Logic Analyzer to Measure Thread Execution Time A logic analyzer provides the most accurate measurements of execution time The resolution is sufficient to capture operating system overhead, fast interrupts, and even the shortest of code segments It provides an independent time base, so that if hardware timers are inaccurate for any reason (such as a faulty oscillator), it will still catch the problem. The main drawback is that it is harder to use than some other tools. This is a typical trade-off: quality vs. effort.

81 Dave Stewart, ESC Silicon Valley – April 2010 Troubleshooting Real-Time Issues using a Logic Analyzer © 2010 InHand Electronics, Inc. – 81 For Most Embedded Systems: Execution Time = $$$ Money ($$$) Have a limited amount Bills have deadlines Cant spend more than you have* Count it to know how much you are spending Dont buy without knowing the price Good budgeting leads to good cash flow CPU Execution Time Have a limited amount Real-time deadlines Cant use more than you have* Measure it to know how much you are using Dont execute code without knowing its CPU usage Good real-time design leads to good CPU usage *except if you use credit

82 Dave Stewart, ESC Silicon Valley – April 2010 Troubleshooting Real-Time Issues using a Logic Analyzer © 2010 InHand Electronics, Inc. – 82 For Most Embedded Systems: Execution Time = $$$ Measuring Execution Time Its like counting $$$ Analyzing Real-Time Performance Its like budgeting $$$

83 Dave Stewart, ESC Silicon Valley – April 2010 Troubleshooting Real-Time Issues using a Logic Analyzer © 2010 InHand Electronics, Inc. – 83 Timing and Synchronization Model of a Good Software Task. Read Inputs/Events Do Processing Write Outputs Wait for Event For periodic tasks, event is time-based. For other tasks, event could be an interrupt, message arrival, semaphore wakeup, or any other signal.

84 Dave Stewart, ESC Silicon Valley – April 2010 Troubleshooting Real-Time Issues using a Logic Analyzer © 2010 InHand Electronics, Inc. – 84 Timing and Synchronization Give each module two unique codes. One for Begin Another for End Add unique codes to interrupt handlers One as handler is entered Other as it exits Read Inputs/Events Do Processing Write Outputs Wait for Event PDEBUG_HEX8(stopcode) PDEBUG_HEX8(startcode)

85 Dave Stewart, ESC Silicon Valley – April 2010 Troubleshooting Real-Time Issues using a Logic Analyzer © 2010 InHand Electronics, Inc. – 85 MEZ_START() MEZ_STOP() #define MEZ_START(taskid) \ PDEBUG_HEX8(taskid); #define MEZ_STOP(taskid) \ PDEBUG_HEX8(0x80|(taskid));

86 Dave Stewart, ESC Silicon Valley – April 2010 Troubleshooting Real-Time Issues using a Logic Analyzer © 2010 InHand Electronics, Inc. – 86 MEZ_START() MEZ_STOP() Read Inputs/Events Do Processing Write Outputs Wait for Event MEZ_STOP(gettid()) MEZ_START(gettid()) Assumes a threaded environment with Task IDs Modify macros to accommodate different OS task ID assignments.

87 Dave Stewart, ESC Silicon Valley – April 2010 Troubleshooting Real-Time Issues using a Logic Analyzer © 2010 InHand Electronics, Inc. – 87 Measuring Execution Time of Tasks Sample Logic Analyzer Event Log TraceDataRel TimeAbs Time 0510ns0ns ms3.0669ms ms9.0010ms ms12.067ms us12.308ms ms17.382ms ms18.980ms ms22.047ms ms28.677ms us29.114ms ms32.180ms us32.954ms ms38.048ms us38.944ms ms42.318ms ms48.928ms ms51.975ms ms55.083ms ms57.883ms ms58.892ms ms61.938ms ms66.255ms Task Thread ID Period(msec) Priority TaskA High TaskB Medium TaskC Low Task A Task B Task C

88 Dave Stewart, ESC Silicon Valley – April 2010 Troubleshooting Real-Time Issues using a Logic Analyzer © 2010 InHand Electronics, Inc. – 88 Tabulate Execution Time by Task Example for set of 8 tasks and 1 interrupt handler Use one of the measurement methods to create the following table (in seconds): Total sum of execution times = sec ID Cref Tref Cavg Cmax Deadline T= Period C = Extended Time ref = reference (i.e. from specs avg = measured average max = measured worst-case

89 Dave Stewart, ESC Silicon Valley – April 2010 Troubleshooting Real-Time Issues using a Logic Analyzer © 2010 InHand Electronics, Inc. – 89 Measuring Execution Time of Tasks Sample Logic Analyzer Event Log TraceDataRel TimeAbs Time 0510ns0ns ms3.0669ms ms9.0010ms ms12.067ms us12.308ms ms17.382ms ms18.980ms ms22.047ms ms28.677ms us29.114ms ms32.180ms us32.954ms ms38.048ms us38.944ms ms42.318ms ms48.928ms ms51.975ms ms55.083ms ms57.883ms ms58.892ms ms61.938ms ms66.255ms Task Thread ID Period(msec) Priority TaskA High TaskB Medium TaskC Low

90 Dave Stewart, ESC Silicon Valley – April 2010 Troubleshooting Real-Time Issues using a Logic Analyzer © 2010 InHand Electronics, Inc. – 90 Measuring Execution Time of Tasks Sample Logic Analyzer Event Log TraceDataRel TimeAbs Time 0510ns0ns ms3.0669ms ms9.0010ms ms12.067ms us12.308ms ms17.382ms ms18.980ms ms22.047ms ms28.677ms us29.114ms ms32.180ms us32.954ms ms38.048ms us38.944ms ms42.318ms ms48.928ms ms51.975ms ms55.083ms ms57.883ms ms58.892ms ms61.938ms ms66.255ms Task Thread ID Period(msec) Priority TaskA High TaskB Medium TaskC Low Task A Task B Task C

91 Dave Stewart, ESC Silicon Valley – April 2010 Troubleshooting Real-Time Issues using a Logic Analyzer © 2010 InHand Electronics, Inc. – 91 Measuring Execution Time of Tasks Sample Logic Analyzer Event Log TraceDataRel TimeAbs Time 0510ns0ns ms3.0669ms ms9.0010ms ms12.067ms us12.308ms ms17.382ms ms18.980ms ms22.047ms ms28.677ms us29.114ms ms32.180ms us32.954ms ms38.048ms us38.944ms ms42.318ms ms48.928ms ms51.975ms ms55.083ms ms57.883ms ms58.892ms ms61.938ms ms66.255ms Task Thread ID Period(msec) Priority TaskA High TaskB Medium TaskC Low Task B execution time for this iteration of Task B is msec

92 Dave Stewart, ESC Silicon Valley – April 2010 Troubleshooting Real-Time Issues using a Logic Analyzer © 2010 InHand Electronics, Inc. – 92 Measuring Execution Time of Tasks Sample Logic Analyzer Event Log TraceDataRel TimeAbs Time 0510ns0ns ms3.0669ms ms9.0010ms ms12.067ms us12.308ms ms17.382ms ms18.980ms ms22.047ms ms28.677ms us29.114ms ms32.180ms us32.954ms ms38.048ms us38.944ms ms42.318ms ms48.928ms ms51.975ms ms55.083ms ms57.883ms ms58.892ms ms61.938ms ms66.255ms Task Thread ID Period(msec) Priority TaskA High TaskB Medium TaskC Low Task C execution time for this iteration of Task C is ( )-( ) = msec

93 Dave Stewart, ESC Silicon Valley – April 2010 Troubleshooting Real-Time Issues using a Logic Analyzer © 2010 InHand Electronics, Inc. – 93 Measuring Execution Time of Tasks Sample Logic Analyzer Event Log TraceDataRel TimeAbs Time 0510ns0ns ms3.0669ms ms9.0010ms ms12.067ms us12.308ms ms17.382ms ms18.980ms ms22.047ms ms28.677ms us29.114ms ms32.180ms us32.954ms ms38.048ms us38.944ms ms42.318ms ms48.928ms ms51.975ms ms55.083ms ms57.883ms ms58.892ms ms61.938ms ms66.255ms Task Thread ID Period(msec) Priority TaskA High TaskB Medium TaskC Low The conversion from event log to table should be automated, using either a custom program, spreadsheet, or a software analyzer tool that has the necessary features to give you execution time for each cycle of every task.

94 Dave Stewart, ESC Silicon Valley – April 2010 Troubleshooting Real-Time Issues using a Logic Analyzer © 2010 InHand Electronics, Inc. – 94 Tabulate Execution Time by Task Example for set of 8 tasks and 1 interrupt handler Create the following table (in seconds): Total sum of execution times = sec ID Cref Tref Cavg Cmax Deadline T= Period C = Extended Time ref = reference (i.e. from specs avg = measured average max = measured worst-case Task ID 0 is actually an interrupt handler Estimated Worst-Case Execution Time Desired Period (or rate) period = 1/rate Measured Average Execution Time Measured Worst-Case Execution Time Number of RTOS-Detected Missed Deadlines (requires an instrumentation point with the timing error handler.

95 Expanded Model of a Task Wait for Event

96 Dave Stewart, ESC Silicon Valley – April 2010 Troubleshooting Real-Time Issues using a Logic Analyzer © 2010 InHand Electronics, Inc. – 96 Analyzing Effect of Operating System Overhead Overhead is Already Part of the Measurements Using the logic analyzer technique, overhead becomes part of the measurements, but it is not uniformly distributed across tasks. This is important to consider when precise measurements are needed.

97 Dave Stewart, ESC Silicon Valley – April 2010 Troubleshooting Real-Time Issues using a Logic Analyzer © 2010 InHand Electronics, Inc. – 97 Analyzing Effect of Operating System Overhead Distribution of Overhead in Measurements

98 Dave Stewart, ESC Silicon Valley – April 2010 Troubleshooting Real-Time Issues using a Logic Analyzer © 2010 InHand Electronics, Inc. – 98 Measuring Operating System Overhead Δ thr Low priority task continuarlly sends pulses to channel 1 High priority task provides a single pulse to channel 2 This results in the following diagram: High Priority task execution time = Operating system overhead to switch tasks Note that overhead is not necessarily constant. The overhead is often a function of the number of tasks in the system. For most analysis, however, assuming that the overhead is constant is sufficient.

99 Dave Stewart, ESC Silicon Valley – April 2010 Troubleshooting Real-Time Issues using a Logic Analyzer © 2010 InHand Electronics, Inc. – 99 Validating Periods for Periodic Tasks Are tasks in the application executing at the rates requested? Quite often, they are not, and could be the cause of errors! For Example Resolution of system clock 400 Hz task cannot be implemented if system clock is 1 msec since it requires a tick of 2.5 msec Resolution of hardware timer affects accuracy of system clock A 1 msec clock might be approximated to 998 usec This affects time by over 3 minutes per day! Skew due to implementation error Occurs when using sleep(X) block task for X msec Instead, must use pause(X) block task until time=X Real-Time Scheduling Problems cause actual periods to vary Overloaded processor Priority Inversion Missed Deadlines Interrupts executing for too long Note: Many software analyzers are unable to detect these problems.

100 Dave Stewart, ESC Silicon Valley – April 2010 Troubleshooting Real-Time Issues using a Logic Analyzer © 2010 InHand Electronics, Inc. – 100 Validating Periods for Periodic Tasks Recall the MEZ_START() and MEZ_STOP Macros

101 Dave Stewart, ESC Silicon Valley – April 2010 Troubleshooting Real-Time Issues using a Logic Analyzer © 2010 InHand Electronics, Inc. – 101 Validating Periods for Periodic Tasks Find Non-Delayed Starting Points in Event Log Non-delayed starting points starting points occur when MEZ_START() is detected to occur either: after MEZ_START() of a lower-priority tasks cycle, but before the lower- priority tasks MEZ_STOP() when the idle task is executing Example:

102 Dave Stewart, ESC Silicon Valley – April 2010 Troubleshooting Real-Time Issues using a Logic Analyzer © 2010 InHand Electronics, Inc. – 102 Validating Periods for Periodic Tasks Compute Time Differences Compute the time difference between each two non-delayed starting points Divide by (number_of_delayed_starting_points+1) between those starting point. This number should be exactly the period of the task; if it isnt, there could be timing errors. Example of what should be expected period = ( )(1 + 1) = 80/2 = 40 msec Repeat for every pair of non-delayed starting points.

103 Dave Stewart, ESC Silicon Valley – April 2010 Troubleshooting Real-Time Issues using a Logic Analyzer © 2010 InHand Electronics, Inc. – 103 Validating Periods for Periodic Tasks Example of a Problem Suppose the following timing diagram was obtained: period = (88 – 10)(1+1) = 78/2 = 39 msec In this case, 40 msec period is requested, but because the system clock is set to 3 msec, exactly 40 msec cannot be achieved. The RTOS instead rounded to a 39 msec period, without any notification of error to the designer. Not 40 msec as requested!

104 Dave Stewart, ESC Silicon Valley – April 2010 Troubleshooting Real-Time Issues using a Logic Analyzer © 2010 InHand Electronics, Inc. – 104 Checking for Missed Deadlines Missed Deadline: Periodic task is not finished executing an iteration of code, when it is time to start its next iteration. To check for missed deadlines, do the following for each task: Find non-delayed starting points for each task in the event log (use same method as for validating periods) For the delayed starting points, compute the requested starting point by using the non-delayed starting points and adding to it the measured period Compute the deadline time relative to each starting point The MEZ_STOP() event that indicates end of a task should execute before this computed deadline time every cycle. This procedure can be automated by using data downloaded from the logic analyzer as input.

105 Dave Stewart, ESC Silicon Valley – April 2010 Troubleshooting Real-Time Issues using a Logic Analyzer © 2010 InHand Electronics, Inc. – 105 Analyzing Real-Time Performance and Potential for Overload Use Real-Time Scheduling Analysis Real-Time Scheduling Analysis is very theoretical and mathematically oriented. Here, a practical approximation is presented, using real measured data, that can lead to quickly converging on resolving real-time loading issues in a system. Well start with the theory, then simplify it to make it more practical for use in system design and debugging.

106 Dave Stewart, ESC Silicon Valley – April 2010 Troubleshooting Real-Time Issues using a Logic Analyzer © 2010 InHand Electronics, Inc. – 106 Practical Real-Time Scheduling Analysis Fixed-Priority Scheduling Theory A task set consisting of n periodic threads is schedulable using the Rate Monotonic Algorithm if the following equation holds: Example, lets apply equation to the data collected and shown in the table: Total sum of execution times = ID Cref Tref Cavg Cmax Deadline

107 Dave Stewart, ESC Silicon Valley – April 2010 Troubleshooting Real-Time Issues using a Logic Analyzer © 2010 InHand Electronics, Inc. – 107 Fixed-Priority Scheduling Theory Simplifying the Equations First, some approximations with negligible effect except when during the most critical fine-tuning. Assume ID 0 is highest priority task instead of an interrupt Instead of 0 < t <= D¡, use longest length of trace that is a multiple of the slowest task This reduces the equation: i This simply leads to conservative estimates, since it results in computing more overhead than there really is This means try equation first with i=1, then i=2, then i=3, until LHS >1, or i = = n thr to:

108 Dave Stewart, ESC Silicon Valley – April 2010 Troubleshooting Real-Time Issues using a Logic Analyzer © 2010 InHand Electronics, Inc. – 108 Fixed-Priority Scheduling Analysis Applying the Equations using Real Data With 1 Task: Total sum of execution times = ID Cref Tref Cavg Cmax Deadline LHS = i = 1 Plug into equation This means task 0 is always schedulable

109 Dave Stewart, ESC Silicon Valley – April 2010 Troubleshooting Real-Time Issues using a Logic Analyzer © 2010 InHand Electronics, Inc. – 109 Fixed-Priority Scheduling Analysis With 2 Tasks With 2 Tasks: Total sum of execution times = ID Cref Tref Cavg Cmax Deadline LHS = i = 2 Plug into equation This means tasks 0 & 1 are always schedulable =

110 Dave Stewart, ESC Silicon Valley – April 2010 Troubleshooting Real-Time Issues using a Logic Analyzer © 2010 InHand Electronics, Inc. – 110 Applying the Equations using Real Data With 3 Tasks With 3 Tasks: Total sum of execution times = ID Cref Tref Cavg Cmax Deadline Plug into equation i = 3 LHS = = This means task 0 & 1 & 2 are always schedulable

111 Dave Stewart, ESC Silicon Valley – April 2010 Troubleshooting Real-Time Issues using a Logic Analyzer © 2010 InHand Electronics, Inc. – 111 Applying the Equations using Real Data With 4 Tasks With 4 Tasks: Total sum of execution times = ID Cref Tref Cavg Cmax Deadline Plug into equation i = 4 LHS = = This means task 0 & 1 & 2 & 4 are NOT always schedulable

112 Dave Stewart, ESC Silicon Valley – April 2010 Troubleshooting Real-Time Issues using a Logic Analyzer © 2010 InHand Electronics, Inc. – 112 Applying the Equations using Real Data So What? What does the analysis mean? It says a lot: Unless changes are made only Tasks 0, 1, and 2 are guaranteed to not miss deadlines In order for Task 3 to meet deadlines, either Decrease collective execution time (C i ) of Tasks 0, 1, 2, and 3 Increase period (T i ) of Tasks 0, 1, 2, and/or 3 None of Tasks 4 through 8 are guaranteed to meet deadlines as long as Task 3 is not guaranteed. Question: To make Task 3 schedulable, by how much do we decrease C i or increase T i ?

113 Dave Stewart, ESC Silicon Valley – April 2010 Troubleshooting Real-Time Issues using a Logic Analyzer © 2010 InHand Electronics, Inc. – 113 Fixed Priority Scheduling Analysis Making an Unschedulable Task Set Schedulable Suppose goal is to fix the system so that the interrupt handler (PID 0) and first three tasks (PID 1 through 3) are guaranteed to meet their deadlines. First, revise C ref using the values of C max. This refines the estimated worst- case execution times to ensure they are more accurate. To make tasks schedulable, must reduce left side of equation to below 1.0 (it was in example). One of the following must be done: Decrease CPU usage by optimizing code I.e. reduce C ref for one or more tasks Increase period if application and hardware allow it I.e. increase T ref for one or more tasks Repeat the math only, without actually performing the above optimization or period changes. Only when a valid answer is obtained should the actual code be modified. Increasing period T ref is a lot easier than reducing execution time C ref. To reduce C ref, it is necessary to optimize or delete some code.

114 Dave Stewart, ESC Silicon Valley – April 2010 Troubleshooting Real-Time Issues using a Logic Analyzer © 2010 InHand Electronics, Inc. – 114 Real-Time Scheduling Discussion: Getting 110% Effort from the CPU Real-Time Scheduling is often idealized. In reality, the following situations are quite common, and appear to negate the use of proper real-time analysis: A tasks worst-case execution time is much greater than its typical case. The system is mostly event-driven. This is not a hard real-time system: deadlines are soft. A tasks period changes depending on what it is doing. Tasks waste time busy-waiting. Its a fast CPU, so it doesnt matter how long the code takes. Cant use more than 80% utilization to ensure a buffer zone for timing problems I need 110% utilization! Equivalent to I need 110% $$$$ => Use Credit! No matter what the problem, however, knowing the execution time of each task, and using it as the basis for analyzing the system, is essential to ensure a predictable real-time system that meets application requirements. Regardless of the application, systematically engineering the timing of the system will lead to a system with minimal timing errors and efficient budgeting of the CPU while lowering the amount of time needed to implement and debug the code.

115 Dave Stewart, ESC Silicon Valley – April 2010 Troubleshooting Real-Time Issues using a Logic Analyzer © 2010 InHand Electronics, Inc. – 115 Summary Logic Analyzer provides a Window into Embedded Code A window with micro-second resolution Use it in addition to other methods It is complimentary to using print statements and emulators Defining the right macros requires some creativity Many possibilities. This class only scratches the surface. Look for Anomalies and anything that just doesnt look right These are usually the clues to where the problems lie Use it to troubleshoot bugs or measure performance Wide variety of tricks and techniques using the logic analyzer Good Luck! This is a tool to tackle the toughest embedded system bugs. Hopefully weeks will become days, and days will become hours


Download ppt "© 2010 InHand Electronics, Inc. – www.inhand.com Troubleshooting Real-Time Software Issues Using a Logic Analyzer Dave Stewart Director of Software Engineering."

Similar presentations


Ads by Google