© 2011 InHand Electronics, Inc. – www.inhand.com Troubleshooting Real-Time Software Issues Using a Logic Analyzer Dave Stewart Director of Software Engineering.

Slides:



Advertisements
Similar presentations
Numbers Treasure Hunt Following each question, click on the answer. If correct, the next page will load with a graphic first – these can be used to check.
Advertisements

Symantec 2010 Windows 7 Migration EMEA Results. Methodology Applied Research performed survey 1,360 enterprises worldwide SMBs and enterprises Cross-industry.
Symantec 2010 Windows 7 Migration Global Results.
Simplifications of Context-Free Grammars
Variations of the Turing Machine
Adders Used to perform addition, subtraction, multiplication, and division (sometimes) Half-adder adds rightmost (least significant) bit Full-adder.
3rd Annual Plex/2E Worldwide Users Conference 13A Batch Processing in 2E Jeffrey A. Welsh, STAR BASE Consulting, Inc. September 20, 2007.
Process Description and Control
AP STUDY SESSION 2.
1
Chapter 7 Constructors and Other Tools. Copyright © 2006 Pearson Addison-Wesley. All rights reserved. 7-2 Learning Objectives Constructors Definitions.
Copyright © 2003 Pearson Education, Inc. Slide 1 Computer Systems Organization & Architecture Chapters 8-12 John D. Carpinelli.
1 Copyright © 2013 Elsevier Inc. All rights reserved. Chapter 4 Computing Platforms.
Sequential Logic Design
Processes and Operating Systems
Copyright © 2013 Elsevier Inc. All rights reserved.
Process a Customer Chapter 2. Process a Customer 2-2 Objectives Understand what defines a Customer Learn how to check for an existing Customer Learn how.
Chapter 5 Input/Output 5.1 Principles of I/O hardware
Date: File: PRO1_17E.1 SIMATIC S7 Siemens AG All rights reserved. Information and Training Center Knowledge for Automation Solutions (Version.
1 Click here to End Presentation Software: Installation and Updates Internet Download CD release NACIS Updates.
© Tally Solutions Pvt. Ltd. All Rights Reserved Shoper 9 License Management December 09.
Version 1.0 digitaloffice.intel.com Intel ® vPro Technology Intel ® Active Management Technology Setup and Configuration HP Laptop – Compaq 6910p Small.
Break Time Remaining 10:00.
Figure 12–1 Basic computer block diagram.
Microprocessor Architecture Pipelined Architecture
Factoring Quadratics — ax² + bx + c Topic
Turing Machines.
ETS4 - What's new? - How to start? - Any questions?
PP Test Review Sections 6-1 to 6-6
User Friendly Price Book Maintenance A Family of Enhancements For iSeries 400 DMAS from Copyright I/O International, 2006, 2007, 2008, 2010 Skip Intro.
© 2010 InHand Electronics, Inc. – Troubleshooting Real-Time Software Issues Using a Logic Analyzer Dave Stewart Director of Software Engineering.
Project 5: Virtual Memory
EIS Bridge Tool and Staging Tables September 1, 2009 Instructor: Way Poteat Slide: 1.
Chapter 3 Logic Gates.
John Ogilvie High School - CfE Physics
Outline Minimum Spanning Tree Maximal Flow Algorithm LP formulation 1.
INTRODUCTION Lesson 1 – Microsoft Word Word Basics
Operating Systems Operating Systems - Winter 2012 Chapter 4 – Memory Management Vrije Universiteit Amsterdam.
Operating Systems Operating Systems - Winter 2010 Chapter 3 – Input/Output Vrije Universiteit Amsterdam.
Chapter 20 Network Layer: Internet Protocol
Sample Service Screenshots Enterprise Cloud Service 11.3.
Copyright © 2012, Elsevier Inc. All rights Reserved. 1 Chapter 7 Modeling Structure with Blocks.
Success with ModelSmart3D Pre-Engineering Software Corporation Written by: Robert A. Wolf III, P.E. Copyright 2001, Pre-Engineering Software Corporation,
 Copyright I/O International, 2013 Visit us at: A Feature Within from Item Class User Friendly Maintenance  Copyright.
Adding Up In Chunks.
MaK_Full ahead loaded 1 Alarm Page Directory (F11)
1 10 pt 15 pt 20 pt 25 pt 5 pt 10 pt 15 pt 20 pt 25 pt 5 pt 10 pt 15 pt 20 pt 25 pt 5 pt 10 pt 15 pt 20 pt 25 pt 5 pt 10 pt 15 pt 20 pt 25 pt 5 pt Synthetic.
: 3 00.
5 minutes.
1 hi at no doifpi me be go we of at be do go hi if me no of pi we Inorder Traversal Inorder traversal. n Visit the left subtree. n Visit the node. n Visit.
Types of selection structures
Speak Up for Safety Dr. Susan Strauss Harassment & Bullying Consultant November 9, 2012.
Converting a Fraction to %
Numerical Analysis 1 EE, NCKU Tien-Hao Chang (Darby Chang)
Clock will move after 1 minute
1 © 2004, Cisco Systems, Inc. All rights reserved. CCNA 1 v3.1 Module 9 TCP/IP Protocol Suite and IP Addressing.
Chapter 11 Creating Framed Layouts Principles of Web Design, 4 th Edition.
Physics for Scientists & Engineers, 3rd Edition
Thomas L. Floyd Digital Fundamentals, 9e
Select a time to count down from the clock above
Import Tracking and Landed Cost Processing An Enhancement For AS/400 DMAS from  Copyright I/O International, 2001, 2005, 2008, 2012 Skip Intro Version.
1.step PMIT start + initial project data input Concept Concept.
Page 1 Orchard Harvest ™ LIS Find a Patient Training.
1 Dr. Scott Schaefer Least Squares Curves, Rational Representations, Splines and Continuity.
TCP/IP Protocol Suite 1 Chapter 18 Upon completion you will be able to: Remote Login: Telnet Understand how TELNET works Understand the role of NVT in.
1 Decidability continued…. 2 Theorem: For a recursively enumerable language it is undecidable to determine whether is finite Proof: We will reduce the.
FIGURE 3-1 Basic parts of a computer. Dale R. Patrick Electricity and Electronics: A Survey, 5e Copyright ©2002 by Pearson Education, Inc. Upper Saddle.
Princess Sumaya University
Presentation transcript:

© 2011 InHand Electronics, Inc. – Troubleshooting Real-Time Software Issues Using a Logic Analyzer Dave Stewart Director of Software Engineering InHand Electronics Embedded Systems Conference Chicago 2011 Class ESC 217

Dave Stewart, ESC Chicago – June 2011 Troubleshooting Real-Time Issues using a Logic Analyzer © 2011 InHand Electronics, Inc. – 2 What are The Tough Bugs to Debug in Real-Time Systems? Glitches Timing and Synchronization Problems Driver Errors Misbehaving Interrupts Memory Corruption Priority Inversion Performance Issues Hardware Errors

Dave Stewart, ESC Chicago – June 2011 Troubleshooting Real-Time Issues using a Logic Analyzer © 2011 InHand Electronics, Inc. – 3 Limitations of Traditional Debugging Print Statements There is no console, so Print Statements dont work Print Statements are too slow thus provide insufficient information Print function significantly affects real-time performance Writing debug output to a serial port changes the timing too much Adding print statements changes program behavior Cant measure performance at a fine granularity Max 50 to 100 print statements per second The code crashes, but there is insufficient feedback as to where Symbolic Debuggers (e.g. IDEs, via JTAG, ActiveSync, or other comm link) A symbolic debugger or emulator is not available Stepping through the code makes the program behave differently Breakpoints will break real-time performance There is real I/O, it doesnt work Debugger doesnt deal with interrupts properly There might be a race condition or other synchronization problem

Dave Stewart, ESC Chicago – June 2011 Troubleshooting Real-Time Issues using a Logic Analyzer © 2011 InHand Electronics, Inc. – 4 Solution: Use a Logic Analyzer for Real-Time Issues Most detailed view of your code Microsecond view of software Easily 50,000 debug data points per second Info is time-stamped for timing assessment Real-Time Can use it for interrupts and I/O drivers Impact on real-time execution is negligible Identify temporal relationships among tasks Monitor interrupts and how they may affect execution Watch variables without changing execution time Anomalies Spot anomalies and different patterns of execution Obtain sufficient proof that a problem is hardware, not software Fine-grain timing measurements to identify performance culprits

Dave Stewart, ESC Chicago – June 2011 Troubleshooting Real-Time Issues using a Logic Analyzer © 2011 InHand Electronics, Inc. – 5 Logic Analyzer for Debugging Should I use it all the time? Using a Logic Analyzer is NOT easy Use Print Statements and Symbolic Debuggers to solve easy and non-real-time problems first Add to your repertoire of available tools to solve hard problems Solve functional problems using print statements. If necessary, run the functions on the desktop, and debug them there. Only move to embedded environment when it is working well. Use Symbolic Debuggers mainly for Tracing through code that fails in a consistent manner Post-Mortem debugging of crashes, to view all variables

Dave Stewart, ESC Chicago – June 2011 Troubleshooting Real-Time Issues using a Logic Analyzer © 2011 InHand Electronics, Inc. – 6 Sample Logic Analyzer Output Timing Diagram This output format will be used for most examples in this class. More details interpreting this diagram to follow.

Dave Stewart, ESC Chicago – June 2011 Troubleshooting Real-Time Issues using a Logic Analyzer © 2011 InHand Electronics, Inc. – 7 Sample Logic Analyzer Output State Listing counttrigmode16modedatatime-relativetime-absolute

Dave Stewart, ESC Chicago – June 2011 Troubleshooting Real-Time Issues using a Logic Analyzer © 2011 InHand Electronics, Inc. – 8 Logic Analyzer Features Preferably at least 8 channels 16 to 24 channels is nice to have, to monitor many additional signals under 7 channels still useable, but it prevents use of some techniques Large memory depth in M-Samples, not K-Samples Multiple Views Timing Diagrams State Listings Decoding of Serial Protocols Useful features: Search, Filtering, and Triggering High-Speed Interface to a PC Built-in PC, USB Memory Stick, Ethernet all OK Common logic analyzer features that are NOT needed: High Speed. Dont need GHz. Very slow analyzers, e.g. 10MHz, are OK. Many channels. Dont need 64 or 128 channel analyzers. Low cost USB Logic Analyzer Pods are OK A $25,000 logic analyzer is nice to have and will have many extras that could be useful Examples in this presentation use the Tektronix TLA700 What is discussed in this talk can be accomplished with a $400 USB pod. Examples in this presentation use the Intronix LogicPort

Dave Stewart, ESC Chicago – June 2011 Troubleshooting Real-Time Issues using a Logic Analyzer © 2011 InHand Electronics, Inc. – 9 Target Setup: Interconnections Logic Analyzer (Assuming 8-bit channels) System Under Test D0-D7Ch.1 RX/TX Ch.2 I2C SPI Keys This is an example setup given 16 bits. With fewer bits, connect whatever you can.

Dave Stewart, ESC Chicago – June 2011 Troubleshooting Real-Time Issues using a Logic Analyzer © 2011 InHand Electronics, Inc. – 10 An Aside Print Statement Debugging Forms the basis for logic analyzer methods

Dave Stewart, ESC Chicago – June 2011 Troubleshooting Real-Time Issues using a Logic Analyzer © 2011 InHand Electronics, Inc. – 11 An Aside Print Statement Debugging myfunc() { code here printf(I got here\n); more code here printf(Going to call yourfunc()\n); result = yourfunc(); printf(My result is %d\n,result); etc } I got here Going to call yourfunc() My result is 384

Dave Stewart, ESC Chicago – June 2011 Troubleshooting Real-Time Issues using a Logic Analyzer © 2011 InHand Electronics, Inc. – 12 Print Statement Debugging: Problems A Lot of typing Minimal information per statement Hard to separate between debug and normal print statements Prone to errors Cannot easily disable them Ultimately very inefficient

Dave Stewart, ESC Chicago – June 2011 Troubleshooting Real-Time Issues using a Logic Analyzer © 2011 InHand Electronics, Inc. – 13 Debug Macros instead of Print Statements DEBUG_WHERE() #define DEBUG_PRINT fprintf(stderr, #define DEBUG_WHERE() \ DEBUG_PRINT \ [%s:%u-%s]\n,\ __FILE__,__LINE__,__FUNCTION__) Complete.h file for all macros given in this class are in the paper that accompanies this presentation on the conference website or CD. Note: Not all compilers support __FUNCTION__.

Dave Stewart, ESC Chicago – June 2011 Troubleshooting Real-Time Issues using a Logic Analyzer © 2011 InHand Electronics, Inc. – 14 Print Statement Debugging myfunc() { code here DEBUG_WHERE(); more code here DEBUG_WHERE(); result = yourfunc(); DEBUG_INT(result); etc } [myfile.c:3-myfunc] [myfile.c:5-myfunc] [myfile.c:7-myfunc] result=384

Dave Stewart, ESC Chicago – June 2011 Troubleshooting Real-Time Issues using a Logic Analyzer © 2011 InHand Electronics, Inc. – 15 Debug Macros instead of Print Statements DEBUG_INT() #define DEBUG_INT(_var) \ DEBUG_PRINT \ [%s:%u-%s] %s=%d\n,\ __FILE__,__LINE__,__FUNCTION__ \ #_var,_var) result = 384 DEBUG_INT(result) [myfile.c:7-myfunc] result=384

Dave Stewart, ESC Chicago – June 2011 Troubleshooting Real-Time Issues using a Logic Analyzer © 2011 InHand Electronics, Inc. – 16 Debug Macros instead of Print Statements DEBUG_HEX8() #define DEBUG_HEX8(_var) \ DEBUG_PRINT \ [%s:%u-%s] %s=0x%02X\n,\ __FILE__,__LINE__,__FUNCTION__ \ #_var,_var) result = 0xA4 DEBUG_HEX8(result) [myfile.c:7-myfunc] result=0xA4

Dave Stewart, ESC Chicago – June 2011 Troubleshooting Real-Time Issues using a Logic Analyzer © 2011 InHand Electronics, Inc. – 17 Debug Macros instead of Print Statements DEBUG_HEX16() #define DEBUG_HEX16(_var) \ DEBUG_PRINT \ [%s:%u-%s] %s=0x%04X\n,\ __FILE__,__LINE__,__FUNCTION__ \ #_var,_var) result = 0x52A4 DEBUG_HEX16(result) [myfile.c:7-myfunc] result=0x52A4

Dave Stewart, ESC Chicago – June 2011 Troubleshooting Real-Time Issues using a Logic Analyzer © 2011 InHand Electronics, Inc. – 18 Debug Macros for Logic Analyzers Similar to debugging with Print Statements, Except: (disadvantage) Much more difficult to use Learning curve could be days, not minutes (but its worth it for the hard bugs that could otherwise take weeks to debug) (advantage) 1000 times more information Print Statement: 50 Lines/Data Points per second Logic Analyzer: Data Points per second is easy (advantage) Real-time view of the system Data points are time-stamped with microsecond resolution

Dave Stewart, ESC Chicago – June 2011 Troubleshooting Real-Time Issues using a Logic Analyzer © 2011 InHand Electronics, Inc. – 19 Basics of Logic Analyzer Debugging bart = 0x0063; lisa = 0x1018; homer= 0xF04E; marge= 0x00E5; PDEBUG_HEX8(0x40); PDEBUG_HEX16(bart); PDEBUG_HEX8(0x41); PDEBUG_HEX16(lisa); PDEBUG_HEX8(0x42); PDEBUG_HEX16(homer); PDEBUG_HEX8(0x42); PDEBUG_HEX16(marge); See next pages for zoom of this timing diagram. This example assumes only a single 8-bit port is available.

Dave Stewart, ESC Chicago – June 2011 Troubleshooting Real-Time Issues using a Logic Analyzer © 2011 InHand Electronics, Inc. – 20 Basics of Logic Analyzer Debugging PDEBUG_HEX8(0x40);bart = 0x0063; PDEBUG_HEX16(bart); Timing Diagram of each bit. Well see later the value of these for creating code patterns.

Dave Stewart, ESC Chicago – June 2011 Troubleshooting Real-Time Issues using a Logic Analyzer © 2011 InHand Electronics, Inc. – 21 Basics of Logic Analyzer Debugging Each code represents a different variable. Correlate with source code. PDEBUG_HEX8(0x40); PDEBUG_HEX16(bart); PDEBUG_HEX8(0x41); PDEBUG_HEX16(lisa); PDEBUG_HEX8(0x42); PDEBUG_HEX16(homer); PDEBUG_HEX8(0x43); PDEBUG_HEX16(marge); bartlisahomermarge

Dave Stewart, ESC Chicago – June 2011 Troubleshooting Real-Time Issues using a Logic Analyzer © 2011 InHand Electronics, Inc. – 22 Basics of Logic Analyzer Debugging Timescale is Microseconds. Easily add debug statements with minimal intrusion on real-time code. In contrast, each print statement takes 2+ milliseconds. 1 usec

Dave Stewart, ESC Chicago – June 2011 Troubleshooting Real-Time Issues using a Logic Analyzer © 2011 InHand Electronics, Inc. – 23 PDEBUG_HEX8() Macro Initialization (e.g. MSP430) // Example, init MSP430 Port 4 // Simple memory-mapped port uint8_t *pdebug_dout8 = (uint8_t *) 0x001D; uint8_t *pdebug_dir8 = (uint8_t *) 0x001E; *pdebug_dir8 = 0xFF;// Initialize to output

Dave Stewart, ESC Chicago – June 2011 Troubleshooting Real-Time Issues using a Logic Analyzer © 2011 InHand Electronics, Inc. – 24 PDEBUG_HEX8() Macro (E.g. MSP430) #define PDEBUG_HEX8(_val) \ *pdebug_dout8 = (uint8_t)(_val) More details and more macros are provided in the paper that accompanies this presentation on the conference website or CD. bart = 0x0063; PDEBUG_HEX8(0x40); PDEBUG_HEX16(bart);

Dave Stewart, ESC Chicago – June 2011 Troubleshooting Real-Time Issues using a Logic Analyzer © 2011 InHand Electronics, Inc. – 25 PDEBUG_HEX16() Macro If only 8 bits available #define PDEBUG_HEX16(_val) \ PDEBUG_HEX8( (_val) >> 8);\ PDEBUG_HEX8( (_val) ) bart = 0x0063; PDEBUG_HEX8(0x40); PDEBUG_HEX16(bart);

Dave Stewart, ESC Chicago – June 2011 Troubleshooting Real-Time Issues using a Logic Analyzer © 2011 InHand Electronics, Inc. – 26 PDEBUG_VAR16() Shortcut bart = 0x0063; PDEBUG_HEX8(0x40); PDEBUG_HEX16(bart); bart = 0x0063; PDEBUG_VAR16(0x40,bart); #define PDEBUG_VAR16(_code,_val) \ PDEBUG_HEX8( _code);\ PDEBUG_HEX16(_val )

Dave Stewart, ESC Chicago – June 2011 Troubleshooting Real-Time Issues using a Logic Analyzer © 2011 InHand Electronics, Inc. – 27 Test your Macros Example to test PDEBUG_HEX8() Always test your macros before using them. PDEBUG_HEX8(0xFF); PDEBUG_HEX8(0x01); PDEBUG_HEX8(0x02); PDEBUG_HEX8(0x04); PDEBUG_HEX8(0x08); PDEBUG_HEX8(0x10); PDEBUG_HEX8(0x20); PDEBUG_HEX8(0x40); PDEBUG_HEX8(0x80); PDEBUG_HEX8(0x55); PDEBUG_HEX8(0xAA);

Dave Stewart, ESC Chicago – June 2011 Troubleshooting Real-Time Issues using a Logic Analyzer © 2011 InHand Electronics, Inc. – 28 Test your Macros Example to test PDEBUG_HEX8() Allows you to test logic analyzer setup too. If color analyzer, adjust colors to your liking too.

Dave Stewart, ESC Chicago – June 2011 Troubleshooting Real-Time Issues using a Logic Analyzer © 2011 InHand Electronics, Inc. – 29 PDEBUG_HEX32() Macro If only 8 bits available #define PDEBUG_HEX32(_val) \ PDEBUG_HEX8( (_val) >> 24);\ PDEBUG_HEX8( (_val) >> 16);\ PDEBUG_HEX8( (_val) >> 8);\ PDEBUG_HEX8( (_val) )

Dave Stewart, ESC Chicago – June 2011 Troubleshooting Real-Time Issues using a Logic Analyzer © 2011 InHand Electronics, Inc. – 30 PDEBUG_HEX32() Macro mickey = 0x0009B900; donald = 0x0000CFB8; PDEBUG_HEX8(0x44); PDEBUG_HEX32(mickey); PDEBUG_HEX8(0x45); PDEBUG_HEX32(donald);

Dave Stewart, ESC Chicago – June 2011 Troubleshooting Real-Time Issues using a Logic Analyzer © 2011 InHand Electronics, Inc. – 31 PDEBUG_HEX8() Macro (PXA270) // Example, PXA270, use LCD data bits LDD6..LDD13 // 32-bit architecture, uses GPIO2 // Separate registers to set/clear bits uint32_t *pdebug_set8 = (uint32_t *) 0x40E00020; uint32_t *pdebug_clr8 = (uint32_t *) 0x40E0002C; uint32_t *pdebug_dir8 = (uint32_t *) 0x40E00014; *pdebug_dir8 |= 0x000000FF;// Init to output

Dave Stewart, ESC Chicago – June 2011 Troubleshooting Real-Time Issues using a Logic Analyzer © 2011 InHand Electronics, Inc. – 32 PDEBUG_HEX8() Macro Initialization (PXA270) #define PDEBUG_HEX8(_hex8) \ { uint32_t h32 = (uint32_t) hex8;\ uint32_t sg2,cg2;\ \ sg2= ( h32 & 0x000000FF); // set bits\ cg2= ((~h32) & 0x000000FF); // clear bits\ *_gpsr2 = sg2;\ *_gpcr2 = cg2;} Note race condition between set and clear operations; if logic analyzer samples between the two, then it will capture invalid data that needs to be ignored. Option, with an extra bit, pulse it once valid data is written, and only look at data when the extra bit indicates it is valid.

Dave Stewart, ESC Chicago – June 2011 Troubleshooting Real-Time Issues using a Logic Analyzer © 2011 InHand Electronics, Inc. – 33 Logic Analyzer Debug Macros There are many possibilities for defining macros Use macros to enable a common interface across platforms For each new platform, define the macros in a hardware-dependent manner, and place in a.h file Simply include a different.h file for each platform Additional macro examples Paper that accompanies this presentation includes additional macro examples not covered here.

Dave Stewart, ESC Chicago – June 2011 Troubleshooting Real-Time Issues using a Logic Analyzer © 2011 InHand Electronics, Inc. – 34 Tracing Drivers, Low-Level Code, and Real-time Code Recall prior example using Print Statements myfunc() { code here DEBUG_WHERE(); more code here DEBUG_WHERE(); result = yourfunc(); DEBUG_INT(result); etc } [myfile.c:3-myfunc] [myfile.c:5-myfunc] [myfile.c:7-myfunc] result = 384

Dave Stewart, ESC Chicago – June 2011 Troubleshooting Real-Time Issues using a Logic Analyzer © 2011 InHand Electronics, Inc. – 35 Tracing Drivers, Low-Level Code, and Real-time Code #define PDEBUG_WHERE() \ PDEBUG_HEX8(0x11);\ PDEBUG_HEX8( (__LINE__) >> 8);\ PDEBUG_HEX8( (__LINE__) ) #define PDEBUG_INT(code,_val) \ PDEBUG_WHERE();\ PDEBUG_HEX16(_val);

Dave Stewart, ESC Chicago – June 2011 Troubleshooting Real-Time Issues using a Logic Analyzer © 2011 InHand Electronics, Inc. – 36 Tracing Drivers, Low-Level Code, and Real-time Code myfunc() { code here PDEBUG_WHERE(); more code here PDEBUG_WHERE(); result = yourfunc(); PDEBUG_INT(result); etc } 384 = 0x180 Line Numbers in Hex just a code to spot line numbers The output is cryptic compared to using print statements. Hence, continue using print statements if you can. But use the logic analyzer to trace code when print statements dont work. (Some analyzers will print decimal!)

Dave Stewart, ESC Chicago – June 2011 Troubleshooting Real-Time Issues using a Logic Analyzer © 2011 InHand Electronics, Inc. – 37 Fine-Grained Code Optimization ALWAYS measure execution time of a code segment before and after optimization. If an optimization breaks a pattern that the compiler recognizes, it could in fact make things take longer. Not all optimizations are obvious. What is faster, pointers or index arrays? What is faster, case statements or if-then else? What is faster, for loops or while loops? Measure to find out. Use PDEBUG Macros to measure code segments PDEBUG_HEX8(x) before the code segment PDEBUG_HEX8(x+1) after the code segment X is unique in the code, to enable measuring multiple code segments at once

Dave Stewart, ESC Chicago – June 2011 Troubleshooting Real-Time Issues using a Logic Analyzer © 2011 InHand Electronics, Inc. – 38 Performance Anomalies Quite often, performance is not as expected Why Not? Scatter PDEBUG_HEX8() statements throughout code Start and stop of each function and thread is a good starting point. Often, just keep all the prior debug statements in place, they may provide the data needed. Run the code, and monitor on the logic analyzer. Look for patterns of behavior. Identify patterns that use too much CPU. Verify what is happening when analyzer shows periods of inactivity. Monitor execution time of interrupts. Some examples on next few slides.

Dave Stewart, ESC Chicago – June 2011 Troubleshooting Real-Time Issues using a Logic Analyzer © 2011 InHand Electronics, Inc. – 39 Performance Anomalies Green is commmunication. We can clearly see communication stopped for a period of time. Why? Is that normal?

Dave Stewart, ESC Chicago – June 2011 Troubleshooting Real-Time Issues using a Logic Analyzer © 2011 InHand Electronics, Inc. – 40 Performance Anomalies See the Pattern? Four short communication packets, one long one, then none for what seems like two or three system cycles.

Dave Stewart, ESC Chicago – June 2011 Troubleshooting Real-Time Issues using a Logic Analyzer © 2011 InHand Electronics, Inc. – 41 Performance Anomalies What line of code executes immediately after PDEBUG_HEX8(0x9F)? Whatever it is, it takes a LONG time. If there is a performance issue, focus on that code first.

Dave Stewart, ESC Chicago – June 2011 Troubleshooting Real-Time Issues using a Logic Analyzer © 2011 InHand Electronics, Inc. – 42 Performance Anomalies This is an example of code executed with PDEBUG_WHERE() plus a few other PDEBUG statements between every two lines of code. What is happening on Line 51 (0x33)?

Dave Stewart, ESC Chicago – June 2011 Troubleshooting Real-Time Issues using a Logic Analyzer © 2011 InHand Electronics, Inc. – 43 Using Additional Signals Create second PDEBUG_HEX8-like macro E.g. call second one PDEBUG_MODULE() Upon entry or exit to any function in a module, call this one. Use PDEBUG_HEX8 to trace and print variable data. Makes it easier to correlate logic analyzer output to source code Reserve some bits as triggers A lot easier to setup logic analyzer to trigger on an individual bit, rather than a sequence of bytes.

Dave Stewart, ESC Chicago – June 2011 Troubleshooting Real-Time Issues using a Logic Analyzer © 2011 InHand Electronics, Inc. – 44 Additional Signals Example Keys and Scope Monitor key presses They often serve as trigger points Ensure that if debouncing is needed, it is properly handled Oscilloscope Some logic analyzers have integrated oscilloscopes Use for instantaneous analog measurements that are correlated with the PDEBUG outputs Aids in software design and troubleshooting of potential hardware issues

Dave Stewart, ESC Chicago – June 2011 Troubleshooting Real-Time Issues using a Logic Analyzer © 2011 InHand Electronics, Inc. – 45 Additional Signals Example Logic Analyzer Screen Captures Follow Next few pages show a logic analyzer setup with: PDEBUG_HEX8(): 8-bit debug port (yellow) PDEBUG_MODE(): 3-bit [cpu_speed/100MHz] (green) PDEBUG_TRIG(): 1-bit trigger (magenta) 2 keys on keypad (cyan) serial TX/RX (grey) instantaneous current draw (green analog, lower means more current) input voltage (red). Each page is a zoom of the prior page. The zoom area is shown by the magenta rectangle.

Dave Stewart, ESC Chicago – June 2011 Troubleshooting Real-Time Issues using a Logic Analyzer © 2011 InHand Electronics, Inc. – 46 Setup Extras Example zoom

Dave Stewart, ESC Chicago – June 2011 Troubleshooting Real-Time Issues using a Logic Analyzer © 2011 InHand Electronics, Inc. – 47 Setup Extras Example zoom

Dave Stewart, ESC Chicago – June 2011 Troubleshooting Real-Time Issues using a Logic Analyzer © 2011 InHand Electronics, Inc. – 48 Setup Extras Example zoom

Dave Stewart, ESC Chicago – June 2011 Troubleshooting Real-Time Issues using a Logic Analyzer © 2011 InHand Electronics, Inc. – 49 Setup Extras Example

Dave Stewart, ESC Chicago – June 2011 Troubleshooting Real-Time Issues using a Logic Analyzer © 2011 InHand Electronics, Inc. – 50 Setup Extras Monitoring Serial Transmissions Serial Communication Protocol Analyzers exist to monitor integrity of data. Built-in to some logic analyzers In many cases, you dont need anything that complex. Rather, you just want to know how many bytes, and when. Use PDEBUG_HEX8() macros to print the bytes being transmitted.

Dave Stewart, ESC Chicago – June 2011 Troubleshooting Real-Time Issues using a Logic Analyzer © 2011 InHand Electronics, Inc. – 51 Serial Communication E.g. I 2 C CLK Data The code that immediately follows the I 2 C transfer can then be traced line by line. Some Analyzers will convert clock/data to HEX values and validate bus protocols. (Example later)

Dave Stewart, ESC Chicago – June 2011 Troubleshooting Real-Time Issues using a Logic Analyzer © 2011 InHand Electronics, Inc. – 52 Serial Communication E.g. I2C Actual Signals. Verify: - Clock Rates - Glitches and Noise - Sequence - Timing between edges of different signals Protocol. Verify: Number of Bytes Transmitted – Acks and NAcks – Start and Stop Bits – Value of each byte –

Dave Stewart, ESC Chicago – June 2011 Troubleshooting Real-Time Issues using a Logic Analyzer © 2011 InHand Electronics, Inc. – 53 Serial Communication Real-Time Gotchas Code will write register, but serial transmission is usually NOT done when code continues, as operations continue in parallel. For many hardware platforms, then the code in fact is pre-loading the NEXT byte, while previous byte is being transmitted, but then it waits on a status flag of the current byte to be completed. This could get confusing when creating the code or debugging the low-level read and write functions. Use logic analyzer coupled with PDEBUG to view precisely what is happening in the code.

Dave Stewart, ESC Chicago – June 2011 Troubleshooting Real-Time Issues using a Logic Analyzer © 2011 InHand Electronics, Inc. – 54 Serial Communication Miscellaneous Notes Use built-in interpreters provided by logic analyzer software to analyze the protocol I2C, SPI, RS232 For more complex protocols, use a protocol analyzer, not a logic analyzer. E.g. USB or Ethernet Use a oscilloscope to verify signal integrity Use a protocol analyzer to validate the protocol Collect data in parallel with PDEBUG information, to see relationship between code execution and serial transmission

Dave Stewart, ESC Chicago – June 2011 Troubleshooting Real-Time Issues using a Logic Analyzer © 2011 InHand Electronics, Inc. – 55 Summary Logic Analyzer provides a Window into Embedded Code A window with micro-second resolution Use it in addition to other methods It is complimentary to using print statements and emulators Defining the right macros requires some creativity Many possibilities. This class only scratches the surface. Look for Anomalies and anything that just doesnt look right These are usually the clues to where the problems lie Use it to troubleshoot bugs or measure performance Wide variety of tricks and techniques using the logic analyzer Good Luck! This is a tool to tackle the toughest embedded system bugs. Hopefully weeks will become days, and days will become hours