Towards Target-Level Testing and Debugging Tools For Embedded Software Harry Koehnemann, Arizona State University Dr. Timothy Lindquist, Arizona State University Presented by: Soubhagya Kumar Dash

Goal: Identify the problems associated with test case execution for Embedded Systems Goal: Identify the problems associated with test case execution for Embedded Systems To propose solutions for making embedded testing more effective at revealing errors. To propose solutions for making embedded testing more effective at revealing errors.

Introduction Currently: Testing and Debugging of Embedded Real Time Software: A “black art” - ad hoc methods and techniques. Currently: Testing and Debugging of Embedded Real Time Software: A “black art” - ad hoc methods and techniques. - Ineffective and inadequate. Huge costs associated with validation of embedded applications. Despite this, most difficult errors are discovered extremely late in the testing process, making them even more costly to repair. Requirement: Formal methods, development of architectural and software capabilities which support testing and debugging with minimal intrusion on the executing system. This is critical for testing Embedded applications. Test cases must take care of the real time and environmental factors as well. Not just check for correct input-output (functional behavior) mapping.

Software Testing and Debugging Process Testing: Executing a piece of software in order to reveal errors- A substantial portion of the validation process. - Development of test procedures, generation and execution of test cases. Debugging: This is concerned with locating and correcting the cause of an error once it has been revealed. - Developer must recreate exact execution scenario. - Same instruction sequences - All environmental variants must be accounted for.

Embedded Systems Correct execution of embedded applications absolutely critical. Correct execution of embedded applications absolutely critical. Testing and Debugging: Greatly restricted by embedded systems, with constraints such as: Testing and Debugging: Greatly restricted by embedded systems, with constraints such as: - Concurrent Designs - Real-time constraints - Embedded target environments - Distributed hardware architectures - Device control dependencies These restrict execution visibility and control. Target environment: grossly inadequate computing resources.

Software Testing Software testing for embedded systems:4 basic stages 1. Module level testing 2. Integration testing 3. System testing 4. Hardware/Software Integration testing – This is unique to embedded systems. Techniques of particular interest to us: - Testing Concurrent Systems - Non-intrusive testing

Testing Concurrent Systems Concurrency increases the difficulty of s/w testing. -Unmanageably large set of legal execution sequences that a concurrent program may take. -Subsequent execution could lead to different-yet correct results. -Dealing with abstraction -Static and Dynamic testing Non-intrusive testing For host based- intrusion is acceptable. Embedded applications have strict timing requirements. Absolutely imperative that there be no intrusions on a test execution. Embedded tools: ROM/Bus monitors, Emulators

Embedded Testing Embedded systems have critical issues/concerns as discussed on an earlier slide. -Typically developed on custom h/w configurations, each would require own set of tools and techniques. Errors discovered during H/S integration testing are most difficult of all. Often require significant modifications to the s/w system. 2 environments: Host and the Target. Target has little support for s/w development tools.

Current Solutions Hardware Solutions: Attempt to gain execution visibility and program control. - Bus monitors, ROM monitors, in-circuit emulators. - Minimal effectiveness for s/w development, since aid only in gathering low level data, which have to be subsequently mapped- difficult, cumbersome. Software Solutions: Attempt to reduce cost & time spent testing on target. Factors: Level of criticality, Test platform availability, test classification etc. - Will likely lead to extensive modifications, and hence extensive retesting.

Problems with Embedded Testing 1. Expense of testing process - Little reuse, expensive custom validation facilities required for every project. Retests extremely costly. 2. Level of functionality on target Very little, hence a lot of effort to discover errors. Very little, hence a lot of effort to discover errors. 3. Late discovery of errors - s/w modified to rectify h/w errors, delays error discovery. 4. Poor test selection criteria Test case selection rarely on theoretical criteria Test case selection rarely on theoretical criteria 5. Potential use in advancing architectures Little chance of current methods remaining applicable to future architectures. Little chance of current methods remaining applicable to future architectures.

Increasing Target Functionality -Tool support for embedded systems: Lacking. -Current approaches will soon be rendered obsolete. -Proposal: Adding facilities to the underlying system to better support testing and debugging tools. Underlying system: Hardware architecture and the run-time system.

Model Debugging System

The model debugging system continued.. The data path from the debugging/testing tool represents symbol table information that allows the mapping of machine level information to source level constructs. ASIS toolkit provides easy implementation. Any debugging system has at least two processes executing: test program and the debugger. Part of the debugger runs on the host, and the other on the target machine. To make this non-intrusive: 1-Execute code only at break point OR 2-Run debugger as a separate process OR 3-Provide separate execution unit to execute debugger

Architectural Additions 1-Hardware partitioning of memory: Each process has its own protected address space, provision for access control and shared memory. 2-Computational facilities for Debugger: The internal debugger requires computational facilities, it could be modeled as a regular process on the processor, or architecture provisions may be made. 3-Hardware Break Points: S/w break points are intrusive. Architecture has to provide for a set of hardware registers- for data and instruction, and the logic to compare them with the operands of the current instruction.

Architectural Additions Continued… 4-Architectural Support for Abstractions: Abstractions to be incorporated into the instruction sets. The processor would hence fewer and more meaningful messages, and therefore lesser mapping will be required. 5-Dedicated Bus: Interface that allows the processor to communicate with the external world without interfering with the system under test. Should be detachable, required for deployment. Architectural Additions are costly in both time and space.

Run-Time System Additions RTS requirements describe an interface between a tool and the underlying system: logical interface requiring substantial hardware support. Goal is to minimize the data and computational requirements of the internal debugger as well as the required communication between the internal and external debuggers. MRTSI and CIFO implements the abstractions as per a standard.

Run-Time System Additions continued.. The CIFO/MRTSI provide abstractions for: Processes Interrupt Management Time Management Memory Management Exception/Fault Handling RTS additions are minimal.

Conclusions With the RTS and Architectural additions, a solution to the problems in embedded system testing can be addressed. Testing and Debugging of embedded, real time software remains a “black art”, with ad hoc methods and techniques. Evaluation of the additions and determination of their feasibility is to be done.

Strengths Emphasis on the requirement of detection of the most difficult to fix errors as early as possible during the testing phases. Appropriate Caution while using software solutions, has been advised. The foresight on the effects of certification on test case selection is thought provoking. An Elegant model for debugging suggested.

Weakness The costs of the additions have been neglected. The authors have identified the need to study the same for the feasibility of the model, though. It is not very clear as to how the Hardware break points would not require computation cycles as the software break points do. The same applies to the internal debugger.