1. Real Time Fault Injection Using Enhanced OCD A Performance Analysis DFT 2006 André V. Fidalgo 1,2 Gustavo R. Alves 1 José M. Ferreira 2 anf@isep.ipp.ptgca@isep.ipp.ptjmf@fe.up.pt 1 Instituto Superior de Engenharia do Porto 2 Faculdade de Engenharia da Universidade do Porto

2. Presentation Overview Background -Real Time Fault Injection -The NEXUS Standard -COTS Fault Injection System Case Study -Debugger -OCD Infrastructure -FI Module -Scenarios -MethodologyResults -Performance -Logic Overhead -AnalysisConclusions

3. Real Time Fault Injection Real Time Systems –Must generate adequate responses within a specified window of time –Debug and fault injection should be performed with target system running at full speed RTFI Requirements: –On-Line Read / Write access to fault targets (memory / registers) –Internal (Watchpoint) or External Triggering RTFI Limitations: –Modification of memory contents may take several milliseconds. –Available debuggers cannot react automatically to watchpoint signals or trace data (and add some delays)

4. The NEXUS Standard Debug FeatureClassUsability for FI Run-Control1External Triggering Breakpoints1Internal Triggering Watchpoints1 Real Time Triggering Static Register and Memory Access 1 Static Fault Insertion Program Trace2 Fault Effects Diagnosis Dynamic Register and Memory Access 3 Real Time Fault Insertion Data Trace3 Enhanced Fault Effects Diagnosis On-Chip Debug (OCD): –Specific debugging circuitry on the target chip –Implements common debug functions (run control, breakpoints, program trace, etc…) –May include access to registers and memory NEXUS 5001: –A IEEE proposed standard for OCD (www.nexus5001.org) –Defines the feature set, interface port and communication protocol –Implementations can adapt to four levels of compliance Fault Injection use of Debug Features by Compliance Class

5. COTS Fault Injection System Components: -MPC565  P -ISystems IC3000 Debugger -Winidea 2005 software for debugCapabilities: -Debugger and target are both NEXUS compliant -Trace data output (by OCD) during program execution -Memory can be accessed via OCD in real timeLimitations: -Fault campaigns must be controlled by the host machine -Long delays sometimes cause incorrect fault insertions

6. Case Study – Debugger Features & Commands: -INPUT RAM is preloaded with FI script -OUTPUT RAM stores data for analysis -Message Data In (MDI) bus is configurable (part of AUX) -Allows bandwidth customization CommandDescription HALTHalts target execution. RUNStarts or resumes the target microprocessor execution. RESETResets the target microprocessor. DRESETResets the debugger. DCONFIGConfigures the debugger WAITWaits for a number of clock cycles WAITFOR Waits for a specific message or a watchpoint hit signal from the target OCD, during a specific period of time. READRAMReads the contents of the memory cell WRITERAMWrites a byte of data to the memory cell READREGReads the contents of a register WRITEREGWrites a byte of data to the register

7. Case Study – OCD Infrastructure MQM (Message Queuing and Management) –Controls the OCD infrastructure –Translates commands from and into formatted messages (NEXUS compliant) –Messages use prioritized FIFOs RCT (Run Control and Trace) –Controls the CPU modes (RUN, HALT, DEBUG) –Breakpoint / Watchpoint detection –Gathers Trace information RWA (Read Write Access) –Allows read / write operations on the CPU registers and memory space –Memory access can be performed on-the-fly –Also used to configure OCD through specific registers

8. Case Study – FI Module Main Features -FI module integrated within OCD circuitry -Controls already implemented debug functions Basic Version -Faulty data is preloaded -Automatic fault insertion on watchpoint hit Plus Version -Faulty Data determined online -Bit Flip performed inside FI module (based on Data Mask)

9. Case Study - Scenarios FI Environment: -Target is 32 bit RISC  P -Debugger manages entire fault injection process - Watchpoints are set on the OCD by the debuggerConfigurations: -MDI2 and MDI8 scenarios differ on OCD input bandwidth -Predetermination (Predet) of the faulty value can be used to improve performance -Two versions of FI module (Basic and Plus) ScenarioOCD Input Bandwidth Predet FI Module MPC565READI1/2NoNone OCD only MDI22YesNone MDI2+2NoNone MDI88YesNone MDI8+8NoNone OCD-FI MDI2_FI2YesBasic MDI2_FI+2NoPlus MDI8_FI8YesBasic MDI8_FI+8NoPlus Fault Injection Scenarios

10. Case Study – Methodology NStepDescription 1Set Up Microprocessor is reset Fault injection script is uploaded to the debugger Watchpoint is set on the target 2 Fault Triggering Triggering condition can be: - An external signal received by the debugger - A watchpoint hit signaled by the OCD (to the debugger) 3 Fault Activation The debugger activates a memory write operation 4 Fault Insertion Debugger transmits to the OCD: - The target memory cell address - The data value to be written NStepDescription 1Set Up Microprocessor is reset Fault injection script is uploaded to the debugger Watchpoint is set on the target The fault injection mode is enabled and data is preloaded 2 Fault Triggering Triggering condition is: - A watchpoint hit signaled internally by the OCD-FI 3 Fault Activation OCD-FI activates a memory write operation 4 Fault Insertion OCD-FI inserts the faulty data into the target memory cell OCD Only OCD-FI Fault Injection Steps

11. Results – Performance Configuration Inconclusive Results (%) Delays (in CLK cycles) MAdderVSorterSetUpInsertion MDI24 %5 %2235 MDI2+5 %7 %2244 MDI82 %3 %69 MDI8+3 %4 %618 MDI2_FI0 % 572 MDI2_FI+1 % 574 MDI8_FI0 % 152 MDI8_FI+1 % 154 ConfigurationReal TimeHalted Access MDI2+454 k365 k MDI8+1250 k1150k MDI2_FI+491 k483k MDI8_FI+1578 k1500k Fault Injection Process Results Maximum Fault Rates (Faults / Second) Execution: -All scenarios tested using severall fault campaigns -Matrix Adder and Vector Sorter algorithms used as workloads -Inconclusive results (INC) mostly caused by CPU access during FI process -Set Up executed prior to fault injection operations -Fault insertion performed with target running

12. Results – Logic Overhead Tools –Synthesized with Xilinx ISE 7.1i and simulated with Modelsim Synthesis Results –The OCD imposes a rather large overhead as the used CPU is simplistic (few instructions and registers) –The FI module imposes a very low overhead on the OCD circuitry (about 0,5% of the OCD area) CPU CoreOCDFI Module AreaOverheadMax f Eq Gates%MHz x-- 5392675,437 xMDI2-71524100,036 xMDI2Basic71602100,136 xMDI2Plus71842100,435 xMDI8-76127106,436 xMDI8Basic76202106,536 xMDI8Plus76436106,735 Equivalent Gate Counts by Scenario

13. Results - Analysis Fault Injection –Faults can be inserted in memory space with absolute precision without halting the target system –Higher bandwidths reduce inconclusive results occurrence –Fault injection on internal registers requires halting the targetSynthesis –The OCD overhead decreases for more complex CPUs –The customized debugger core is reasonably small Scenario Comparison –OCD only configurations can be applied to most compliant devices and depend on performance vs overhead tradeoffs –OCD-FI improves RTFI performance with minimum overhead –If using OCD-FI the extra bandwidth is not required for RTFI

14. Conclusions Advantages: –Efficient fault injection campaigns on compliant targets –Real time fault injection capabilities –Acceptable overheads overall –Complete controllability and good observabilityLimitations: –Requires compliant or adaptable targets –Some limitations on coverage Ongoing Research: –Testing with different target architectures –Improving fault coverage –Extensions to deal with hardware fault tolerance