1. Fault-Tolerant Platforms for Automotive Safety-Critical Applications Baver Şahin 2006701344

2. Agenda Introduction Introduction Fault-Tolerance in SOC Fault-Tolerance in SOC Fault-Tolerant Multi-Processor Architectures Fault-Tolerant Multi-Processor Architectures SOC Fault-Tolerant Architecture Implementation SOC Fault-Tolerant Architecture Implementation Implementation Issues & Comparisons Implementation Issues & Comparisons Concluding Remarks Concluding Remarks

3. Introduction Electonics in the car Electonics in the car - In the late 70’s - In the late 70’s - digitally controlled combustion engines - digitally controlled combustion engines - digitally controlled anti-lock brake systems(ABS) - digitally controlled anti-lock brake systems(ABS) - Synergy between mechanics and electronics - Synergy between mechanics and electronics - better fuel economy - better fuel economy - better vehicle performance - better vehicle performance - driver assisting functions (ABS, TCS, ESP, BA & safety features) - driver assisting functions (ABS, TCS, ESP, BA & safety features)

4. Introduction - X-by-wire systems: To design cars with better performance and higher level of safety, engineers must substitute mechanical interfaces between the driver and the vehicle with electronic systems. - throttle pedal, brake pedal, gear selector, steering wheel - electrical output is processed by micro-controllers that manage the power-train, braking and steering activities via electrical actuators.

5. Introduction - An example of a Brake-by Wire system: It consists of several computer nodes controlling various sensors and actuators that communicate through a fault tolerant real time network, and form together a distributed real-time computer system.

6. Introduction Fault-Tolerance Requirements: Because of the fact that drive-by-wire systems have no mechanical backup, they are assigned a high Safety Integrity Level. This means that their design must incorporate all the necessary techniques for achieving fault-tolerance. Fault-Tolerance Requirements: Because of the fact that drive-by-wire systems have no mechanical backup, they are assigned a high Safety Integrity Level. This means that their design must incorporate all the necessary techniques for achieving fault-tolerance. Fault-Tolerant Design Approaches Fault-Tolerant Design Approaches - hardware - hardware redundancy: 1) Static redundancy that is based on the voting of the outputs of a number of modules to mask the effects of a fault within these units. The simplest form of this arrangement consists of three modules and a voter and is termed a triple modular redundant system (TMR). 2) Dynamic redundancy on the other hand is based on fault detection rather than fault masking. This is achieved by using two modules and some sort of comparison on their outputs that can detect possible faults. This method has lower component count but is not suitable for real-time applications. 2) Dynamic redundancy on the other hand is based on fault detection rather than fault masking. This is achieved by using two modules and some sort of comparison on their outputs that can detect possible faults. This method has lower component count but is not suitable for real-time applications. 3) Hybrid redundancy uses a combination of voting, fault-detection and module switching, thus combining static and dynamic redundancy. 3) Hybrid redundancy uses a combination of voting, fault-detection and module switching, thus combining static and dynamic redundancy.

7. Fault-Tolerance in SOC New trends in the automotive industry, like the New trends in the automotive industry, like the development of drive-by-wire systems, have generated the need for computer systems with high levels of fault tolerance and also low cost. This can be achieved by using system-on-chip (SoC) design methods. Common mode failures Common mode failures - clock tree - power supply - silicon substrate

8. Fault-Tolerance in SOC Experienced Faults Experienced Faults - hard fails: permanent failures that are caused by an irreversible physical change and derive from the term ‘hardware failure’. - hard fails: permanent failures that are caused by an irreversible physical change and derive from the term ‘hardware failure’. soft fails (single event upsets SEU): Soft fails (or soft errors) are defined as a spontaneous error or change in stored information which cannot be reproduced. - soft fails (single event upsets SEU): Soft fails (or soft errors) are defined as a spontaneous error or change in stored information which cannot be reproduced. - external electronic noise - nuclear particles that come either from the decay of radioactive atoms

9. Fault-Tolerance in SOC While the occurrence of a permanent fault may impair or While the occurrence of a permanent fault may impair or even stop the correct functionality of the system, soft errors caused by transient faults often drastically reduce the system availability. As a matter of fact, it is often the case that soft error avoidance is strongly required to maintain the system availability at an acceptable level. - static temporal redundancy - static temporal redundancy - triple execution and majority voting - triple execution and majority voting - mask any single soft error - mask any single soft error - dynamic technique duplication and comparison - duplication and comparison - deploying error detection - deploying error detection

10. Fault-Tolerance in SOC While the error detection drastically simplifies the system While the error detection drastically simplifies the system roll-back and restart, error masking eliminate (or at least reduce) this need thus maintaining the provided availability at an acceptable level.

11. Fault-Tolerant Multi-Processor Architectures Lock-Step Dual Processor Architecture: Lock-Step Dual Processor Architecture:

12. Fault-Tolerant Multi-Processor Architectures Lock-Step Dual Processor Architecture: Lock-Step Dual Processor Architecture: two processors (master & checker): execute the same code being strictly synchronized. - two processors (master & checker): execute the same code being strictly synchronized. - master: has access to the system memory and drives all system outputs. - master: has access to the system memory and drives all system outputs. - checker: continuously executes the instructions moving on the bus (i.e. those fetched by the master processor)

13. Fault-Tolerant Multi-Processor Architectures - compare logic (monitor): consisting of a comparator circuit at the master’s and checker’s bus interfaces, that checks the consistency of their data-address- and control-lines. The detection of a disagreement on the value of any pair of duplicated bus lines reveals the presence of a fault on either CPU without giving the chance to identify the faulty CPU. - compare logic (monitor): consisting of a comparator circuit at the master’s and checker’s bus interfaces, that checks the consistency of their data-address- and control-lines. The detection of a disagreement on the value of any pair of duplicated bus lines reveals the presence of a fault on either CPU without giving the chance to identify the faulty CPU. - source of common-mode failure: bus and memory errors - source of common-mode failure: bus and memory errors - error detection (correction) techniques - error detection (correction) techniques - parity bits - parity bits

14. Fault-Tolerant Multi-Processor Architectures The lock-step architecture can be employed as a fail- silent node providing the capability of detecting any (100% coverage) single error (permanent or transient) occurring indifferently on the CPU, memory or communication sub-system. Error correcting codes are required when errors occurring on busses and memories turn out to be relatively frequent due to the occurrence of transient faults. The lock-step architecture can be employed as a fail- silent node providing the capability of detecting any (100% coverage) single error (permanent or transient) occurring indifferently on the CPU, memory or communication sub-system. Error correcting codes are required when errors occurring on busses and memories turn out to be relatively frequent due to the occurrence of transient faults.

15. Fault-Tolerant Multi-Processor Architectures Loosely-Synchronized Dual Processor Architecture: Loosely-Synchronized Dual Processor Architecture:

16. Fault-Tolerant Multi-Processor Architectures Loosely-Synchronized Dual Processor Architecture: Loosely-Synchronized Dual Processor Architecture: - two CPU’s: run independently having access to distinct memory subsystems. - two CPU’s: run independently having access to distinct memory subsystems. - A real-time operating system running on both CPUs - A real-time operating system running on both CPUs - interprocessor communication - interprocessor communication - synchronization - synchronization - error detection (e.g. by means of cross-checks), correction and containment (e.g. memory protection) - error detection (e.g. by means of cross-checks), correction and containment (e.g. memory protection) - A subset of the tasks executed by the processors are defined as critical. The image of critical tasks is duplicated on both memories. Critical tasks are executed in parallel as software replicas and their outputs are exchanged after each run on a time triggered basis. Both processors are responsible for checking their consistency. - A subset of the tasks executed by the processors are defined as critical. The image of critical tasks is duplicated on both memories. Critical tasks are executed in parallel as software replicas and their outputs are exchanged after each run on a time triggered basis. Both processors are responsible for checking their consistency.

17. Fault-Tolerant Multi-Processor Architectures A mismatch: indicates a fault on the CPU, memory or communication sub-system and prevents outputs from being committed. - A mismatch: indicates a fault on the CPU, memory or communication sub-system and prevents outputs from being committed. - cross-check mismatch - sanity-check - self-testing commitment of agreed outputs - commitment of agreed outputs - First technique: to prevent outputs from being committed before being cross-checked, time guardians can restrict CPU access to system outputs to a predefined time-window. - First technique: to prevent outputs from being committed before being cross-checked, time guardians can restrict CPU access to system outputs to a predefined time-window.

18. Fault-Tolerant Multi-Processor Architectures - Second Technique: Each processor adds its own signature to the outputs of critical tasks and the receiver checks for both signatures before accepting the data. - Second Technique: Each processor adds its own signature to the outputs of critical tasks and the receiver checks for both signatures before accepting the data. - According to the subset of critical task, the architecture - According to the subset of critical task, the architecture can appear in several different configurations. At the one end, fully critical applications must be entirely replicated, thus requiring twice as much memory while providing the same performance as a single processor architecture.

19. Fault-Tolerant Multi-Processor Architectures The execution of a function on both CPUs guarantees the detection of any error (100% coverage) occurring indifferently on one of the CPUs, busses or memories. Since busses and memories (at least for critical tasks) are replicated, no other form of redundancy (e.g. parity bits) is needed to detect errors on these components. Nevertheless, ECCs may be employed in the case of high memory (or bus) failure rate. The execution of a function on both CPUs guarantees the detection of any error (100% coverage) occurring indifferently on one of the CPUs, busses or memories. Since busses and memories (at least for critical tasks) are replicated, no other form of redundancy (e.g. parity bits) is needed to detect errors on these components. Nevertheless, ECCs may be employed in the case of high memory (or bus) failure rate.

20. Fault-Tolerant Multi-Processor Architectures Triple Modular Redundant (TMR) Architecture: Triple Modular Redundant (TMR) Architecture:

21. Fault-Tolerant Multi-Processor Architectures Triple Modular Redundant (TMR) Architecture: Triple Modular Redundant (TMR) Architecture: - three identical CPUs: execute the same code in lock- step. - majority voter: majority vote of the outputs masks any possible single CPU fault. The memory and communication sub-system faults can be masked employing ECC (Error Correcting Codes) techniques. The memory and communication sub-system faults can be masked employing ECC (Error Correcting Codes) techniques.

22. Fault-Tolerant Multi-Processor Architectures Dual Lock-Step Architecture: Dual Lock-Step Architecture:

23. Fault-Tolerant Multi-Processor Architectures Dual Lock-Step Architecture: A configuration largely employed in multi-chip fault-tolerant systems consists of the combination of two fail-silent channels, each one consisting of a lock-step architecture as the one presented in Lock-Step Dual Processor Architecture, building up a single fail-operational unit. In this case, the architecture provides fault-tolerance only for the replicated tasks, whose outputs are checked before being committed. Dual Lock-Step Architecture: A configuration largely employed in multi-chip fault-tolerant systems consists of the combination of two fail-silent channels, each one consisting of a lock-step architecture as the one presented in Lock-Step Dual Processor Architecture, building up a single fail-operational unit. In this case, the architecture provides fault-tolerance only for the replicated tasks, whose outputs are checked before being committed. Software design errors can be prevented as well. Software design errors can be prevented as well.

24. Fault-Tolerant Multi-Processor Architectures In contrast to solution presented in Loosely- Synchronized Dual Processor Architecture, the execution of sanity-checks is no more required, since self-checking capabilities are already provided in hardware by means of duplication and yield a 100% fault coverage. In contrast to solution presented in Loosely- Synchronized Dual Processor Architecture, the execution of sanity-checks is no more required, since self-checking capabilities are already provided in hardware by means of duplication and yield a 100% fault coverage.

25. SOC Fault-Tolerant Architecture Implementation Cost: Due to the costs associated to the higher integration level, single-chip implementations should have enough flexibility to support a wide range of applications in order to share the silicon development cost across a set of different final electronic systems. Cost: Due to the costs associated to the higher integration level, single-chip implementations should have enough flexibility to support a wide range of applications in order to share the silicon development cost across a set of different final electronic systems. Flexibility: the capability of a silicon solution to correctly adapt to performance, cost and fault-tolerance requirements of a set of applications, after silicon production. Flexibility: the capability of a silicon solution to correctly adapt to performance, cost and fault-tolerance requirements of a set of applications, after silicon production.

26. SOC Fault-Tolerant Architecture Implementation In contrast to multi-chip solutions, in a single-chip dual- processor architecture the memory sub-system can be shared between the processors at much lower cost. Since the two cores can run independently, the memory and communication sub-systems are likely to become a major performance bottleneck. For this reason the memory sub-system is split into 4 banks (2 for code and data respectively) and the traditional bus is replaced by a more performant crossbar switch, which guarantees sufficient bandwidth between the processor and memory subsystems. In contrast to multi-chip solutions, in a single-chip dual- processor architecture the memory sub-system can be shared between the processors at much lower cost. Since the two cores can run independently, the memory and communication sub-systems are likely to become a major performance bottleneck. For this reason the memory sub-system is split into 4 banks (2 for code and data respectively) and the traditional bus is replaced by a more performant crossbar switch, which guarantees sufficient bandwidth between the processor and memory subsystems.

27. SOC Fault-Tolerant Architecture Implementation The single-chip loosely-synchronized dual The single-chip loosely-synchronized dual processor architecture, called Shared-Memory (SM) Loosely Synchronized Dual-Processor: Since the memory sub-system is shared between the processors, the duplication of critical code becomes a trade-off between system integrity, memory size and performance: while critical code takes up costly memory space, non-duplicated critical code, which must be executed on both cores, runs at half the speed of a single processor.

28. SOC Fault-Tolerant Architecture Implementation Shared-Memory (SM) Loosely Shared-Memory (SM) Loosely Synchronized Dual Processor Architecture:

29. SOC Fault-Tolerant Architecture Implementation SM Dual Lock-Step architecture: The two fail-silent SM Dual Lock-Step architecture: The two fail-silent channels share the same memory sub-system. This solution largely enhances flexibility, since it covers the TMR solution (same fault-tolerance properties), while implementing the dual lock-step architecture. Lock-Step mode: When fail-operational capability is Lock-Step mode: When fail-operational capability is required, the two channels can be arranged in lock-step mode, in which case the architecture provides masking capabilities of CPU’s faults as in the TMR solution.

30. SOC Fault-Tolerant Architecture Implementation Parallel Mode: Two channels can be used as two Parallel Mode: Two channels can be used as two completely parallel fail-silent channels providing double performance. Memories and buses are protected using ECCs in order Memories and buses are protected using ECCs in order to retain error masking capabilities on these components when operating in lock-step mode.

31. SOC Fault-Tolerant Architecture Implementation SM Dual Lock-Step architecture: SM Dual Lock-Step architecture:

32. Implementation Issues & Comparisons The performance and the fault-tolerance features of the The performance and the fault-tolerance features of the different solutions are compared and their costs are evaluated on the basis of the area estimates. Table summarizes the area of memory components Table summarizes the area of memory components (both RAM and FLASH) and buses, normalized to the CPU footprint. Area of embedded memory components normalized to CPU footprint

33. Implementation Issues & Comparisons Cost of different architectures for low-/mid-range X-by-wire systems

34. Implementation Issues & Comparisons The single CPU architecture can be considered as a The single CPU architecture can be considered as a reference design satisfying computational and memory requirements but not providing any fault-tolerance capability. The lockstep architecture: The lock-step architecture The lockstep architecture: The lock-step architecture cannot provide any performance boost over the single processor solution, since the two cores are bound to execute the same code cycle by cycle. Rather, due to the introduction of the compare logic and the ECC coders/decoders in the critical path, the clock rate may be decreased.

35. Implementation Issues & Comparisons However, with a relatively low area overhead, this However, with a relatively low area overhead, this solution provides a 100% fault coverage within an error detection time in the order of the clock period. Both processors execute the same code, the lockstep Both processors execute the same code, the lockstep configuration does not provide any protection against software design errors.

36. Implementation Issues & Comparisons SM loosely-synchronized dual-processor SM loosely-synchronized dual-processor Architecture: In the SM loosely-synchronized dual Architecture: In the SM loosely-synchronized dual processor architecture the two CPUs can run independently having full access to the memory sub-system and system I/O. Since only critical tasks must be duplicated for safety Requirements. As the lock-step configuration, the SM loosely As the lock-step configuration, the SM loosely synchronized architecture provides a 100% error detection when running full-critical applications. However, this requires roughly twice as much memory space to accommodate the duplicated code. Memory footprint is mostly responsible for the huge area overhead as shown in Table.

37. Implementation Issues & Comparisons Moreover, fault diagnosis is complicated by the longer Moreover, fault diagnosis is complicated by the longer error detection time, proportional to the check execution period, and by the fact that error detection only performed on selected outputs. Nonetheless, in contrast to the lock step solution, the SM loosely-synchronized architecture has the ability of supporting both hardware and software design diversity and provides a degraded mode of operation. Both configurations presented above provide no fault masking mechanism, except for the possible implementation of ECCs on buses and memories. This may be a major draw-back especially in the case of a high transient fault rate.

38. Implementation Issues & Comparisons Triple modular redundant architecture: The TMR Triple modular redundant architecture: The TMR configuration represents a “low-cost” solution. In fact, the area overhead over the lock-step architecture is as low as 9% and 1.5% for low- and mid-range systems respectively. However, it also inherits almost all of the features and flaws of the lock-step architecture. Excepting its unique capability of masking any single fault, at the cost of an additional CPU, it offers a 100% error detection coverage within a single clock period.

39. Implementation Issues & Comparisons SM dual lockstep architecture: The SM dual lock SM dual lockstep architecture: The SM dual lock step architecture combines the advantages of the SM loosely-synchronized solution in terms of flexibility with the fault masking capabilities provided by the TMR architecture. When the two cores execute the same code in lock-step, When the two cores execute the same code in lock-step, they provide fault-tolerance capabilities. On the other hand, if the fail-silence property suffices for the application at hand, the two channels can operate completely independently and the architecture behaves like a “traditional” dual processor solution.

40. Implementation Issues & Comparisons This great deal of flexibility comes at a relatively low This great deal of flexibility comes at a relatively low price. In fact, if compared with the fault-tolerant TMR architecture, while the introduction of the 4th CPU yields a 10% overhead for low-range applications, the overhead falls down to just 2-3% for more memory demanding applications. Notice that to cover software design faults2 via design diversity, we need to double the memory footprint as done for the SM loosely-synchronized architecture. Also in this case, comparing the two alternatives, we come out with a modest increase in area, in the order of about 8% and 2% for low- and mid-range applications respectively.

41. Implementation Issues & Comparisons Tradeoff analysis: Tradeoff analysis: - SM loosely-synchronized architecture most area-demanding solution - most area-demanding solution - lock-step and the TMR architectures - lock-step and the TMR architectures - cannot provide any performance improvement over the single processor solution, while representing “low-cost” solutions SM dual lock-step architecture - SM dual lock-step architecture - 100% single fault-tolerance - 100% single fault-tolerance - wider range of applications - wider range of applications - reducing engineering costs - reducing engineering costs - best alternative between the four architectures - best alternative between the four architectures

42. Concluding Remarks A single-chip solution is proposed, devised for fault A single-chip solution is proposed, devised for fault tolerant automotive applications, which is based on the use of two lock-step channels (4 CPUs overall), a cross-bar communication architecture and embedded memories.

43. References [1] R. Baumann. The impact of technology scaling on soft error rate performance and limits to the efficacy of error correction. In Digest of the Internation Electron Devices Meeting IEDM’02., pages 329–332, 2002. [1] R. Baumann. The impact of technology scaling on soft error rate performance and limits to the efficacy of error correction. In Digest of the Internation Electron Devices Meeting IEDM’02., pages 329–332, 2002. [2] R.C. Baumann. Soft errors in advanced semiconductor devices - part I: The three radiation sources. IEEE Transaction on Device and Materials Reliability, 1(1):17–22, Mar 2001. [2] R.C. Baumann. Soft errors in advanced semiconductor devices - part I: The three radiation sources. IEEE Transaction on Device and Materials Reliability, 1(1):17–22, Mar 2001. [3] E. B¨ohl, Th. Lindenkreuz, and R. Stephan. The fail-stop controller AE11. In Proceedings of the International Test Conference, pages 567–577, Nov 1997. [3] E. B¨ohl, Th. Lindenkreuz, and R. Stephan. The fail-stop controller AE11. In Proceedings of the International Test Conference, pages 567–577, Nov 1997. [4] M. Baleani, A. Ferrari, L. Mangeruca, Maurizio Peri, Saverio Pezzini. Fault-Tolerant Platforms for Automotive Safety Critical Applications In: Proceedings of the 2003 international conference on Compilers, architecture and synthesis for embedded systems, pages 170 – 177, 2003. [4] M. Baleani, A. Ferrari, L. Mangeruca, Maurizio Peri, Saverio Pezzini. Fault-Tolerant Platforms for Automotive Safety Critical Applications In: Proceedings of the 2003 international conference on Compilers, architecture and synthesis for embedded systems, pages 170 – 177, 2003. [5] R. Iserman, R. Schwarz, and S. St¨olzl, “Fault-Tolerant Drive-by-Wire Systems,” IEEE Control Systems Magazine, vol. 22, no. 5, pp. 64–81, October 2002. [6] K. Ahlstr¨om and J. Torin, “Future Architecture of Flight Control Systems,” IEEE Aerospace and Electronic Systems Magazine, vol. 17, no. 12, pp. 21–27, December 2002. [7] P. H. Jesty, K. M. Hobley, R. Evans, and I. Kendall, “Safety Analysis of Vehicle-Based Systems,” in Proceedings of the Eighth Safety-critical Systems Symposium, 2000, pp. 90–110. [8] C. Constantinescu, “Trends and Challenges in VLSI Circuit Reliability,” IEEE Micro, vol. 23, no. 4, pp. 14–19, July-August 2003.

44. THANKS THANKSQ&A