CS, AUHenrik Bærbak Christensen1 Fault Tolerant Architectures Lyu Chapter 14 Sommerville Chapter 20 Part II

CS, AUHenrik Bærbak Christensen2 Application domains Fault tolerant systems are used in various domains, primarily for safety-critical systems –First documented example is for railway systems (1978) –Nuclear power plants –Airplanes (Airbus) –Space program

CS, AUHenrik Bærbak Christensen3 Experience Somewhat mixed, actually… Why? –Redundancy works for hardware because hardware often fails randomly Due to wearing out (component failure, not design failure) –Software fails due to specification and design errors Thus simple replication does not provide protection… Review the Ariane failure reported by Tan Redundant software units require diversity –However, there are evidence of failure correlations even over diverse implementations…

CS, AUHenrik Bærbak Christensen4 The origins: Hardware Triple Modular redundancy (TMR) –Three identical hardware units process input –Output is compared for equality Deviating output is ignored Fault manager may try repair, or reconfigure to take unit out of service

CS, AUHenrik Bærbak Christensen5 Terminology for Software FT Principal requirement –Redundancy of functional equivalent but different software units different teams, tools, processes, … –Oracle Method to dynamically determine if output is correct or within acceptable limits –Recovery Defect will lead to error state that leads to failure if not handled A detected error state, results in recovery being initiated

CS, AUHenrik Bærbak Christensen6 Terminology for Software FT Recovery –Backward Recovery Recovery points are stored during normal execution System rolled back/restored to a previous restore point and restarted from that –Forward Recovery Transition into degraded mode state which is functional but quality is lowered Or: error compensation in which algorithms derive the correct answer. Exercise: –Give examples of each type

CS, AUHenrik Bærbak Christensen7 Oracles Result verification/Dynamic self-checking Acceptance test –Internal accept test Test for correctness, or if answer is within limits or bounds Require that testing correctness is easier than calculating the result, like |sqr(x)*sqr(x) – x| < E Examples –Checksums, used to accept test datagram contents –Data structure validation methods –Hardware self tests

CS, AUHenrik Bærbak Christensen8 Oracles Result verification External consistency: –Uses additional knowledge outside of the unit producing results Examples –Watchdogs (heartbeat in Bass) use timings to detect and resolve problems –Exceptions: for instance floating point errors

CS, AUHenrik Bærbak Christensen9 Diversity The rationale behind diversity: Modules fail on disjoint subsets of the input space – one will always process input correct! Program 1 execution state I_e error states Input space Program 2 execution state I_e error states Input space

CS, AUHenrik Bærbak Christensen10 Redundancy Require a software unit to judge acceptability of redundant modules: adjudicator –As it is a software unit – it may contain defects –Techniques Voting Median value Acceptance testing And more…

CS, AUHenrik Bærbak Christensen11 Failure classes In unit testing, failures occur because of defects in the software unit. –A test case either fail or pass A redundant system (= N functionally identical but different units) introduces more types/classes of failures –k-fold coincident failures (sammenfaldende) k out of N units fails on the same test case –U1 says 7, U2 says 13, but answer is 42. Identical-and wrong (IAW) answer –U1 and U2 says 7, but answer is 42

CS, AUHenrik Bærbak Christensen12 Failure classes Correlated/Dependent failures –P( U1 fails | U2 fails) ≠ P( U1 fails ) Hvis sandsynligheden for at U1 fejler på en test case givet at vi ved U2 fejler på test cases er forskellig fra ss for at U1 fejler givet at vi ikke ved om U2 fejler på test casen. Tænk på at U1 er lig U2. Hvis vi ved U2 fejler og U1 er identisk med U2 så ved vi sten sikkert at U1 fejler: P(U1 fails | U2 fail) = 1. Men hvis vi ved at U1 er identisk med U2 men ikke om U2 vil fejle, så kender vi kun fordelingen som måske er at SS for at U1 fejler er 0,1%.

CS, AUHenrik Bærbak Christensen13 Failure classes

CS, AUHenrik Bærbak Christensen14 Adjudication Techniques

CS, AUHenrik Bærbak Christensen15 Voting Majority voting –m = number of matching outputs –m = ceil[(n+1)/2] –Usually N = 3 which means m = ? Two-out-of-N voting –Actually m = 2 is enough regardless of N (usually) –Note: agreement ≠ correctness –Best argument I have: Hitler was democratic elected Median voting –Sort the answers and select the middle element –Used in aerospace …

CS, AUHenrik Bærbak Christensen16 Voting Consensus voting

CS, AUHenrik Bærbak Christensen17 Redundancy Techniques

CS, AUHenrik Bærbak Christensen18 Recovery Blocks Failed Accept Test –Often roll-back / recovery of system state –Single processors suffers from sequential processing ‘core dumped’ in first unit is bad…

CS, AUHenrik Bærbak Christensen19 N-version Programming Executed in parallel –Voting used to select proper answer

CS, AUHenrik Bærbak Christensen20 Variants Lyu discuss various variants and combinations. One I find interesting is Acceptance voting –N versions execute in parallel, and the answers are subjected to acceptance testing. –Only accepted answers are then feed to the voter –Voter must be dynamic as the number of inputs, Ni <= N, to the voter varies according to the number of accepted outputs.