# FIT5174 Distributed & Parallel Systems

## Presentation on theme: "FIT5174 Distributed & Parallel Systems"— Presentation transcript:

FIT5174 Distributed & Parallel Systems
Lecture 6 Election Algorithms, Distributed Concurrent Transactions, Fault Tolerance FIT5174 Parallel & Distributed Systems Dr. Ronald Pose Lecture

These slides are based on slides and material by: Carlo Kopp
September 4, 1997 Acknowledgement These slides are based on slides and material by: Carlo Kopp FIT5174 Parallel & Distributed Systems Dr. Ronald Pose Lecture 2

Election Algorithms September 4, 1997 Distributed algorithms often elect one process to acts as a coordinator or a server. Election Assumptions We assume each process is assigned a unique process number A process knows all process numbers A process does not know which process number is alive. An Election: Select the maximum process number among active processes Appoint the process the coordinator. All the processes must agree upon this decision. FIT5174 Parallel & Distributed Systems Dr. Ronald Pose Lecture 3

The Bully Algorithm (Garcia-Molina)
September 4, 1997 When a process P notices that the coordinator is no longer responding to requests, it initiates an election: P sends an Election message to all processes with higher numbers. If one of the higher numbered answers, that process becomes the new leader. If no one responds, P wins the election and becomes coordinator. Timeout overhead Delayed response FIT5174 Parallel & Distributed Systems Dr. Ronald Pose Lecture 4

The Bully Algorithm September 4, 1997 At any moment, a process can get an Election message from one of its lower-numbered colleagues. The receiver sends an OK message back to the sender. The receiver then holds an election, unless it is already holding one. Eventually, all processes give up but one, and that one is the new coordinator. It announces its victory by sending Coordinator messages to all processes. If a process that was previously down comes back up, it holds an election. If it happens to be the highest-numbered process currently running, it will win the election and take over the coordinator's job. The highest numbered process always wins, thus “bully algorithm”. FIT5174 Parallel & Distributed Systems Dr. Ronald Pose Lecture 5

A Ring Algorithm September 4, 1997 Processes form a logical ring. Each process knows which process comes after it. If a process finds the coordinator dead, it sends an Election message which contains its own process number to its successor in the ring. If the successor is down, the message is sent to the next process along the ring. Each process appends its process number to the message upon receiving an Election message and forward it to the successor. When the message returns to the process that originally sent the message, the process changes the message type from Election message to Coordinator message and circulates the message again. The purpose of the Coordinator message is to inform all processes that the highest-numbered process is the new coordinator and the processes contained in the message are the members of the ring. FIT5174 Parallel & Distributed Systems Dr. Ronald Pose Lecture 6

Distributed Transactions
September 4, 1997 Client X Y Z M N T 1 2 11 P 12 21 22 Simple Distributed Transaction Nested Distributed Transaction FIT5174 Parallel & Distributed Systems Dr. Ronald Pose Lecture 7

Distributed Atomic Transactions
September 4, 1997 An atomic transaction is such, that either all in a set of distributed operations are completed or none of them are completed. First a process declares to all other processes (involved in a transaction) that it is starting the transaction. The processes exchange information and perform their specific operations. The initiator announces that it wants all the other processes to “commit” to all the operations done up to that point. If all processes agree to commit, the results are made permanent. If at least one of the processes refuse, all the states - usually opened files/databases (on all machines) - are reversed to the point before starting the transaction. FIT5174 Parallel & Distributed Systems Dr. Ronald Pose Lecture 8

Transaction Processing
September 4, 1997 System Model for a Transaction A system consists of independent processes where each process may fail at random. No communication error may be encountered. A “stable storage” device is available. “Stable storage” guarantees no data is lost unless the storage is physically lost by disasters such as floods or earthquakes. Note: Data in dynamic RAM disappears when power is turned off. Data on a hard disk becomes lost if the disk head crashes, or other hardware fails. Thus neither are “stable storage” devices. FIT5174 Parallel & Distributed Systems Dr. Ronald Pose Lecture 9

Stable Storage – Mirrored Disks
September 4, 1997 Stable storage can be implemented with a pair of disks. Each block on drive 2 is a copy of the corresponding block on drive 1. When a block is updated, the block on drive 1 is first updated and verified. Then, the same block on drive 2 is updated and verified. If the system crashes after the update of drive 1 but before the update of drive 2, the blocks on the two drives are compared and the blocks on drive 1 are copied to drive 2 after system reboot. If checksum error is encountered because of a bad block, the block is copied from the disk without an error. FIT5174 Parallel & Distributed Systems Dr. Ronald Pose Lecture 10

Transaction Primitives
September 4, 1997 Programming a transaction requires special primitives that must be supplied by the OS or by the programming language. Examples of the primitives are: Begin Transaction: Specifies the beginning of a transaction. End Transaction: Specifies the end of a transaction, and tries to commit on the results. Abort Transaction: Abort the transaction and restore the old value. Read: Reads data from a file. Write: Writes data to a file. Begin Transaction and End Transaction specify the scope of a transaction. The operations delimited with the two operations form the transaction. These primitives are implemented as either system calls, library functions, or constructs of a programming language. FIT5174 Parallel & Distributed Systems Dr. Ronald Pose Lecture 11

Transaction Properties
September 4, 1997 Four essential properties: Atomic Consistent Isolated Durable FIT5174 Parallel & Distributed Systems Dr. Ronald Pose Lecture 12

Atomicity Property September 4, 1997 A transaction happens indivisibly to the outside world. (Either all operations of the transaction are completed or none is completed.) Other processes cannot observe the intermediate states of the transaction. FIT5174 Parallel & Distributed Systems Dr. Ronald Pose Lecture 13

Consistency Property A transaction does not violate system invariants.
September 4, 1997 A transaction does not violate system invariants. For example, an invariant in a banking system is the conservation law regarding the total amount of money. The sum of money in the source and destination accounts after any transfer is the same as the sum before the transfer (although the conservation law may be violated while the transfer is in progress). FIT5174 Parallel & Distributed Systems Dr. Ronald Pose Lecture 14

Isolation Property September 4, 1997 Even if more than one transactions are running simultaneously, the final result is the same as the result of running the transactions sequentially in some order. Transactions with this property are also called “Serializable.” FIT5174 Parallel & Distributed Systems Dr. Ronald Pose Lecture 15

Durability Property Once a transaction commits, changes are permanent.
September 4, 1997 Once a transaction commits, changes are permanent. No failure after the commit can undo the results or cause these to be lost. FIT5174 Parallel & Distributed Systems Dr. Ronald Pose Lecture 16

Nested Transactions September 4, 1997 Transactions may contain sub-transactions. Such transactions are called nested transactions. Problems with nested transactions Assume a transaction starts several sub-transactions in parallel. One of these commits and makes its results visible to the parent transaction. For some reason parent is aborted, the entire system needs to be restored to its original state. Consequently the results of the sub-transaction that were committed must be undone. Undoing a committed transaction is a violation. FIT5174 Parallel & Distributed Systems Dr. Ronald Pose Lecture 17

Nested Transactions Characteristics
September 4, 1997 Characteristics Permanence of a sub-transaction is only valid in the world of its direct parent and is invalid in the world of further ancestors. Permanence of the results is valid only for the outermost transaction. Addressing Transaction Violation Problem One way to implement nested transactions is to create a private copy of all objects in the system for each sub-transaction. If a sub-transaction commits, its private copy replaces its parent’s copy. FIT5174 Parallel & Distributed Systems Dr. Ronald Pose Lecture 18

Implementing Atomic Transactions
September 4, 1997 Implementing atomic transactions means Hiding of operations within a transaction from outer processes. Making the results visible only after commit. Restoring the old state if the transaction is aborted. Two ways to restore the old state: Private workspace Write-ahead log FIT5174 Parallel & Distributed Systems Dr. Ronald Pose Lecture 19

Private Workspaces September 4, 1997 When a process starts a transaction, it is given a private workspace containing all the objects including data and files. Read and write operations are performed within the private workspace. The data within the private workspace is written back when the transaction commits. The problem with this technique is that the cost of copying every object to a private workspace is high. The following optimisation is possible: If a process only reads an object, there is no need for a private copy. If a process updates an object, a copy of the object is made in the private workspace. FIT5174 Parallel & Distributed Systems Dr. Ronald Pose Lecture 20

Private Workspaces September 4, 1997 If we use indices to objects, we can reduce the number of copy operations. When an object is to be updated, A copy of the index and the objects is made in the private workspace. The private index is updated. On commit, the private index is copied to the parent’s index. FIT5174 Parallel & Distributed Systems Dr. Ronald Pose Lecture 21

Private Workspaces September 4, 1997 If the transaction aborts, the private copies of objects and the private indices are discarded. FIT5174 Parallel & Distributed Systems Dr. Ronald Pose Lecture 22

Write-ahead Log September 4, 1997 Before any changes to objects are made, a record (log) is written to a “write-ahead log” on stable storage. The log contains the following information which transaction is making the change, which object is being changed, and what the old and new values are. Only after the log has been written successfully, the change is made to the object. FIT5174 Parallel & Distributed Systems Dr. Ronald Pose Lecture 23

Write-ahead Log September 4, 1997 If the transaction committed, a commit record is written to the log (data has already been written). If the transaction aborts, the log can be used to rollback to the original state. The log can be used for recovering from crashes. FIT5174 Parallel & Distributed Systems Dr. Ronald Pose Lecture 24

Distributed Commit Protocol
September 4, 1997 Two-Phase Commit Protocol The action of committing a transaction must be done instantaneously and indivisibly. In a distributed system, the commit requires the cooperation of multiple processes (may be) on different machines. Two-phase commit protocol is the most widely used protocol to implement atomic commit. FIT5174 Parallel & Distributed Systems Dr. Ronald Pose Lecture 25

Two Phase Commit Protocol
September 4, 1997 One process functions as the coordinator. The other participating processes are subordinates (Cohorts). FIT5174 Parallel & Distributed Systems Dr. Ronald Pose Lecture 26

Two Phase Commit Protocol
September 4, 1997 The coordinator writes an entry labelled “prepare” in the log on a stable storage device and sends the other processes involved (the subordinates) a message telling them to prepare. When a subordinate receives the message, it checks to see if it is ready to commit, makes a log entry, and sends back its decision. After collecting all the responses, if all the processes are ready to commit, the transaction is committed. Otherwise, the transaction is aborted. The coordinator writes a log entry and then sends a message to each subordinate informing it of the decision. The coordinator’s writing commit log actually means committing the transaction and no rolling back occurs no matter what happens afterward. FIT5174 Parallel & Distributed Systems Dr. Ronald Pose Lecture 27

Two Phase Commit Protocol
September 4, 1997 This protocol is highly resilient in the face of crashes because it uses stable storage devices. If the coordinator crashes after writing a “prepare” or “commit” log entry, it can still continue from sending a “prepare” or “commit” message. If a subordinate crashes after writing a “ready” or “commit” log entry, it can still continue from sending a “ready” or “finished” message upon restart. FIT5174 Parallel & Distributed Systems Dr. Ronald Pose Lecture 28

Three Phase Commit Protocol
September 4, 1997 Assumptions each site uses the write-ahead-log protocol at most one site can fail during the execution of the transaction Basic Concept: Before the commit protocol begins, all the sites are in state q. If the coordinator fails while in state q1, all the cohorts perform the timeout transition, thus aborting the transition. Upon recovery, the coordinator performs the failure transition. FIT5174 Parallel & Distributed Systems Dr. Ronald Pose Lecture 29

Distributed System Concurrency Control
September 4, 1997 Distributed System Concurrency Control When several transactions run simultaneously in different processes, we must guarantee the final result is the same as the result of running the transactions sequentially in some order. That is, we need a mechanism to guarantee isolation or serializability. This mechanism is termed Concurrency Control. Three typical concurrency algorithms: Using locks Using timestamps Optimistic FIT5174 Parallel & Distributed Systems Dr. Ronald Pose Lecture 30

Distributed System Concurrency Control Using Locks
September 4, 1997 Distributed System Concurrency Control Using Locks This is the oldest and most widely used concurrency control algorithm. When a process needs to access shared resource as part of a transaction, it first locks the resource. If the resource is locked by another process, the requesting process waits until the lock is released. The basic scheme is overly restrictive. The algorithm can be improved by distinguishing read locks from write locks. Data only to be read (referenced) is locked in shared mode. Data need to be updated (modified) is locked in exclusive mode. FIT5174 Parallel & Distributed Systems Dr. Ronald Pose Lecture 31

Lock Based Concurrency Control
September 4, 1997 Lock Based Concurrency Control Lock modes have the following compatibility: If a resource is not locked or locked in shared mode, a transaction can lock the resource in shared mode. A transaction can lock a resource in exclusive mode only if the resource is not locked. FIT5174 Parallel & Distributed Systems Dr. Ronald Pose Lecture 32

Lock Based Concurrency Control
September 4, 1997 Lock Based Concurrency Control If two transactions are trying to lock the same resource in incompatible mode, the two transactions are in conflict. A conflict relation can be shared-exclusive conflict (or read-write conflict) exclusive-exclusive conflict (or write-write conflict). FIT5174 Parallel & Distributed Systems Dr. Ronald Pose Lecture 33

September 4, 1997 Two Phase Locking Using locks does not necessarily guarantee serializability. Only transactions executed concurrently as follows will be serializable. A transaction locks shared resource before accessing the resource and release the lock after using it. Locking is granted according to the compatibility constraint. A transaction does not acquire any locks after releasing a lock. The usage of locks that satisfies above conditions are 2 phase lock protocol. FIT5174 Parallel & Distributed Systems Dr. Ronald Pose Lecture 34

Two Phase Locking September 4, 1997 A transaction acquires all necessary locks in the first phase, growing phase. The transaction releases all the locks in the second phase, shrinking phase. Two phase locking is the sufficient condition to guarantee serializability Suppose a transaction aborts after releasing some of the locks. If other transactions have used resources protected by these released locks, the transactions must also be aborted - cascading abort. FIT5174 Parallel & Distributed Systems Dr. Ronald Pose Lecture 35

Strict Two Phase Locking
September 4, 1997 To avoid cascading abort, many systems use “strict two-phase locking” in which transactions do not release any lock until commit is completed. Strict two-phase locking is done as follows: Begin transaction Acquire locks before read or write resources Commit Release all locks End transaction FIT5174 Parallel & Distributed Systems Dr. Ronald Pose Lecture 36

Strict Two Phase Locking
September 4, 1997 The strict two-phase locking has the following advantages: A transaction always reads a value written by a committed transaction. A transaction never has to be aborted - the cascading abort never happens. All lock acquisitions and releases can be handled by the system without the transaction being aware of these. Locks are acquired whenever data are accessed and released when the transaction has finished. Two-phase locking, can cause deadlocks to occur. two processes try to acquire the same pair of locks but in the reverse order. FIT5174 Parallel & Distributed Systems Dr. Ronald Pose Lecture 37

Lock Granularity September 4, 1997 The size of resources that can be locked by a single lock is called “lock granularity.” The finer the granularity, the more concurrent processing is possible while more locks are required. For example, let’s compare locking by file and locking by record contained in a file. Suppose two transactions access different records in the same file. If record is the unit of lock, two transactions can access the data simultaneously. On the other hand, if file is the unit of lock, the two transactions cannot access the data simultaneously. FIT5174 Parallel & Distributed Systems Dr. Ronald Pose Lecture 38

Concurrency Control Using Timestamps (Reed)
September 4, 1997 Concurrency Control Using Timestamps (Reed) Assign each transaction a timestamp at Begin Transaction. Timestamps must be unique (we can ensure the uniqueness using Lamport’s algorithm). Each resource has a read timestamp and a write timestamp. FIT5174 Parallel & Distributed Systems Dr. Ronald Pose Lecture 39

Timestamp Based Concurrency Control
September 4, 1997 Timestamp Based Concurrency Control A read timestamp is the timestamp of the transaction which read the resource most recently. Let TRD(x) be the read timestamp of resource x. A write timestamp is the timestamp of the transaction which updated the resource most recently. Let TWR(x) be the write timestamp of resource x. Note that Read timestamps and Write-timestamps are NOT the values of actual time when the data items are being actually read or written. FIT5174 Parallel & Distributed Systems Dr. Ronald Pose Lecture 40

Timestamp Based Concurrency Control
September 4, 1997 Timestamp Based Concurrency Control When a transaction with timestamp T tries to read resource x, If T < TWR(x), that means the transaction needs to read a value of x that is already overwritten. Hence the request is rejected and the transaction is aborted. If T ≥ TWR(x), the transaction is allowed to read the resource. FIT5174 Parallel & Distributed Systems Dr. Ronald Pose Lecture 41

Timestamp Based Concurrency Control
September 4, 1997 Timestamp Based Concurrency Control When a transaction with timestamp T tries to update resource x, If T < TWR(x) the transaction is attempting to write an obsolete value of x the request is rejected and the transaction is aborted. if T < TRD(x), then the value of x that the transaction is producing was needed previously, and the system assumed that value would never be produced, the request is rejected and the transaction is aborted. Otherwise, the transaction is allowed to update the resource. FIT5174 Parallel & Distributed Systems Dr. Ronald Pose Lecture 42

Timestamp Based Concurrency Control
September 4, 1997 Timestamp Based Concurrency Control The timestamp-ordering protocol guarantees serializability since all the arcs in the precedence graph are of the form: Thus, there will be no cycles in the precedence graph Timestamp protocol ensures freedom from deadlock as no transaction ever waits. But the schedule may not be recoverable. transaction with smaller timestamp transaction with larger timestamp FIT5174 Parallel & Distributed Systems Dr. Ronald Pose Lecture 43

Optimistic Concurrency Control
September 4, 1997 Optimistic Concurrency Control Run the transactions, be optimistic about transaction conflicts. At the end of a transaction, i.e. when it is committing, existence of conflict is examined. If no conflict is detected the local copies are written on the real resource. Otherwise, the transaction is aborted.  Progression in phases Phase: and 3 The conflict check is done by examining whether the resource used by this transaction is updated by other transaction(s) while this transaction was executing. To make this check possible, the system must keep track of the resources used by the transactions. FIT5174 Parallel & Distributed Systems Dr. Ronald Pose Lecture 44

Optimistic Concurrency Control
September 4, 1997 Optimistic Concurrency Control Time-stamping Each transaction is assigned a timestamp TS(Ti) at the beginning of the validation phase. Validation conditions For every pair of transactions Ti, Tj, such that TS(Ti ) < TS(Tj), one of the following validation conditions must hold: 1. Ti completes (all three phases) before Tj begins. 2. Ti completes before Tj starts its write phase, and Ti does not write any object read by Tj. 3. Ti completes its Read phase before Tj completes its Read phase, and Ti does not write any object that is either read or written by Tj. FIT5174 Parallel & Distributed Systems Dr. Ronald Pose Lecture 45

Optimistic Concurrency Control
September 4, 1997 Optimistic Concurrency Control Optimistic concurrency control is designed with the assumption that conflicts are rare and we do not need to abort transactions often. Update is performed on local copies. This method fits well with the private workspace. The advantage is that it allows maximum parallel execution since transactions never wait and deadlocks never occur. The disadvantage is that a transaction may be aborted at the very end since the conflict check is done at the commit point. All operations of the aborted transaction must be done again from the start. FIT5174 Parallel & Distributed Systems Dr. Ronald Pose Lecture 46

Summarizing Concurrency and Transactions
September 4, 1997 Why do we need Election Algorithms? To coordinate distributed transactions Two Election Algorithms? Bully Algorithm, Ring Algorithm Four Key properties of a distributed transaction? Atomic, consistent, isolated, durable Main programming primitives of a distributed transaction? Begin Transaction, Abort, Read, Write, End Transaction Two main approaches to implement distributed transactions? Private workspace, Write-ahead Log Two distributed commit protocols? 2 Phase commit, 3 Phase Commit Why Concurrency Control is necessary for distributed transactions? For implementing serialization, deadlock prevention, improving parallelism Three Schemes for implementing Concurrency Control in Distributed Systems? Lock based CC Timestamp based CC Optimistic CC FIT5174 Parallel & Distributed Systems Dr. Ronald Pose Lecture 47

Fault Tolerance in Distributed Systems
September 4, 1997 Faults and Fault-Tolerant Systems Why is fault tolerance important? Parallel and distributed systems usually relay upon a large number of “nodes” connected over a network, very often a large network; networks are prone to errors, dropouts and localised failures; hardware will fail; operating systems will fail or crash. Ensuring a parallel/distributed system can perform robustly is therefore important, since otherwise a small portion of the overall system can cripple the whole system. FIT5174 Parallel & Distributed Systems Dr. Ronald Pose Lecture 48

Dependability Availability: System is ready to be used immediately
September 4, 1997 Availability: System is ready to be used immediately Reliability: System can run continuously without failure Safety: When a system (temporarily) fails to operate correctly, nothing catastrophic happens Maintainability: How easily a failed system can be repaired - Building a dependable system comes down to controlling failure and faults. FIT5174 Parallel & Distributed Systems Dr. Ronald Pose Lecture 49

Total versus Partial Failure
September 4, 1997 Total Failure: All components in a system fail Typical in non-distributed system Partial Failure: One or more (but not all) components in a distributed system fail Some components affected Other components completely unaffected Considered as fault for the whole system FIT5174 Parallel & Distributed Systems Dr. Ronald Pose Lecture 50

Failure September 4, 1997 Failure: a system fails when it fails to meet its promises or cannot provide its services in the specified manner Error: part of the system state that leads to failure (i.e., it differs from its intended value) Fault: the cause of an error (results from design errors, manufacturing faults, deterioration, or external disturbance) Recursive: Failure may be initiated by a mechanical fault Manufacturing fault leads to disk failure Disk failure is a fault that leads to database failure Database failure is a fault that leads to service failure FIT5174 Parallel & Distributed Systems Dr. Ronald Pose Lecture 51

Classification of Failure Types
September 4, 1997 Our view of a distributed system is a process-level view, so we begin with the description of certain types of failures that are visible at the process level; The major classes of failures are as follows: Crash Failure Omission Failure Transient Failure Byzantine Failure Software Failure Temporal Failure Security Failure FIT5174 Parallel & Distributed Systems Dr. Ronald Pose Lecture 52

Classification of Failure Types
September 4, 1997 Crash (Soft) Failure A process undergoes crash failure, when it permanently ceases to execute its actions. This is an irreversible change. In an asynchronous model, crash failures cannot be detected with total certainty, since there is no lower bound of the speed at which a process can execute its actions. In a synchronous system where processor speed and channel delays are bounded, crash failures can be detected using timeouts. FIT5174 Parallel & Distributed Systems Dr. Ronald Pose Lecture 53

Classification of Failure Types
September 4, 1997 Omission Failure If the receiver does not receive one or more of the messages sent by the transmitter, an omission failure occurs. For wireless networks, collision occurs in MAC layer or receiving node moves out of range. Transient Failure A transient failure can disturb the state of processes in an arbitrary way. The agent inducing this problem may be momentarily active but it can make a lasting effect on the global state. E.g., a power surge, or a mechanical shock, or a lightning strike. FIT5174 Parallel & Distributed Systems Dr. Ronald Pose Lecture 54

Classification of Failure Types
September 4, 1997 Byzantine Failure Byzantine failures represent the weakest of all failure model that allows every conceivable form of erroneous behaviour. The term alludes to uncertainty and was first proposed by Pease et al. Assume that process i forwards the value x of a local variable to each of its neighbours. The followings inconsistencies may occur: two distinct neighbours j and k receive values x and y, where x ≠ y one or more neighbours do not receive any data from i every neighbour receives a value z where z ≠ x FIT5174 Parallel & Distributed Systems Dr. Ronald Pose Lecture 55

Classification of Failure Types
September 4, 1997 Some possible causes of the Byzantine failures are: total or partial breakdown of a link joining i with one of its neighbours software problems in process i hardware synchronization problems – assume that every neighbour is connected to the same bus, and reading the same copy sent out by i, but since the clocks are not perfectly synchronized, they may not read the value of x at the same time. If value of x varies with time, then different neighbours of i may read different values of x from process i. FIT5174 Parallel & Distributed Systems Dr. Ronald Pose Lecture 56

Classification of Failure Types
September 4, 1997 Temporal Failure Real-time systems require actions to be completed within a specific time frame. When this time limit is not met, a temporal failure occurs. Security Failure Virus and other malicious software may lead to unexpected behaviour that manifests itself as a system fault. FIT5174 Parallel & Distributed Systems Dr. Ronald Pose Lecture 57

History of Fault Tolerant Systems
September 4, 1997 The first known fault-tolerant computer was SAPO, built in 1951 in Czechoslovakia by Antonin Svoboda; Most of the development in the so called LLNM (Long Life, No Maintenance) computing was done by NASA during the 1960's, in preparation for Project Apollo and other research aspects. NASA's first machine went into a space observatory, and their second attempt, the JSTAR computer, was used in Voyager. This computer had a backup of memory arrays to use memory recovery methods and thus it was called the JPL Self-Testing-And-Repairing computer. It could detect its own errors and fix them or bring up redundant modules as needed; Smart Sensor Network in the Ageless Space Vehicle Project [Topic05-ref02] Don Price et al; Tandem Computer ( ) – absorbed by HP - manufactured fault tolerant minicomputers for critical applications; FIT5174 Parallel & Distributed Systems Dr. Ronald Pose Lecture 58

Fault Tolerant Systems
September 4, 1997 We designate a system that does not tolerate failures as a fault-tolerant system. In such systems, the occurrence of a fault violates liveness and safety properties. The are four major types of fault-tolerance Masking tolerance Non-masking tolerance Fail-safe tolerance Graceful degradation Note: Safety properties specify that “something bad never happens” Doing nothing easily fulfils a safety property as this will never lead to a “bad” situation Safety properties are complemented by liveness properties Liveness properties assert that: “something good” will eventually happen [Lamport] FIT5174 Parallel & Distributed Systems Dr. Ronald Pose Lecture 59

September 4, 1997 Let P be the set of configurations for the fault-tolerance system. Given a set of fault actions F, the fault span Q corresponds to the largest set of configurations that the system can support. In Masking tolerance systems, when a fault F is masked its occurrence has no impact on the application, that is P = Q. Masking tolerance is important in many safety-critical applications where the failure can endanger human life or cause massive loss of property. An aircraft must be able to fly even if one of its engines malfunctions. Masking tolerance preserve both safety and liveness properties of the original system. FIT5174 Parallel & Distributed Systems Dr. Ronald Pose Lecture 60

September 4, 1997 In non-masking fault tolerance, faults may temporarily affect and violate the safety property, that is However, liveness is not compromised, and eventually normal behaviour is restored. Consider that while watching a movie, the server crashed, but the system automatically restored the service by switching to a standby proxy server. Stabilization and Checkpointing represent two opposing scenario in non-masking tolerance. Checkpointing relies on history and recovery is achieved by retrieving the lost computation. Stabilization is history-insensitive and does not care about lost computation as long as eventual recovery is guaranteed. FIT5174 Parallel & Distributed Systems Dr. Ronald Pose Lecture 61

Fail-Safe Tolerance September 4, 1997 Certain faulty configurations do not affect the application in an adverse way and therefore considered harmless. A fail-safe system relaxes the tolerance requirement by avoiding only those faulty configurations that will have catastrophic consequences (not withstanding the failure). As an example: At a four-way traffic crossing, if the lights are green in both directions then a collision is possible. However, if the lights are red, at best traffic will stall but will not have any catastrophic side effect. FIT5174 Parallel & Distributed Systems Dr. Ronald Pose Lecture 62

Graceful Degradation September 4, 1997 There are systems that neither mask, nor fully recover from the effect of failures, but exhibit a degraded behaviour that falls short of normal behaviour, but is still considered acceptable. The notion of acceptability is highly subjective and entirely dependent on the user running the application. Some examples While routing a message between two points in a network, a program computes the shortest path. In the presence of a failure, if this program returns another path but not the shortest, then this may be acceptable. An operating system may switch to a safe mode where users cannot create or modify files, but can read the files that already exist. FIT5174 Parallel & Distributed Systems Dr. Ronald Pose Lecture 63

Summary We looked at Failure because:
September 4, 1997 We looked at Failure because: Distributed systems are inherently more prone to failure Parallelism by definition means many things happening at once and thus more things to go wrong We may indeed use a distributed system to provide redundancy to help deal with failure Dealing with failure in distributed and parallel systems can require difference techniques compared with simple systems. FIT5174 Parallel & Distributed Systems Dr. Ronald Pose Lecture 64