1. ECE 753: FAULT-TOLERANT COMPUTING
7/24/2019
ECE 753: FAULT-TOLERANT COMPUTING
Kewal K.Saluja
Department of Electrical and Computer Engineering
HIGH Level Fault-Tolerance: Checkpointing and recovery (more slides) – Forward error recovery

2. ECE 753 Fault Tolerant Computing
7/24/2019
Overview
Forward error recovery
Summary
ECE 753 Fault Tolerant Computing

3. Checkpointing: Dist. Sys. (contd.)
7/24/2019
Checkpointing: Dist. Sys. (contd.)
Checkpoint based recovery – Comm. induced
Place system wide constraints on message passing (communication pattern) and checkpointing to guarantee progression of recovery line
For example if within every checkpoint interval all messages received precede all messages sent, then the system is domino effect free. Such a message passing system will appear to be as follows
Global checkpoint state
Global checkpoint state
ECE 753 Fault Tolerant Computing

4. Checkpointing: Dist. Sys. (contd.)
7/24/2019
Checkpointing: Dist. Sys. (contd.)
Checkpoint based recovery – Comm. induced
Consistency of state can be guaranteed (no domino effect will take place) if we take a checkpoint before every non deterministic event (note this is a special case of what we saw before). The challenge here is to reduce the number of checkpoints
Generalization of the previous statement (all “receives” to precede “sends”) have been studied in literature
ECE 753 Fault Tolerant Computing

5. Checkpointing: Dist. Sys. (contd.)
7/24/2019
Checkpointing: Dist. Sys. (contd.)
Log based recovery - Pessimistic logging (contd.)
Example and logs
Logs of
P0 is {m0, m4, m7}
P1 is {m1, m3, m6}
P2 is {m2, m5} m2 m0 m1 m3 m4 m5 m6 m7 P0 P1 P2 A B C
ECE 753 Fault Tolerant Computing

6. Forward error recovery
7/24/2019
Forward error recovery
Consider the following scheme for checkpoint based rollback recovery
Two processors P1 and P2
At checkpoints P1 and P2 compare results and save state (checkpoints)
Error detected at such a comparison, causes the processors P1 and P2 to roll back
Note: if there were three processors, they could potentially mask the faulty process/processor
ECE 753 Fault Tolerant Computing

7. Forward error recovery (contd.)
7/24/2019
Forward error recovery (contd.)
Roll forward – two processors and a spare
error P1 P2
chk2
compare
chk1
spare
ECE 753 Fault Tolerant Computing

8. Forward error recovery (contd.)
7/24/2019
Forward error recovery (contd.)
Additional issues
If we had three processors to begin with why not use fault masking?
Think of more than one job and the number of processors required – a single spare can be spare for many pairs of jobs
What if a second fault occurs while processor P1 is continuing?
Use spare for one more period
ECE 753 Fault Tolerant Computing

9. ECE 753 Fault Tolerant Computing
7/24/2019
Summary
Discussed checkpointing and logging issues at length
ECE 753 Fault Tolerant Computing