1. 1 Fault Tolerance and Recovery Mostly taken from www.logos.ic.i.u-tokyo.ac.jp/~horita/study/recovery.ppt

2. 2 Contents Introduction Recovery Checkpointing Difficulty of Checkpointing Synchronous checkpointing / recovery （ Asynchronous checkpointing / recovery ）

3. 3 Introduction Long computation in distributed environments  High failure rate Host failure (a lot of hosts) Network failure One failure may disturb entire computation ⇒ Need to start it again from the beginning High cost Why don’t we utilize the previous computation? Recovery

4. 4 Recovery is not easy Suppose that a parallel computation is running in distributed resources… for(i=0; i<MAXITER; i++){ local_compute(); // compute at each host global_state_exchange(); // communicate with neighbors } 1 1 1 1 7 7 77 8 8 need to save process states periodically usually other processes have to restore to previous state overhead

5. 5 Recovery

6. 6 Back/Forward Error Recovery Forward-error recovery Only when it is possible to remove errors Enable processes to move forward Ex) Redundancy, vote Backward-error recovery General Restore to a previous error-free state Ex) Checkpoint

7. 7 Backward-error recovery operational-based approach Record all modifications of a process’ state state-based approach Record complete state at certain point Recovery Forward- error Backward- error operational- based state- based

8. 8 State-based approach Terminology checkpointing the process of saving state checkpoint the recovery point at which checkpointing occurs rolling back the process of restoring a process to a prior-state

9. 9 Checkpointing

10. 10 Problem of naïve checkpointing Orphan Messages and the Domino Effect Orphan message : a message that make an inconsistent state Domino Effect : what a single rolling back induce other rolling back Lost Messages Livelocks

11. 11 Orphan message and Domino Effect X Y Z [ [ [ x1x1 y1y1 z1z1 [ [ [ [ x2x2 x3x3 y2y2 z2z2 Roll back Y has not sent yet, but X has received. : Orphan message : Domino Effect Suppose Y fails after sending message m and is rolled back to y2. Now X has received message m from Y, but Y has no record of sending it. m is referred to as an orphan message. if Z is rolled back, all three processes must roll back to their very first recovery points, namely, x1, y1, and z1.

12. 12 Lost messages X Y Z [ [ [ x1x1 y1y1 z1z1 [ [ [ x2x2 y2y2 z2z2 [ x3x3 Roll back X has sent, but Y cannot receive forever : Lost message If Y fails after receiving message m, the system is restored to state {x1, y1}, in which message is lost as process X is past the point where it sends message m.

13. 13 Livelocks X Y[ y1y1 [ x1x1 n1 m1 n1 m2 n2 Process Y fails before receiving message n1. When Y rolls back to y1, there is no record of sending message m1, hence X must roll back to x1. And then Y receives message n1, but X has no record of sending it, so Y must roll back again. The situation will repeat infinitely.

14. 14 Consistency of Checkpoint Strongly consistent set of checkpoints no messages penetrating the set Consistent set of checkpoints no messages penetrating the set backward [ [ [ x1x1 y1y1 z1z1 [ [ [ y2y2 x2x2 z2z2 Strongly consistent consistent No orphans and domino effect but need to deal with lost messages

15. 15 Consistent Set of Checkpoints Informally, a cut (global state) in the time diagram is consistent if no arrow starts on the right-hand side and ends on the left-hand side of it. C1 is consistent, but C2 is inconsistent. p q r s C1 C2

16. 16 Useless Checkpoints Q_2 is a useless checkpoint Q_2 records the receipt of m1, but not the sending of m2 {P1,Q_2} cannot be consistent (otherwise m1 would become an orphan); similarly {P_2,Q_2} cannot be consistent (since otherwise m2 would become an orphan)

17. 17 Checkpoint/Recovery Algorithm Synchronous with global synchronization at checkpointing Asynchronous without global synchronization at checkpointing

18. 18 Preliminary (Assumption) Goal To make a consistent global checkpoint Assumptions Communication channels are FIFO No partition of the network End-to-end protocols cope with message loss due to rollback recovery and communication failure No failure during the execution of the algorithm ～ Synchronous Checkpoint ～

19. 19 Preliminary (Two types of checkpoint) tentative checkpoint : a temporary checkpoint a candidate for permanent checkpoint permanent checkpoint : a local checkpoint at a process a part of a consistent global checkpoint ～ Synchronous Checkpoint ～

20. 20 Checkpoint Algorithm Algorithm 1. an initiating process (a single process that invokes this algorithm) takes a tentative checkpoint 2. it requests all the processes to take tentative checkpoints 3. it waits for receiving from all the processes whether taking a tentative checkpoint has been succeeded 4. if it learns all the processes has succeeded, it decides all tentative checkpoints should be made permanent; otherwise, should be discarded. 5. it informs all the processes of the decision 6. The processes that receive the decision act accordingly Supplement Once a process has taken a tentative checkpoint, it shouldn’t send messages until it is informed of initiator’s decision. ～ Synchronous Checkpoint ～

21. 21 Diagram of Checkpoint Algorithm [ [ [ | | Tentative checkpoint | request to take a tentative checkpoint OK decide to commit [ permanent checkpoint [ [ consistent global checkpoint Unnecessary checkpoint Initiator ～ Synchronous Checkpoint ～

22. 22 Optimized Algorithm Each message is labeled by order of sending Labeling Scheme ⊥ : smallest label т : largest label last_label_rcvd X [Y] : the last message that X received from Y after X has taken its last permanent or tentative checkpoint. if not exists, ⊥ is in it. first_label_sent X [Y] : the first message that X sent to Y after X took its last permanent or tentative checkpoint. if not exists, ⊥ is in it. ckpt_cohort X : the set of all processes that may have to take checkpoints when X decides to take a checkpoint. ～ Synchronous Checkpoint ～ [ [ X Y x2 x3 y1y2 x2 Checkpoint request need to be sent to only the processes included in ckpt_cohort

23. 23 Optimized Algorithm ckpt_cohort X : { Y | last_label_rcvd X [Y] > ⊥ } Y takes a tentative checkpoint only if last_label_rcvd X [Y] >= first_label_sent Y [X] > ⊥ ～ Synchronous Checkpoint ～ X Y [ [ last_label_rcvdX[Y] first_label_sentY[X]

24. 24 Optimized Algorithm Algorithm 1. an initiating process takes a tentative checkpoint 2. it requests p ∈ ckpt_cohort to take tentative checkpoints ( this message includes last_label_rcvd[reciever] of sender ) 3. if the processes that receive the request need to take a checkpoint, they do the same as 1.2.; otherwise, return OK messages. 4. they wait for receiving OK from all of p ∈ ckpt_cohort 5. if the initiator learns all the processes have succeeded, it decides all tentative checkpoints should be made permanent; otherwise, should be discarded. 6. it informs p ∈ ckpt_cohort of the decision 7. The processes that receive the decision act accordingly ～ Synchronous Checkpoint ～

25. 25 Diagram of Optimized Algorithm [ [ [ [ A C B D ab1 ac1 bd1 dc1 dc2 cb1 ba1ba2 ac2 cb2 cd1 | Tentative checkpoint ca2 last_label_rcvdX[Y] >= first_label_sentY[X] > ⊥ 2 >= 1 > 0 | 2 >= 2 > 0 | 2 >= 0 > 0 OK decide to commit [ Permanent checkpoint [ [ ckpt_cohortX : { Y | last_label_rcvdX[Y] > ⊥ } ～ Synchronous Checkpoint ～

26. 26 Correctness A set of permanent checkpoints taken by this algorithm is consistent No process sends messages after taking a tentative checkpoint until the receipt of the decision New checkpoints include no message from the processes that don’t take a checkpoint The set of tentative checkpoints is fully either made to permanent checkpoints or discarded. ～ Synchronous Checkpoint ～

27. 27 Recovery Algorithm Labeling Scheme ⊥ : smallest label т : largest label last_label_rcvd X [Y] : the last message that X received from Y after X has taken its last permanent or tentative checkpoint. If not exists, ⊥ is in it. first_label_sent X [Y] : the first message that X sent to Y after X took its last permanent or tentative checkpoint. If not exists, ⊥ is in it. roll_cohort X : the set of all processes that may have to roll back to the latest checkpoint when process X rolls back. last_label_sent X [Y] : the last message that X sent to Y before X takes its latest permanent checkpoint. If not exist, т is in it. ～ Synchronous Recovery ～

28. 28 Recovery Algorithm roll_cohort X = { Y | X can send messages to Y } Y will restart from the permanent checkpoint only if last_label_rcvd Y [X] > last_label_sent X [Y] ～ Synchronous Recovery ～

29. 29 Recovery Algorithm Algorithm 1. an initiator requests p ∈ roll_cohort to prepare to rollback ( this message includes last_label_sent[reciever] of sender ) 2. if the processes that receive the request need to rollback, they do the same as 1.; otherwise, return OK message. 3. they wait for receiving OK from all of p ∈ ckpt_cohort. 4. if the initiator learns p ∈ roll_cohort have succeeded, it decides to rollback; otherwise, not to rollback. 5. it informs p ∈ roll_cohort of the decision 6. the processes that receive the decision act accordingly ～ Synchronous Recovery ～

30. 30 Diagram of Synchronous Recovery [ [ [ [ A C B D ab1 ac1 bd1 dc1 dc2 cb1 ba1ba2 ac2 cb2 dc1 request to roll back 0 > 1 last_label_rcvdY[X] > last_label_sentX[Y] 2 > 1 0 >т OK [ [ 2 > 1 0 >т [ decide to roll back roll_cohortX = { Y | X can send messages to Y }

31. 31 Drawbacks of Synchronous Approach Additional messages are exchanged Synchronization delay An unnecessary extra load on the system if failure rarely occurs

32. 32 Asynchronous Checkpoint Characteristic Each process takes checkpoints independently No guarantee that a set of local checkpoints is consistent A recovery algorithm has to search consistent set of checkpoints No additional message No synchronization delay Lighter load during normal execution

33. 33 Preliminary (Assumptions) Goal To find the latest consistent set of checkpoints Assumptions Communication channels are FIFO Communication channels are reliable The underlying computation is event-driven ～ Asynchronous Checkpoint / Recovery ～

34. 34 Asynchronous Checkpointing Basic idea Save states, messages sent at each event Volatile logging Each processor notes number of messages sent to others, and received from others Use counters to determine orphan messages

35. 35 Preliminary (Two types of log) save an event on the memory at receipt of messages (volatile log) volatile log periodically flushed to the disk (stable log) ⇔ checkpoint volatile log : quick access lost if the corresponding processor fails stable log : slow access not lost even if processors fail ～ Asynchronous Checkpoint / Recovery ～

36. 36 Preliminary (Definition) Definition CkPt i : the checkpoint (stable log) that i rolled back to when failure occurs RCVD i←j (CkPt i / e ) : the number of messages received by processor i from processor j, per the information stored in the checkpoint CkPt i or event e. SENT i→j (CkPt i / e ) : the number of messages sent by processor i to processor j, per the information stored in the checkpoint CkPt i or event e ～ Asynchronous Checkpoint / Recovery ～

37. 37 Recovery Algorithm Algorithm 1. When one process crashes, it recovers to the latest checkpoint CkPt. 2. It broadcasts the message that it had failed. Others receive this message, and rollback to the latest event. 3. Each process sends SENT(CkPt) to neighboring processes 4. Each process waits for SENT(CkPt) messages from every neighbor 5. On receiving SENT j→i (CkPt j ) from j, if i notices RCVD i←j (CkPt i ) > SENT j→i (CkPt j ), it rolls back to the event e such that RCVD i←j (e) = SENT j→i (e), 6. repeat 3,4,and 5 N times (N is the number of processes) ～ Asynchronous Checkpoint / Recovery ～

38. 38 Asynchronous Recovery X Y Z Ex0Ex1Ex2Ex3 Ey0Ey1Ey2 Ey3 Ez0 Ez1Ez2 [ [ [ x1 y1 z1 (Y,2) (Y,1) (X,2) (X,0) (Z,0) (Z,1) 3 <= 2 RCVDi←j (CkPti) <= SENTj→i(CkPtj) 2 <= 2 X:YX:Z 0 <= 0 1 <= 2 Y:X 1 <= 1 Y:Z 0 <= 0 Z:X 2 <= 1 Z:Y 1 <= 1