CS 162 Section 10 Two-phase commit Fault-tolerant computing

Sec /14/2014 CS162 ©UCB Spring 2014 Administrivia Project 3 code due Thursday 4/17 by 11:59PM Project 4 design due date –Thursday 4/24 by 11:59PM Midterm II is April 28 th 4-5:30pm in 245 Li Ka Shing and 100 GPB –Covers Lectures #13-24 –Closed book and notes, no calculators –One double-sides handwritten page of notes allowed –Review session: Fri Apr pm in 245 Li Ka Shing

Sec /14/2014 CS162 ©UCB Spring 2014 QUIZ

Sec /14/2014 CS162 ©UCB Spring 2014 Quiz True/False 1.Two-phase commit is used to guarantee consistency. False 2.You always need 2PC if you want multiple systems to stay consistent. False (ex: use consistent client-side hashing to put keys amongst slaves, with no replication.) 3.If a master server comes awake after crashing in the WAIT state, it resends VOTE_REQ in order to recount the slaves. False 4.Increasing mean-time-to-repair decreases availability (as defined in lecture). True 5.Bohrbugs won’t be fixed by restarting a task or system. True Short answer 5.What was the purpose of the TLS heartbeat, target of the Heartbleed attack? Make sure NATs/firewalls in the middle don’t shut down the connection.

Sec /14/2014 CS162 ©UCB Spring 2014 TWO-PHASE COMMIT

Sec /14/2014 CS162 ©UCB Spring 2014 Durability and Atomicity, distributed How do you make sure transaction results persist in the face of failures (e.g., disk failures)? Replicate database –Commit transaction to each replica What happens if you have failures during a transaction commit? –Need to ensure atomicity: either transaction is committed on all replicas or none at all How can we replicate with atomicity?

Sec /14/2014 CS162 ©UCB Spring 2014 Two-Phase Commit Coordinator sends VOTE-REQ to all workers – Wait for VOTE-REQ from coordinator – If ready, send VOTE-COMMIT to coordinator – If not ready, send VOTE-ABORT to coordinator – And immediately abort – If receive VOTE-COMMIT from all N workers, send GLOBAL-COMMIT to all workers – If doesn’t receive VOTE-COMMIT from all N workers, send GLOBAL- ABORT to all workers – If receive GLOBAL-COMMIT then commit – If receive GLOBAL-ABORT then abort Coordinator AlgorithmWorker Algorithm

Sec /14/2014 CS162 ©UCB Spring 2014 Worker, Master states INIT WAIT ABORTCOMMIT Recv: START Send: VOTE-REQ Recv: VOTE-ABORT Send: GLOBAL-ABORT Recv: VOTE-COMMIT Send: GLOBAL-COMMIT INIT READY ABORTCOMMIT Recv: VOTE-REQ Send: VOTE-ABORT Recv: VOTE-REQ Send: VOTE-COMMIT Recv: GLOBAL-ABORT Recv: GLOBAL-COMMIT Master Worker

Sec /14/2014 CS162 ©UCB Spring 2014 Failure Free Example Execution coordinator worker 1 time VOTE-REQ VOTE- COMMIT GLOBAL- COMMIT worker 2 worker 3

Sec /14/2014 CS162 ©UCB Spring 2014 Dealing with Worker Failures How to deal with worker failures? –Failure only affects states in which the node is waiting for messages –Coordinator only waits for votes in “WAIT” state –In WAIT, if doesn’t receive N votes, it times out and sends GLOBAL-ABORT INIT WAIT ABORTCOMMIT Recv: START Send: VOTE-REQ Recv: VOTE-ABORT Send: GLOBAL-ABORT Recv: VOTE-COMMIT Send: GLOBAL-COMMIT

Sec /14/2014 CS162 ©UCB Spring 2014 Dealing with Coordinator Failure How to deal with coordinator failures? –worker waits for VOTE-REQ in INIT »Worker can time out and abort (coordinator handles it) –worker waits for GLOBAL-* message in READY »If coordinator fails, workers must BLOCK waiting for coordinator to recover and send GLOBAL_* message INIT READY ABORTCOMMIT Recv: VOTE-REQ Send: VOTE-ABORT Recv: VOTE-REQ Send: VOTE-COMMIT Recv: GLOBAL-ABORTRecv: GLOBAL-COMMIT

Sec /14/2014 CS162 ©UCB Spring 2014 Example of Coordinator Failure VOTE-REQ VOTE-COMMIT INIT READY ABORTCOMM block waiting for coordinator restarted GLOBAL-ABORT coordinator worker 1 worker 2 worker 3

Sec /14/2014 CS162 ©UCB Spring 2014 Remembering Where We Were (Durability) All nodes use stable storage* to store which state they are in Upon recovery, it can restore state and resume: –Coordinator aborts in INIT, WAIT, or ABORT –Coordinator commits in COMMIT –Worker aborts in INIT, ABORT –Worker commits in COMMIT –Worker asks Coordinator in READY * - stable storage is non-volatile storage (e.g. backed by disk) that guarantees atomic writes.

Sec /14/2014 CS162 ©UCB Spring 2014 FAULT-TOLERANT COMPUTING

Lec /14/2014 CS162 ©UCB Spring 2014 Dependability: The 3 ITIES Reliability / Integrity: does the right thing. (Need large MTBF) Availability: does it now. (Need small MTTR MTBF+MTTR) System Availability: if 90% of terminals up & 99% of DB up? (=> 89% of transactions are serviced on time ) Security Integrity Reliability Availability MTBF or MTTF = Mean Time Between (To) Failure MTTR = Mean Time To Repair (see next slide)

Sec /14/2014 CS162 ©UCB Spring 2014 Mean Time to Recovery Critical time as further failures can occur during recovery Total Outage duration (MTTR) = Time to Detect(need good monitoring) + Time to Diagnose(need good docs/ops, best practices) + Time to Decide(need good org/leader, best practices) + Time to Act(need good execution!)

Sec /14/2014 CS162 ©UCB Spring 2014 Traditional Fault Tolerance Techniques Fail fast modules: work or stop Spare modules: yield instant repair time Process/Server pairs: Mask HW and SW faults Transactions: yields ACID semantics (simple fault model)

Sec /14/2014 CS162 ©UCB Spring 2014 Fail-Fast is Good, but Repair is Needed Improving either MTTR or MTBF gives benefit Simple redundancy does not help much (can actually hurt!) Lifecycle of a module fail-fast gives short fault latency High Availability is low UN-Availability Unavailability ~ MTTR MTBF X FaultDetect RepairReturn

Sec /14/2014 CS162 ©UCB Spring 2014 Fail-Fast and High-Availability Execution Process Pairs: Instant repair Use Defensive programming to make a process fail-fast Have separate backup process ready to “take over” if primary faults SW fault is a Bohrbug  no repair “wait for the next release” or “get an emergency bug fix” or “get a new vendor” SW fault is a Heisenbug  restart process “ reboot and retry” Yields millisecond repair times Tolerates some HW faults