Distributed Systems CS Case Study: Replication in Google Chubby Recitation 5, Oct 06, 2011 Majd F. Sakr, Vinay Kolar, Mohammad Hammoud

Today…  Last recitation session:  Google Chubby Architecture  Today’s session:  Consensus and Replication in Google Chubby  Announcement:  Project 2 Interim Design Report is due soon

Overview Recap: Google Chubby Consensus in Chubby Paxos Algorithm

Recap: Google Data center Architecture (To avoid clutter the Ethernet connections are shown from only one of the clusters to the external links)

Chubby Overview A Chubby Cell is the first level of hierarchy inside Chubby ( ls ) /ls/chubby_cell/directory_name/…/file_name Chubby instance is implemented as a small number of replicated servers (typically 5) with one designated master Replicas are placed at failure-independent sites Typically, they are placed within a cluster but not within a rack The consistency of replicated database is ensured through a consensus protocol that uses operation logs

Chubby Architecture Diagram

Consistency and Replication In Chubby Challenges in replication of data in Google infrastructure: 1.Replica Servers may run at arbitrary speed and fail 2.Replica Servers have access to stable persistent storage that can survive crashes 3.Messages may be lost, reordered, duplicated or delayed Google has implemented a consensus protocol, using Paxos algorithm, for ensuring consistency The protocol operates over a set of replicas with the goal of reaching an agreement to update a common value

Paxos Algorithm Another algorithm proposed by Lamport Paxos ensures correctness, but not liveliness Algorithm initiation and termination: Any replica can submit a value with the goal of achieving consensus on a final value In Chubby, if all replicas have this value as the next entry in their update logs, then consensus is achieved Paxos is guaranteed to achieve consensus if: A majority of the replicas run for long enough with sufficient network stability

Paxos Approach Steps 1.Election Group of replica servers elect a coordinator 2.Selection of candidate value Coordinator selects the final value and disseminates to the group 3.Acceptance of final value Group will accept or reject a value that is finally stored in all replicas

1. Election Approach: Each replica maintains highest sequence number seen so far If the replica wants to bid for coordinator: It picks a unique number that is higher than all sequence numbers that the replica has seen till now Broadcast a “propose” message with this unique sequence number If other replicas have not seen higher sequence number, they send a “promise” message Promise message signifies that the replica will not promise to any other candidate lesser than the proposed sequence number The promise message may include a value that the replica wants to commit Candidate replica with majority of “promise” message wins Challenges: Multiple coordinators may co-exist Reject messages from old coordinators

Message Exchanges in Election

2. Selection of candidate values Approach: The elected coordinator will select a value from all promise messages If the promise messages did not contain any value then the coordinator is free to choose any value Coordinator sends the “accept” message (with the value) to the group of replicas Replicas should acknowledge the accept message Coordinator waits until a majority of the replicas answer Possible indefinite wait

Message Exchanges in Consensus

3. Commit the value Approach If a majority of the replicas acknowledge, then the coordinator will send a “commit” message to all replicas Otherwise, Coordinator will restart the election process

Message Exchanges in Commit

References “Paxos Made Live – An Engineering Perspective”, Tushar Chandra, Robert Griesemer, and Joshua Redstone, 26th ACM Symposium on Principles of Distributed Computing, PODC 2007