Election Distributed Systems

Algorithms to Find Global States Why? To check a particular property exist or not in distributed system –(Distributed) garbage collection –(Distributed) deadlock detection, termination What? –Capture the instantaneous states of a collection of processes –And the messages in transit on different communication channels

Detecting global properties Evaluation of predicates of system’s state –stable predicates: distributed garbage collection deadlock detection termination detection –non-stable (transient) predicates: distributed debugging –safety properties: “Nothing bad ever happens” –eg: mutual exclusion –liveness properties: “Something good eventually happens” –eg: fair scheduling

”I can’t find a solution, I guess I’m just too dumb” Picture from Computers and Intractability, by Garey and Johnson 4

”I can’t find an algorithm, because no such algorithm is possible” Picture from Computers and Intractability, by Garey and Johnson 5

”I can’t find an algorithm, but neither can all these famous people.” Picture from Computers and Intractability, by Garey and Johnson 6

A network partition

Garbage Collection An object is considered as garbage if there are no longer any references to it anywhere in distributed system

Detecting global properties

Garbage Collection Process P1 references object O1 Process P2 references object O2 Message M references object O3 O4 is garbage object

Obvious First Solution… Synchronize clocks of all processes (which algorithm?) Ask all processes to record their states at known time t Problems? –Time synchronization possible only approximately –Does not record the state of messages in the channels

Snapshots (I) Chandy & Lamport, 1985 –Can you capture (record) the states of all processes and communication channels at exactly 10:04:50 am? – Is it even necessary to take such an exact snapshot? –Any process may initiate a snapshot at any time –Assumes strong connectivity At least one path between each process pair –Assumes unidirectional, FIFO channels –Assumes reliable delivery of messages –Records snapshot state locally at all processes No direct method for collecting state at a designated “collector” process

Snapshots (II) Application of 2 rules at each process: –Marker sending rule: After P i has recorded its state, it sends a “marker” message over each of its outgoing channels (before sending any other message) –Marker receipt rule: (c := the incoming channel) If P i has not yet recorded its state: –P i records its state & records the state of channel c as “ ” –P i turns on recording of messages arriving over incoming channels Else: –P i records the state of channel c as the set of messages that it has received over c since it saved its state

Snapshots (III) A process that has received a “marker” message, records its state in finite time & relays the “marker” over its outgoing channels in finite time. Strongly connected network The “marker” traverses each channel, exactly once. When a process has received a “marker” over all its incoming channels, its contribution to the snapshot protocol is complete. In-transit messages are accounted for as belonging to the state of a channel between process.

Global State (I) a)Organization of a process and channels for a distributed snapshot

Global State (II) b)Process Q receives a marker for the first time and records its local state c)Q records all incoming message d)Q receives a marker for its incoming channel and finishes recording the state of the incoming channel

Elections Choose a unique process to play a “role” or coordinator –We require that the elected process be chosen as the one with the largest ID Ids must be unique & totally ordered –Eg: ID := Requirements: –safety: A participant process has elected == P, where P is chosen as a non- crashed process with max. ID, or elected is undefined –liveness: All processes participate & eventually set ‘elected’, or crash

Why Election? Example 1: Your Bank maintains multiple servers, but for each customer, one of the servers is responsible, i.e., is the leader Example 2: In the sequencer-based algorithm for total ordering of multicasts, – What happens if the “special” sequencer process fails? Example 3: Coordinator-based mutual exclusion: need to elect (and keep) one coordinator

Bully algorithm The process P sends election message to all processes with higher number (Assumes that each process knows which processes have higher IDs) If no one respond, P wins If one answer, it takes over

The Bully algorithm Uses timeouts to detect failures: T = 2T trans + T process 3 msg types: coordinator: announcement to all processes with lower IDs election: sent to processes with higher IDs answer: answer to “election” - If not received within T, the sender of “election” sends “coordinator”. - Otherwise, the process waits for T’ to receive a “coordinator” msg. If no msg arrives, it begins a new election. Not “safe” if the crashed processes are replaced with processes with the same ID !

The Bully Algorithm (1) The bully election algorithm Process 4 holds an election Process 5 and 6 respond, telling 4 to stop Now 5 and 6 each hold an election

Example: Bully Election OK P1P1 P2P2 P3P3 P4P4 P0P0 P5 1. P2 initiates election 2. P2 receives “replies P1P1 P2P2 P3P3 P4P4 P0P0 P5 3. P3 & P4 initiate election P1P1 P2P2 P3P3 P4P4 P0P0 P5 P1P1 P2P2 P3P3 P4P4 P0P0 4. P3 receives reply OK Election P1P1 P2P2 P3P3 P4P4 P0P0 P5 5. P4 receives no reply P1P1 P2P2 P3P3 P4P4 P0P0 P5 5. P4 announces itself coor dinat or answer=OK

The bully algorithm The election of coordinator p 2, after the failure of p 4 and then p 3

N Processes are organized in a logical ring. – p i has a communication channel to p (i+1) mod N. – All messages are sent clockwise around the ring. Any process that discovers a coordinator has failed initiates an “election” message that contains its own id:attr. When a process receives an election message, it compares the attr in the message with its own. – If the arrived attr is greater, the receiver forwards the message. – If the arrived attr is smaller and the receiver has not forwarded an election message earlier, it substitutes its own id:attr in the message and forwards it. – If the arrived id:attr is that of the receiver, then this process’s attr must be the greatest, and it becomes the new coordinator. This process then sends an “elected” message to its neighbor announcing the election result. When a process p i receives an elected message, it – sets its variable elected i  id of the message. – forwards the message if it is not the new coordinator. Ring Election

Ring-based election Any process can begin an election, by marking itself as “participant” and then sending an “election” msg to its neighbor. Upon receipt of an election msg: if (arrived ID < receiver’s ID and the receiver is not a participant) { Receiver is marked as a participant; Substitute ID in msg & forward it; } else if(receiver’s ID != arrived ID) { if(receiver is not a participant) { Receiver is marked as a participant; Forward msg; } } else { Receiver becomes coordinator; Send “elected” msg; } # msgs: (N -1) + N + N

Election Example: Ring Algorithm Election algorithm using a ring.

Example: Ring Election Election: 2 Election: 4 Election: 3 Election: 4 P1P1 P2P2 P3P3 P4P4 P0P0 P5 1. P2 initiates election P1P1 P2P2 P3P3 P4P4 P0P0 P5 2. P2 receives “election”, P4 dies P1P1 P2P2 P3P3 P4P4 P0P0 P5 3. Election: 4 is forwarded for ever? May not work when process failure occurs during the election! Consider above example where attr==highest id

Example: Ring Election Election: 2 Election: 2, 3,4,0,1 Election: 2,3,4 Election: 2,3 Coord(4): 2 Coord(4): 2,3 Coord(4) 2, 3,0,1 Election: 2 Election: 2,3 Election: 2,3,0 Election: 2, 3,0,1 Coord(3): 2 Coord(3): 2,3 Coord(3): 2,3,0 Coord(3): 2, 3,0,1 P1P1 P2P2 P3P3 P4P4 P0P0 P5 1. P2 initiates election P1P1 P2P2 P3P3 P4P4 P0P0 P5 2. P2 receives “election”, P4 dies P1P1 P2P2 P3P3 P4P4 P0P0 P5 3. P2 selects 4 and announces the result P1P1 P2P2 P3P3 P4P4 P0P0 P5 4. P2 receives “Coord”, but P4 is not included P1P1 P2P2 P3P3 P4P4 P0P0 P5 5. P2 re-initiates election P1P1 P2P2 P3P3 P4P4 P0P0 P5 6. P3 is finally elected

Mutual Exclusion Bank Database: Think of two simultaneous deposits of $10,000 into your bank account, each from one ATM. –Both ATMs read initial amount of $1000 concurrently from the bank server –Both ATMs add $10,000 to this amount (locally at the ATM) –Both write the final amount to the server –What’s wrong? The ATMs need mutually exclusive access to your account entry at the server

 When a process has to write or update shared data, it enters a critical region to achieve mutual exclusion (no other process uses shared data at same time)  Critical section problem: Mutual exclusion is required to prevent interference and ensure consistency when accessing the resources. Mutual Exclusion

– Semaphores, mutexes, etc. in local operating systems – Message-passing-based protocols in distributed systems: enter() the critical section AccessResource() in the critical section exit() the critical section – Distributed mutual exclusion requirements: Safety – At most one process may execute in CS at any time Liveness – Every request for a CS is eventually granted Ordering (desirable) – Requests are granted in FIFO order Mutual Exclusion

Assumptions For all the algorithms studied, we make the following assumptions: –Each pair of processes is connected by reliable channels (such as TCP). Messages are eventually delivered to recipients’ input buffer. –Processes will not fail.

A central coordinator – Grants permission to enter process & keeps a queue of requests to enter the other processes. – Ensures only one process at a time can access the data Operations (coordinator==server) – To enter a process Send a request to the server & wait for token. – On exiting the CS Send a message to the server to release the token. Features: – Safety, liveness and order are guaranteed – Synchronization delay: one round trip time (release + grant) – The coordinator becomes performance bottleneck and single point of failure. Centralized Algrithm

A Centralized Algorithm a)Process 1 asks the coordinator for permission to enter a critical region. Permission is granted b)Process 2 then asks permission to enter the same critical region. The coordinator does not reply. c)When process 1 exits the critical region, it tells the coordinator, when then replies to 2

A Distributed Algorithm a)Two processes want to enter the same critical region at the same moment. b)Process 0 has the lowest timestamp, so it wins. c)When process 0 is done, it sends an OK also, so 2 can now enter the critical region.

Distributed Algoithm The process send message with critical region, process number, and current time to all other process (multicast) If the receiver in not in critical region and does not want it, send OK If receiver is in critical region, not reply If receiver wants to enter critical region, compares time stamp, lowest one wins

A ring-based algorithm i has a channel to (i +1) mod N Wait for token to pass, retain it to enter(), release it to neighbor when done Does not preserve the relation Continuously consumes B/W Sync. Delay: 1 up to N msg’s

Mutual Exclusion Example: Token Ring Algorithm a)An unordered group of processes on a network. b)A logical ring constructed in software.

Timestamp Approach  Features:  Safety, liveness, and ordering (causal) are guaranteed.  It takes 2(N-1) messages per entry operation (N-1 multicast requests + N-1 replies); Client delay: one round-trip time  Synchronization delay: one message transmission time.

Comparison A comparison of three mutual exclusion algorithms. Algorithm Messages per entry/exit Delay before entry (in message times) Problems Centralized32Coordinator crash Distributed2 ( n – 1 ) Crash of any process Token ring 1 to  0 to n – 1 Lost token, process crash