1. Tanenbaum & Van Steen, Distributed Systems: Principles and Paradigms, 2e, (c) 2007 Prentice-Hall, Inc. All rights reserved. 0-13-239227-5 DISTRIBUTED SYSTEMS Principles and Paradigms Second Edition ANDREW S. TANENBAUM MAARTEN VAN STEEN Chapter 6 Synchronization

2. Tanenbaum & Van Steen, Distributed Systems: Principles and Paradigms, 2e, (c) 2007 Prentice-Hall, Inc. All rights reserved. 0-13-239227-5 Outline Synchronization –Time synchronization –Logical clocks Lamport’s logical clocks Vector clocks Causal communications Mutual exclusion –Centralized algorithm –Decentralized algorithm –Distributed algorithm –Token Ring algorithm Global positioning Election

3. Tanenbaum & Van Steen, Distributed Systems: Principles and Paradigms, 2e, (c) 2007 Prentice-Hall, Inc. All rights reserved. 0-13-239227-5 Lamport’s Logical Clocks Before executing an event Pi executes Ci ← Ci + 1. When process Pi sends a message m to Pj, it sets m’s timestamp ts (m) equal to Ci after having executed the previous step. Upon the receipt of a message m, process Pj adjusts its own local counter as Cj ← max{Cj, ts (m)}, after which it then executes the first step and delivers the message to the application.

4. Tanenbaum & Van Steen, Distributed Systems: Principles and Paradigms, 2e, (c) 2007 Prentice-Hall, Inc. All rights reserved. 0-13-239227-5 Vector Clocks Lamport’s clocks provide a total ordering, even where it shouldn’t. Vector clocks are constructed by letting each process P i maintain a vector VC i with the following two properties: 1. Before executing an event P i executes VC i [i] ← VC i [i] + 1. 2.When process P i sends a message m to P j, it sets m’s (vector) timestamp ts(m) equal to VC i after having executed the previous step. 3.Upon the receipt of a message m, process P j adjusts its own vector by setting VC j [k] ← max{VC j [k], ts(m)[k]} for each k, after which it executes the first step and delivers the message to the application.

5. Tanenbaum & Van Steen, Distributed Systems: Principles and Paradigms, 2e, (c) 2007 Prentice-Hall, Inc. All rights reserved. 0-13-239227-5 Enforcing Causal Communication When a process P j receives a message from P i, the middleware layer will delay till: 1. ts(m)[i] = VC j [i] + 1 2. ts(m)[k] <= VC j [k] for all k != i

6. Tanenbaum & Van Steen, Distributed Systems: Principles and Paradigms, 2e, (c) 2007 Prentice-Hall, Inc. All rights reserved. 0-13-239227-5 A Comparison of mutual exclusion Algorithms

7. Tanenbaum & Van Steen, Distributed Systems: Principles and Paradigms, 2e, (c) 2007 Prentice-Hall, Inc. All rights reserved. 0-13-239227-5 Outline Synchronization –Time synchronization –Logical clocks Mutual exclusion Global positioning Election

8. Tanenbaum & Van Steen, Distributed Systems: Principles and Paradigms, 2e, (c) 2007 Prentice-Hall, Inc. All rights reserved. 0-13-239227-5 Election Algorithms Many distributed algorithms require one process to act as a coordinator. In general, it doesn’t matter which process, but one of them must do it, and only one. –Can’t have two processes in a centralized mutual exclusion algorithm, for example. Election algorithms are for this purpose. –Outcome is one, and only one “winner”. –All processes must agree. Biggest challenge is handling failures.

9. Tanenbaum & Van Steen, Distributed Systems: Principles and Paradigms, 2e, (c) 2007 Prentice-Hall, Inc. All rights reserved. 0-13-239227-5 Bully Algorithm (Full connected) When any process notices that the current coordinator is down, it starts an election. A process P holds an election as follows: 1.P sends an ELECTION message to all processes with higher numbers. 2.If no one responds, P wins the election and becomes coordinator. 3.If one of the higher-ups answers, it takes over. P’s job is done.

10. Tanenbaum & Van Steen, Distributed Systems: Principles and Paradigms, 2e, (c) 2007 Prentice-Hall, Inc. All rights reserved. 0-13-239227-5 Bully Algorithm (Full connected) When a process gets an ELECTION message, it sends an OK back, indicating that he will take over. He then holds a (recursive) election himself. When a process knows that it has won, it sends a winner message to all other processes. When a process comes back up, it holds an election.

11. Tanenbaum & Van Steen, Distributed Systems: Principles and Paradigms, 2e, (c) 2007 Prentice-Hall, Inc. All rights reserved. 0-13-239227-5 The Bully Algorithm (1) (a) Process 4 first notices 7 is down and holds an election. (b) Processes 5 and 6 respond, telling 4 to stop. (c) Now 5 and 6 each hold an election.

12. Tanenbaum & Van Steen, Distributed Systems: Principles and Paradigms, 2e, (c) 2007 Prentice-Hall, Inc. All rights reserved. 0-13-239227-5 The Bully Algorithm (2) (d) Process 6 tells 5 to stop. (e) Process 6 wins and tells everyone.

13. Tanenbaum & Van Steen, Distributed Systems: Principles and Paradigms, 2e, (c) 2007 Prentice-Hall, Inc. All rights reserved. 0-13-239227-5 A Ring Algorithm (organized to a ring) When a process notices the coordinator is down, it builds an ELECTION message containing its own process number, and sends the message to its successor. –If successor is down, it will skip it. At each step, each process adds its own process number, and sends it along. Eventually returns to the first process. Message is then resent as COORDINATOR message to tell others who the new coordinator is, which is the process with the highest number.

14. Tanenbaum & Van Steen, Distributed Systems: Principles and Paradigms, 2e, (c) 2007 Prentice-Hall, Inc. All rights reserved. 0-13-239227-5 A Ring Algorithm Both 5 and 2 discover that 7 is down at the same time. Both circulate an ELECTION message around. Both know the highest ID and the same members and order, so send a COORDINATOR msg.

15. Tanenbaum & Van Steen, Distributed Systems: Principles and Paradigms, 2e, (c) 2007 Prentice-Hall, Inc. All rights reserved. 0-13-239227-5 Wireless Environments Unreliable, and nodes may move. Algorithm: 1.Any node starts by sending out an ELECTION message to neighbors. 2.When a node receives an ELECTION message for the first time, it forwards to neighbors, and designates the sender as its parent. 3.It then waits for responses from is neighbors. 1.Responses may carry resource information. 4.When a node receives an ELECTION message for the second time, it just OKs it.

16. Tanenbaum & Van Steen, Distributed Systems: Principles and Paradigms, 2e, (c) 2007 Prentice-Hall, Inc. All rights reserved. 0-13-239227-5 (a) (c) (b) (d)

17. Tanenbaum & Van Steen, Distributed Systems: Principles and Paradigms, 2e, (c) 2007 Prentice-Hall, Inc. All rights reserved. 0-13-239227-5 Elections in Wireless Environment (e) The build-tree phase. (f) Reporting of best node to source. In the end, source a knows h be the best leader and broadcast the info to all the nodes.

18. Tanenbaum & Van Steen, Distributed Systems: Principles and Paradigms, 2e, (c) 2007 Prentice-Hall, Inc. All rights reserved. 0-13-239227-5 Another algorithm: arbitrary networks r: round number L: process id {identifies the leader} Initially r = 0; L(i) = i; Do r < D //D is the diameter of the network –Send L(i) to each neighbor; –Receive all messages from the neighbors; –L(i) := max{L(i) U set of leader ids received from neighbors} –r = r+1 Od When it terminates ? Message overhead? How to compare to the previous algorithm?

19. Tanenbaum & Van Steen, Distributed Systems: Principles and Paradigms, 2e, (c) 2007 Prentice-Hall, Inc. All rights reserved. 0-13-239227-5 Anonymous election Each process uses one bit b, network complete connected Each process contender for the leader if active Initially b(i) = random{0,1}; Send b to every active neighbor; If i is the leader b(i) = random {0,1}, go to next round become passive Fi

20. Tanenbaum & Van Steen, Distributed Systems: Principles and Paradigms, 2e, (c) 2007 Prentice-Hall, Inc. All rights reserved. 0-13-239227-5 Elections in Large-Scale Systems Electing one node is not enough – several. For example: peer-to-peer networks Requirements for superpeer selection : 1.Normal nodes should have low-latency access to superpeers. 2.Superpeers should be evenly distributed across the overlay network. 3.There should be a predefined portion of superpeers relative to the total number of nodes in the overlay network. 4.Each superpeer should not need to serve more than a fixed number of normal nodes.

21. Tanenbaum & Van Steen, Distributed Systems: Principles and Paradigms, 2e, (c) 2007 Prentice-Hall, Inc. All rights reserved. 0-13-239227-5 peer-to-peer superpeer Algorithm 1 DHT-based systems. –m-bit identifier system –If N superpeers, k = [log2(N)] bits to for the N nodes. Leftmost. so each superpeer is in charge of the rest 2^(m-k) nodes. –How to we use it: –To look up a key p P AND 11100000 –To see if a node is a superpeer id AND 11100000

22. Tanenbaum & Van Steen, Distributed Systems: Principles and Paradigms, 2e, (c) 2007 Prentice-Hall, Inc. All rights reserved. 0-13-239227-5 peer-to-peer superpeer Algorithm Even Geo-distribution: –Token has repulsion forces. Start from random selection –Moving tokens away from others, until the forces balance. –Tokens know others position – gossip protocol to disseminate. –When a node holds a token for a while, it claims itself as a superpeer.

23. Tanenbaum & Van Steen, Distributed Systems: Principles and Paradigms, 2e, (c) 2007 Prentice-Hall, Inc. All rights reserved. 0-13-239227-5 Clustering A cluster is a subset of nodes which can communicate with a clusterhead and (possible) with each other. –An interconnected set of clusters covering the entire node population –Should not change the cluster structure too drastically when nodes move and topology changes Clusterhead (CH) –Local coordinator of transmission within the cluster. –Does extra work, may become bottleneck States: Init, CH, Ordinary, Gateway Election weight: e.g. ID

24. Tanenbaum & Van Steen, Distributed Systems: Principles and Paradigms, 2e, (c) 2007 Prentice-Hall, Inc. All rights reserved. 0-13-239227-5 Passive Clustering

25. Tanenbaum & Van Steen, Distributed Systems: Principles and Paradigms, 2e, (c) 2007 Prentice-Hall, Inc. All rights reserved. 0-13-239227-5 Lowest-ID clustering algorithm A node broadcast a list of nodes that it can hear (including itself) A node hears nodes with ID higher than itself is a CH. The lowest-ID that a node hears is its CH; unless the lowest-ID node specifically gives up its role as a CH (deferring to a lower ID node) A node hears two or more CH is a gateway (GW) Otherwise, a node is an ordinary node E.g. node 1,2,4 are CHs; node 8,9 are GWs

26. Tanenbaum & Van Steen, Distributed Systems: Principles and Paradigms, 2e, (c) 2007 Prentice-Hall, Inc. All rights reserved. 0-13-239227-5 Highest-connectivity clustering algorithm A node broadcast a list of nodes that it can hear (including itself) A node is elected as a CH if it is the most highly connected node of all its “ uncovered ” neighbors (in case of a tie, lowest ID prevails) A node which is not elected its CH is an “ uncovered ” node, otherwise, it is a “ covered ” node. A node which has already elected another node as its CH gives up its role as a CH E.g: node 5,7,8 are CHs node 2,3,9,10 are GWs

27. Tanenbaum & Van Steen, Distributed Systems: Principles and Paradigms, 2e, (c) 2007 Prentice-Hall, Inc. All rights reserved. 0-13-239227-5 Properties No CHs are directly linked –One CH per cluster In a cluster, any two nodes are at most two-hops away. In a cluster, each node is either a clusterhead itself or is directly linked to one or more clusterheads.

28. Tanenbaum & Van Steen, Distributed Systems: Principles and Paradigms, 2e, (c) 2007 Prentice-Hall, Inc. All rights reserved. 0-13-239227-5 evaluations Metrics: –# of nodes that changes their roles as CHs –# of node that switches clusters. Environment: –100X100, tx range varies, 30 nodes –mobility: (1) Random move; (2)Predefined trajectories

29. Tanenbaum & Van Steen, Distributed Systems: Principles and Paradigms, 2e, (c) 2007 Prentice-Hall, Inc. All rights reserved. 0-13-239227-5

31. Results: –Lowest-ID yields the fewest changes in either metric –A more stable cluster structure Reasons: –In the highest-Connectivity algorithm, if the CH drops even one link, it may fail to be re-elected as a CH. –While the lowest-ID node can still be a CH.

32. Tanenbaum & Van Steen, Distributed Systems: Principles and Paradigms, 2e, (c) 2007 Prentice-Hall, Inc. All rights reserved. 0-13-239227-5 Cluster-based Broadcast Clustering: grouping nodes into clusters Cluster head: a representative node of each group Gateway: a node connecting more than two clusters Ordinary nodes: Others Efficient Flooding: only cluster heads and gateways rebroadcast ClusterHead Gateway Ordinary Node S

33. Tanenbaum & Van Steen, Distributed Systems: Principles and Paradigms, 2e, (c) 2007 Prentice-Hall, Inc. All rights reserved. 0-13-239227-5 Large scale election and clustering Clustering: node subordinate to the leader (cluster header) –1-hop vs k-hop Partitioning Other election weights! –Lowest ID vs Highest Degree –First declare win (There are tons of MANET election algorithms)

34. Tanenbaum & Van Steen, Distributed Systems: Principles and Paradigms, 2e, (c) 2007 Prentice-Hall, Inc. All rights reserved. 0-13-239227-5 Election (find one or more coordinators) Fully connected networks (centralized) –Bully alg, Ring alg Multi-hop networks –Tree type (with an initiator) –Distributed Large-scale systems (multihop, many coordinators) –Superpeers DHT based Even Geo-distribution with token –Clustering lowest ID highest connectivity

35. Tanenbaum & Van Steen, Distributed Systems: Principles and Paradigms, 2e, (c) 2007 Prentice-Hall, Inc. All rights reserved. 0-13-239227-5 Outline Two more clustering algorithms

36. Tanenbaum & Van Steen, Distributed Systems: Principles and Paradigms, 2e, (c) 2007 Prentice-Hall, Inc. All rights reserved. 0-13-239227-5 Election: first declare win Clustering: –Four states: Init, ClusterHead, Ordinary, and Gateway. – –Set to CH, and delay for a random period –If during the period, receive a CH from a neighbor, set its state to Ordinary and send Ordinary. –If time expired, send CH. –With a tie, lowest ID win. –If an Ordinary hears two CHs, set to Gateway. Advantage?

37. Tanenbaum & Van Steen, Distributed Systems: Principles and Paradigms, 2e, (c) 2007 Prentice-Hall, Inc. All rights reserved. 0-13-239227-5 Election: relative mobility Require: –Mobile nodes exchange their positions and timestamp. –Calculate relative velocity with each neighbor. Clustering: –Exchange relative velocity instead of other weights. –Elect the node with the smallest relative velocity. Advantage?