1. Computer Science and Engineering Parallel and Distributed Processing CSE 8380 February 10, 2005 Session 9

2. Computer Science and Engineering Contents Message Passing Model Complexity Analysis Summation Leader Election

3. Computer Science and Engineering Introduction  Recent migrations to distributed systems have increased he need for a better understanding of message passing computational model and algorithms  A processing unit in such systems is an autonomous computer, which may be engaged in its own private computation while at the same time cooperating with other units in the context of some computational task.

4. Computer Science and Engineering Message Passing Computing Models  Algorithm – collection of local programs running concurrently on different processing units. Each program performs a sequence of computation and message passing operations  Message passing can be represented using a communication graph  Processor vs. Process

5. Computer Science and Engineering Degree of Synchrony Synchronous – Computation and communication are dome in a lockstep manner. (rounds of send, receive, compute) Asynchronous – processing units take steps at arbitrary speeds and communication delay is unpredictable Partially synchronous – restrictions on the relative timing events

6. Computer Science and Engineering Synchronous Model  n processors & communication graph G(V,E)  Modeled as a state machine  System is initialized and set to an arbitrary initial state  For each process i in V repeat in synchronized rounds  Send messages to outgoing neighbors by applying some message generation function to current state  Obtain the new state by applying a state transition function to the current state and the received messages

7. Computer Science and Engineering Asynchronous Model n processors & communication graph G(V,E) Communication does not happen in synchronized rounds Messages incur an unbounded and unpredictable delay I/O automata (simple type of state machine in which transitions are associated with actions) is used to model asynchronous systems

8. Computer Science and Engineering Synchronous model as a state machine  M, a fixed message alphabet  A process i can be modeled as  Qi, a (possibly infinite) set of states. The system state can be represented using a set of variables.  q0,i, the initial state in the state set Qi. The state variables have initial values in the initial state.  GenMsgi, a message generation function. It is applied to the current system state to generate messages to the outgoing neighbors from elements in M.  Transi, a state transition function that maps the current state and the incoming messages into a new state.

9. Computer Science and Engineering Algorithm Template Algorithm S_Template Q i q 0,i ← GenMsg i Trans i

10. Computer Science and Engineering Complexity Analysis  Message Complexity Number of messages sent between neighbors during the execution of the algorithm in the worst case  Time Complexity Synchronous – Number of rounds

11. Computer Science and Engineering Summation on a hypercube Algorithm S_Sum_Hypercube Q i buff, an integer dim, a value in {1, 2,..., log n} q 0,i buff ← xi dim ← log n GenMsg i If the current value of dim = 0, do nothing. Otherwise, send the current value of buff to the neighbor along the dimension dim. Trans i if the incoming message is v & dim > 0, then buff ← buff + v, dim ← dim - 1

12. Computer Science and Engineering Group Work Work the following example with your neighbor

13. Computer Science and Engineering Leader Election Problem  A leader among n processors is the one that is recognized by all other processors as distinguished to perform a special task  The problem occur when the processors must choose one of them as a leader.  Each processor should eventually decide whether or not it is a leader (each processor is only aware of its identification)  Most important when coordination among processors becomes necessary to recover from a failure or topological change.

14. Computer Science and Engineering A solution  Given a communication graph G = (V,E)  Two steps  Each node in the graph would broadcast its unique identifier to all other nodes  After receiving the identifier of all nodes, the node with the highest identifier declares itself as the leader  Complexity  Time  Message

15. Computer Science and Engineering Leader election in synchronous rings  By Chang and Roberts  Assumptions  Communication is unidirectional (clockwise)  The size of the ring is not known  The ID of each node is unique

16. Computer Science and Engineering Leader election in synchronous rings Algorithm 1. Each process sends its ID to its outgoing neighbor 2. When a process receives an ID from an incoming neighbor, then: The process sends null to its outgoing neighbor, if the received ID is less than its own ID The process sends the received ID to its outgoing neighbor, if the received ID is greater than its own ID The process declares itself as a leader, if the received identifier is equal to its own identifier

17. Computer Science and Engineering Complexity Analysis Given n processors connected via a ring Time Complexity – O(n) Message Complexity – O(n 2 ) Why?

18. Computer Science and Engineering Group Work 1- Work the following example with your neighbor 2- Study the improved leader election algorithm