1. CPSC 668Set 1: Introduction1 CPSC 668 Distributed Algorithms and Systems Fall 2006 Prof. Jennifer Welch

2. CPSC 668Set 1: Introduction2 Distributed Systems Distributed systems have become ubiquitous: –share resources –communicate –increase performance speed fault tolerance Characterized by –independent activities (concurrency) –loosely coupled parallelism (heterogeneity) –inherent uncertainty

3. CPSC 668Set 1: Introduction3 Uncertainty in Distributed Systems Uncertainty comes from –differing processor speeds –varying communication delays –(partial) failures –multiple input streams and interactive behavior

4. CPSC 668Set 1: Introduction4 Reasoning about Distributed Systems Uncertainty makes it hard to be confident that system is correct To address this difficulty: –identify and abstract fundamental problems –state problems precisely –design algorithms to solve problems –prove correctness of algorithms –analyze complexity of algorithms (e.g., time, space, messages) –prove impossibility results and lower bounds

5. CPSC 668Set 1: Introduction5 Potential Payoff of Theoretical Paradigm careful specifications clarify intent increased confidence in correctness if abstracted well then results are relevant in multiple situations indicate inherent limitations (cf. NP- completeness)

6. CPSC 668Set 1: Introduction6 Application Areas These areas have provided classic problems in distributed/concurrent computing: –operating systems –(distributed) database systems –software fault-tolerance –communication networks –multiprocessor architectures

7. CPSC 668Set 1: Introduction7 Course Overview: Part I (Fundamentals) Introduce two basic communication models: –message passing –shared memory and two basic timing models: –synchronous –asynchronous

8. CPSC 668Set 1: Introduction8 Course Overview: Basic Models Message passing Shared memory synchronous asynchronous Yes No Yes (Synchronous shared memory model is PRAM)

9. CPSC 668Set 1: Introduction9 Course Overview: Part I Covers the canonical problems and issues: –graph algorithms (Ch 2) –leader election (Ch 3) –mutual exclusion (Ch 4) –fault-tolerant consensus (Ch 5) –causality and time (Ch 6)

10. CPSC 668Set 1: Introduction10 Course Overview: Part II (Simulations) Here "simulations" means abstractions, or techniques for making it easier to program, by making one model appear to be an easier model. For example: –broadcast and multicast (Ch 8) –distributed shared memory (Ch 9) –stronger kinds of shared variables (Ch 10) –more synchrony (Chs 11, 13) –more benign faults (Ch 12)

11. CPSC 668Set 1: Introduction11 Course Overview: Part II For each of the techniques: –describe algorithms for implementing it –analyze the cost of these algorithms –explore limitations mention applications that use the techniques

12. CPSC 668Set 1: Introduction12 Course Overview: Part III (Advanced Topics) Push further in some directions already introduced: –randomized algorithms (Ch 14) –stronger kinds of shared objects of arbitrary type (Ch 15) –what kinds of problems are solvable in asynchronous systems (Ch 16) –failure detectors (Ch 17) –self-stabilization

13. CPSC 668Set 1: Introduction13 Relationship of Theory to Practice time-shared operating systems: issues relating to (virtual) concurrency of processes such as –mutual exclusion –deadlock also arise in distributed systems MIMD multiprocessors: –no common clock => asynchronous model –common clock => synchronous model loosely coupled networks, such as Internet, => asynchronous model

14. CPSC 668Set 1: Introduction14 Relationship of Theory to Practice Failure models: –crash: faulty processor just stops. Idealization of reality. –Byzantine (arbitrary): conservative assumption, fits when failure model is unknown or malicious –self-stabilization: algorithm automatically recovers from transient corruption of state; appropriate for long-running applications

15. CPSC 668Set 1: Introduction15 Message-Passing Model processors are p 0, p 1, …, p n-1 (nodes of graph) bidirectional point-to-point channels (undirected edges of graph) each processor labels its incident channels 1, 2, 3,…; might not know who is at other end

16. CPSC 668Set 1: Introduction16 Message-Passing Model 1 1 2 2 11 3 2 p3p3 p2p2 p0p0 p1p1

17. CPSC 668Set 1: Introduction17 Modeling Processors and Channels Processor is a state machine including –local state of the processor –mechanisms for modeling channels Each channel directed from processor p 1 to processor p 2 is modeled in two pieces: –outbuf variable of p 1 and –inbuf variable of p 2 Outbuf corresponds to physical channel, inbuf to incoming message queue

18. CPSC 668Set 1: Introduction18 Modeling Processors and Channels inbuf[1] p 1 's local variables outbuf[1] inbuf[2] outbuf[2] p 2 's local variables Pink area (local vars + inbuf) is accessible state for a processor.

19. CPSC 668Set 1: Introduction19 Configuration Vector of processor states (including outbufs, i.e., channels), one per processor, is a configuration of the system Captures current snapshot of entire system: accessible processor states (local vars + incoming msg queues) as well as communication channels.

20. CPSC 668Set 1: Introduction20 Deliver Event Moves a message from sender's outbuf to receiver's inbuf; message will be available next time receiver takes a step p1p1 p2p2 m 3 m 2 m 1 p1p1 p2p2

21. CPSC 668Set 1: Introduction21 Computation Event Start with old accessible state (local vars + incoming messages) Apply transition function of processor's state machine; handles all incoming messages End with new accessible state with empty inbufs, and new outgoing messages

22. CPSC 668Set 1: Introduction22 c Computation Event d e old local state abab new local state

23. CPSC 668Set 1: Introduction23 Execution Format is config, event, config, event, config, … in first config: each processor is in initial state and all inbufs are empty for each consecutive (config, event, config), new config is same as old config except: –if delivery event: specified msg is transferred from sender's outbuf to receiver's inbuf –if computation event: specified processor's state (including outbufs) change according to transition function

24. CPSC 668Set 1: Introduction24 Admissibility Definition of execution gives some basic "syntactic" conditions. –usually safety conditions (true in every finite prefix) Sometimes we want to impose additional constraints –usually liveness conditions (eventually something happens) Executions satisfying the additional constraints are admissible. These are the executions that must solve the problem of interest.

25. CPSC 668Set 1: Introduction25 Asynchronous Executions An execution is admissible for the asynchronous model if –every message in an outbuf is eventually delivered –every processor takes an infinite number of steps No constraints on when these events take place: arbitrary message delays and relative processor speeds are not ruled out Models reliable system (no message is lost and no processor stops working)

26. CPSC 668Set 1: Introduction26 Example: Flooding Describe a simple flooding algorithm as a collection of interacting state machines. Each processor's local state consists of variable color, either red or green Initially: –p 0 : color = green, all outbufs contain M –others: color = red, all outbufs empty Transition: If M is in an inbuf and color = red, then change color to green and send M on all outbufs

27. CPSC 668Set 1: Introduction27 Example: Flooding p1p1 p0p0 p2p2 MM p1p1 p0p0 p2p2 M M deliver event at p 1 from p 0 computation event by p 1 deliver event at p 2 from p 1 p1p1 p0p0 p2p2 M M MM p1p1 p0p0 p2p2 M M computation event by p 2

28. CPSC 668Set 1: Introduction28 Example: Flooding (cont'd) deliver event at p 1 from p 2 computation event by p 1 deliver event at p 0 from p 1 etc. to deliver rest of msgs p1p1 p0p0 p2p2 M M M M p1p1 p0p0 p2p2 M M M M p1p1 p0p0 p2p2 M M M p1p1 p0p0 p2p2 M M M

29. CPSC 668Set 1: Introduction29 Nondeterminism The previous execution is not the only admissible execution of the Flooding algorithm on that triangle. There are several, depending on the order in which messages are delivered. For instance, the message from p 0 could arrive at p 2 before the message from p 1 does.

30. CPSC 668Set 1: Introduction30 Termination For technical reasons, admissible executions are defined as infinite. But often algorithms terminate. To model algorithm termination, identify terminated states of processors: states which, once entered, are never left Execution has terminated when all processors are terminated and no messages are in transit (in inbufs or outbufs)

31. CPSC 668Set 1: Introduction31 Complexity Measures These are worst-case. Message complexity: maximum number of messages sent in any admissible execution Time complexity: maximum "time" until termination in any admissible execution. But how is time measured in an asynchronous execution?

32. CPSC 668Set 1: Introduction32 Time Complexity Produce a timed execution from an execution by assigning non-decreasing real times to events such that time between sending and receiving any message is at most 1. Essentially normalizes the greatest message delay in an execution to be one time unit; still allows arbitrary interleavings of events. Time complexity: maximum time until termination in any timed admissible execution.

33. CPSC 668Set 1: Introduction33 Complexity of Flooding Algorithm Define terminated states to those in which color = green. Message complexity: one message is sent over each edge in each direction. So number is 2m, where m = number of edges. Time complexity: diameter + 1 time units. (A node turns green once a "chain" of messages has reached it from p 0.)

34. CPSC 668Set 1: Introduction34 Synchronous Message Passing Systems An execution is admissible for the synchronous model if it is of the form ((deliver)* (comp)*)* regular expression notation Each (deliver)* (comp)* is a round, in which –all messages in outbufs are delivered, then –each processor takes one computation step An admissible execution is an infinite sequence of rounds.

35. CPSC 668Set 1: Introduction35 Synchronous Message Passing Systems The new definition of admissible captures lockstep unison feature of synchronous model. This definition also implies –every message sent is delivered –every processor takes an infinite number of steps. Time is measured as number of rounds until termination.

36. CPSC 668Set 1: Introduction36 Example of Synchronous Model Suppose flooding algorithm is executed in synchronous model on the triangle. Round 1: –deliver M to p 1 from p 0 –deliver M to p 2 from p 0 –p 0 does nothing –p 1 turns green and sends M to p 0 and p 1 –p 2 turns green and sends M to p 0 and p 1

37. CPSC 668Set 1: Introduction37 Example of Synchronous Model Round 2: –deliver M to p 0 from p 1 –deliver M to p 0 from p 2 –deliver M to p 1 from p 2 –deliver M to p 2 from p 1 –p 0 does nothing –p 1 does nothing –p 2 does nothing

38. CPSC 668Set 1: Introduction38 p1p1 p0p0 p2p2 M MM M Example of Synchronous Model p1p1 p0p0 p2p2 MM p1p1 p0p0 p2p2 round 1 events round 2 events

39. CPSC 668Set 1: Introduction39 Complexity of Synchronous Flooding Algorithm Just consider executions that are admissible w.r.t. synchronous model Time complexity is diameter + 1 Message complexity is 2m Same as for asynchronous case. Not true for all algorithms though…