1. 1 Lecture 1 Distributed Algorithms Gadi Taubenfeld © 2011 Distributed Algorithms Gadi Taubenfeld Lecture 1 INTRODUCTION

2. 2 Distributed Algorithms Gadi Taubenfeld © 2011 A note on the use of these ppt slides: I am making these slides freely available to all (faculty, students, readers). They are in PowerPoint form so you can add, modify, and delete slides and slide content to suit your needs. They obviously represent a lot of work on my part. In return for use, I only ask the following:  That you mention their source, after all, I would like people to use my book!  That you note that they are adapted from (or perhaps identical to) my slides, and note my copyright of this material. Thanks and enjoy! Gadi Taubenfeld All material copyright 2011 Gadi Taubenfeld, All Rights Reserved A note on the use of these ppt slides: I am making these slides freely available to all (faculty, students, readers). They are in PowerPoint form so you can add, modify, and delete slides and slide content to suit your needs. They obviously represent a lot of work on my part. In return for use, I only ask the following:  That you mention their source, after all, I would like people to use my book!  That you note that they are adapted from (or perhaps identical to) my slides, and note my copyright of this material. Thanks and enjoy! Gadi Taubenfeld All material copyright 2011 Gadi Taubenfeld, All Rights Reserved This lecture is based on Chapter 1 from the textbook To get the most updated version of these slides go to: http://www.faculty.idc.ac.il/gadi/book.htm

3. 3 Lecture 1 Distributed Algorithms Gadi Taubenfeld © 2011 Synchronization  Type of synchronization m Contention m Coordination  Why synchronization is difficult? m Easy between humans m Communication between computers is restricted reading and writing of notes sending and receiving messages  Type of synchronization m Contention m Coordination  Why synchronization is difficult? m Easy between humans m Communication between computers is restricted reading and writing of notes sending and receiving messages

4. 4 Lecture 1 Distributed Algorithms Gadi Taubenfeld © 2011 The Too-Much-Milk Problem TimeAliceBob 5:00Arrive Home 5:05Look in fridge; no milk 5:10Leave for grocery 5:15Arrive home 5:20Arrive at groceryLook in fridge; no milk 5:25Buy milkLeave for grocery 5:30Arrive home; put milk in fridge 3:40Buy milk 3:45Arrive home; too much milk!

5. 5 Lecture 1 Distributed Algorithms Gadi Taubenfeld © 2011 Solving the Too-Much-Milk Problem Correctness properties  Mutual Exclusion: Only one person buys milk at a time  Progress: Someone always buys milk if it is needed Communication primitives  Leave a note (set a flag)  Remove a note (reset a flag)  Read a note (test the flag)

6. 6 Lecture 1 Distributed Algorithms Gadi Taubenfeld © 2011 Important Distinction  Safety Property m Nothing bad happens ever m Example: mutual exclusion  Liveness Property m Something good happens eventually m Example: progress  Safety Property m Nothing bad happens ever m Example: mutual exclusion  Liveness Property m Something good happens eventually m Example: progress

7. 7 Lecture 1 Distributed Algorithms Gadi Taubenfeld © 2011 Solution 1 Does it work? Alice if (no note) { if (no milk) { leave Note buy milk remove note } Bob if (no note) { if (no milk) { leave Note buy milk remove note } No, may end up with two milk.  mutual exclusion progress

8. 8 Lecture 1 Distributed Algorithms Gadi Taubenfeld © 2011 Solution 2 Alice leave note A if (no note B) { if (no milk) {buy milk} } remove note A Bob leave note B if (no note A) { if (no milk) {buy milk} } remove note B Using labeled notes Does it work? No, may end up with no milk. mutual exclusion  progress

9. 9 Lecture 1 Distributed Algorithms Gadi Taubenfeld © 2011 Solution 3 Does it work? Alice leave note A while (note B) {skip} if (no milk) {buy milk} remove note A Bob leave note B if (no note A) { if (no milk) {buy milk} } remove note B No ! a timing assumption is needed mutual exclusion  progress Bob’s code is as before

10. 10 Lecture 1 Distributed Algorithms Gadi Taubenfeld © 2011 Is solution #3 a good solution? No  A timing assumption is needed!  Unfair to one of the processes  It is asymmetrical and complicated  Alice is busy waiting – consuming CPU cycles without accomplishing useful work  A timing assumption is needed!  Unfair to one of the processes  It is asymmetrical and complicated  Alice is busy waiting – consuming CPU cycles without accomplishing useful work

11. 11 Lecture 1 Distributed Algorithms Gadi Taubenfeld © 2011 Solution 4 Using 4 labeled notes A1A2B2B1 the fridge’s door

12. 12 Lecture 1 Distributed Algorithms Gadi Taubenfeld © 2011 Solution 4 Notations color: there is a note A1A2B2 Alice’s turn B2 dotted line: there is no note solid line without color: we do not care if there is a note who will buy milk ? XX

13. 13 Lecture 1 Distributed Algorithms Gadi Taubenfeld © 2011 A1A2B2 Alice’s turn A1A2B2B1 Alice’s turn A2B2B1 Bob’s turn A1A2B2B1 Alice’s turn A1A2B2B1 Bob’s turn A1A2B2B1 Bob’s turn Solution 4 B2 XX XX

14. 14 Lecture 1 Distributed Algorithms Gadi Taubenfeld © 2011 Alice leave A1 if B2 {leave A2} else {remove A2} while B1 and ((A2 and B2) or (no A2 and no B2)) {skip} if (no milk) {buy milk} remove A1 Bob leave B1 if (no A2) {leave B2} else {remove B2} while A1 and ((A2 and no B2) or (no A2 and B2)) {skip} if (no milk) {buy milk} remove note B1 Solution 4 A correct, fair, symmetric algorithm!

15. 15 Lecture 1 Distributed Algorithms Gadi Taubenfeld © 2011 A variant of Solution 4 A variant of Solution 4 The order of the first two statements is replaced Is it correct ? No, may end up with two milk.  mutual exclusion progress Alice if B2 {leave A2} else {remove A2} leave A1 while B1 and ((A2 and B2) or (no A2 and no B2)) {skip} if (no milk) {buy milk} remove A1 Bob if (no A2) {leave B2} else {remove B2} leave B1 while A1 and ((A2 and no B2) or (no A2 and B2)) {skip} if (no milk) {buy milk} remove note B1 MILK A1B2B1

16. 16 Lecture 1 Distributed Algorithms Gadi Taubenfeld © 2011 Question Alice and Bob have a daughter C, make it work for all the three of them.

17. 17 Lecture 1 Distributed Algorithms Gadi Taubenfeld © 2011 Question Process A for i = 1 to 5 { x:=x+1 } Process B for i = 1 to 5 { x:=x+1 } x is an atomic register, initially 0 Q: What are all the possible values for x after both processes terminate? A: 2 through 10 inclusive. 0 0

18. 18 Lecture 1 Distributed Algorithms Gadi Taubenfeld © 2011 The smallest value is 2 0 0 A 0 1 B 01 1 1 2 2 3 3 4 4 1 1 2 2 3 3 4 4 5 5 12 2 2

19. 19 Lecture 1 Distributed Algorithms Gadi Taubenfeld © 2011 Question Process A x:=x+1; x:=x+1 Process B x:=x+1; x:=x+1 Process C x:=10 x is an atomic register, initially 0 Q: What are all the possible values for x after the three processes terminate? A: 2,3,4,10,11,12,13,14. 0 0

20. 20 Lecture 1 Distributed Algorithms Gadi Taubenfeld © 2011 The coordinated attack problem Blue army wins if both blue camps attack simultaneously Design an algorithm to ensure that both blue camps attack simultaneously Enemy 12 The problem is due to by Jim Gray (1977)

21. 21 Lecture 1 Distributed Algorithms Gadi Taubenfeld © 2011 The coordinated attack problem Communication is done by sending messengers across valley Messengers may be lost or captured by the enemy  Enemy 12

22. 22 Lecture 1 Distributed Algorithms Gadi Taubenfeld © 2011 The coordinated attack problem Theorem: There is no algorithm that solves the problem ! Assume to the contrary that such an algorithm exits. Let P be an algorithm that solves with fewest messages. P should work even when the last messenger is captured. So P should work even if the last messenger is never sent. But this is a new algorithm with one less message. A contradiction. Proof:  The (asynchronous) model is too weak.

23. 23 Lecture 1 Distributed Algorithms Gadi Taubenfeld © 2011 The first problem we cover in detail. A generalization of the too-much-milk problem. The mutual exclusion problem Chapters 2+3

24. 24 Lecture 1 Distributed Algorithms Gadi Taubenfeld © 2011 Blocking and Non-blocking Synchronization Concurrent data structures P1P2P3 data structure counter, stack, queue, link list Chapter 4

25. 25 Lecture 1 Distributed Algorithms Gadi Taubenfeld © 2011 Blocking P1P2P3 data structure Chapter 4

26. 26 Lecture 1 Distributed Algorithms Gadi Taubenfeld © 2011 Non-blocking P1P2P3 data structure Chapter 4

27. 27 Lecture 1 Distributed Algorithms Gadi Taubenfeld © 2011 Coarse-Grained Locking Easier to program, but not scalable.

28. 28 Lecture 1 Distributed Algorithms Gadi Taubenfeld © 2011 Fine-Grained Locking Efficient and scalable, but too complicated (deadlocks, etc.).

29. 29 Lecture 1 Distributed Algorithms Gadi Taubenfeld © 2011 Consensus 1 1 0 1 1 1 1 1 0 1 1 1 0 1 requirements:  validity  agreement  termination

30. 30 Lecture 1 Distributed Algorithms Gadi Taubenfeld © 2011 Impossibility of Consensus with One Faulty Process What can be done?  Strong primitives.  Timing assumptions.  Randomizations. Theorem: There is no algorithm that solve the consensus problem using atomic read/write registers in the presence of a single faulty process. We give a detailed proof of this important result in Chapter 9 of the book.

31. 31 Lecture 1 Distributed Algorithms Gadi Taubenfeld © 2011 Barrier P1 P2 P3 P4 Barrier P1 P2 P3 P4 P1 P2 P3 P4 time Barrier Barrier Synchronization Chapter 5 four processes approach the barrier all except P4 arrive Once all arrive, they continue

32. 32 Lecture 1 Distributed Algorithms Gadi Taubenfeld © 2011 Example: Prefix Sum a begin bcdef a end a+ba+b+ca+b+c+d a+b+c +d+e a+b+c +d+e+f time

33. 33 Lecture 1 Distributed Algorithms Gadi Taubenfeld © 2011 Example: Sequential Prefix Sum a begin bcdef aa+bcdefa a+b+ca+b+c+def a end a+ba+b+ca+b+c+d a+b+c +d+e a+b+c +d+e+f aa+ba+b+cdefaa+ba+b+ca+b+c+d a+b+c+d+e f time

34. 34 Lecture 1 Distributed Algorithms Gadi Taubenfeld © 2011 Example: Parallel Prefix Sum a begin bcdef aa+bb+cc+dd+ee+faa+ba+b+ca+b+c+db+c+d+ec+d+e+f a end a+ba+b+ca+b+c+d a+b+c +d+e a+b+c +d+e+f time

35. 35 Lecture 1 Distributed Algorithms Gadi Taubenfeld © 2011 Example: Parallel Prefix Sum a begin bcdef aa+bb+cc+dd+ee+f aa+ba+b+ca+b+c+db+c+d+ec+d+e+f a end a+ba+b+ca+b+c+d a+b+c +d+e a+b+c +d+e+f barrier time

36. 36 Lecture 1 Distributed Algorithms Gadi Taubenfeld © 2011 Leader election 0010000  O(n log n) messages is possible.

37. 37 Lecture 1 Distributed Algorithms Gadi Taubenfeld © 2011 Distributed minimum weight spanning tree Graph algorithms 4  O(n log n + E) message complexity is possible.  MIS, BFS,... 3 5 1 2 5 9 5 8 7

38. 38 Lecture 1 Distributed Algorithms Gadi Taubenfeld © 2011 Snapshot How to “learn” the global state of a network by collecting the local state of all the processes (and of the links) deadlock detection, termination detection, etc.

39. 39 Lecture 1 Distributed Algorithms Gadi Taubenfeld © 2011 1. P1 reports 200 2. P1 transfer 100 to P2 3. P2 reports 200 Manager knows of 400 $ 1. P1 reports 200 2. P1 transfer 100 to P2 3. P2 reports 200 Manager knows of 400 $ M P2 200100 200 100 Snapshot (example) Solutions  Freezing  Global clock  Sending everything via M  The snapshot algorithm !!! Solutions  Freezing  Global clock  Sending everything via M  The snapshot algorithm !!! P1

40. 40 Lecture 1 Distributed Algorithms Gadi Taubenfeld © 2011 End-to-end communication sender network receiver

41. 41 Lecture 1 Distributed Algorithms Gadi Taubenfeld © 2011 Renaming Renaming reducing the name space 1 100 5 2 500 7 7 1 999 13 700 5 88 10 n 777,888 3  Tight bound of 2n-1 names (using atomic registers).

42. 42 Lecture 1 Distributed Algorithms Gadi Taubenfeld © 2011 Clock synchronization Clock synchronization (Data synchronization) The problem requires processors to bring their clocks close together, by using communication among them.

43. 43 Lecture 1 Distributed Algorithms Gadi Taubenfeld © 2011 Communication models of computation Concurrent programming  Shared memory m safe registers m atomic registers (read/write) m test&set m swap m compare-and-swap m load-link/store-conditional m read-modify-write m semaphore m monitor Concurrent programming  Shared memory m safe registers m atomic registers (read/write) m test&set m swap m compare-and-swap m load-link/store-conditional m read-modify-write m semaphore m monitor What is the relative power of these communication primitives? Answer: see Chapter 9.

44. 44 Lecture 1 Distributed Algorithms Gadi Taubenfeld © 2011 Communication models of computation Message passing m send/receive m multi-cast m broadcast m network topology Message passing m send/receive m multi-cast m broadcast m network topology

45. 45 Lecture 1 Distributed Algorithms Gadi Taubenfeld © 2011 Time models of computation Synchronous (CRCW, EREW) AsynchronousPartially synchronous What is the relative power of these timing models? Concurrent/ Distributed Parallel less powerful realistic extremely powerful unrealistic

46. 46 Lecture 1 Distributed Algorithms Gadi Taubenfeld © 2011 How many can fail ? m constant, n/3,... Wait-freedom, Non-blocking, Lock-freedom m what is it ? Self-stabilization m what is it ? How many can fail ? m constant, n/3,... Wait-freedom, Non-blocking, Lock-freedom m what is it ? Self-stabilization m what is it ? Fault-tolerance models of computation What can fail? m processes m shared memory m links Type of faults m crash m omission m Byzantine What can fail? m processes m shared memory m links Type of faults m crash m omission m Byzantine

47. 47 Lecture 1 Distributed Algorithms Gadi Taubenfeld © 2011 Terminology  Hardware m Processors  Software m Threads, processes  Sometimes OK to confuse them, sometimes not.  Scheduler  Hardware m Processors  Software m Threads, processes  Sometimes OK to confuse them, sometimes not.  Scheduler In this course it is always ok to confuse between processes and threads.

48. 48 Lecture 1 Distributed Algorithms Gadi Taubenfeld © 2011 ProblemsModels read/write  Algorithms  Upper and lower bounds  Impossibility results  Simulations & Implementations Problems & Models barrier send/receive swap LL/SC readers& writers consensus mutex

49. 49 Lecture 1 Distributed Algorithms Gadi Taubenfeld © 2011 History of the mutex problem  1965 m Defined by Dijkstra m First solution by Dekker m General solution by Dijkstra  Fischer, Knuth, Lynch, Rabin, Rivest,...  1974 Lamport’s bakery algorithm  1981 Peterson’s solution  There are hundreds of published solutions  Lots or related problems E. W. Dijkstra www.cs.utexas.edu/users/EWD/ (Photo by Hamilton Richards 2002)