1. CPSC 668Set 12: Causality1 CPSC 668 Distributed Algorithms and Systems Fall 2009 Prof. Jennifer Welch

2. CPSC 668Set 12: Causality2 Logical Clocks Motivation In an asynchronous system, often cannot tell which of two events occurred before the other: Example A Example B p0p0 p1p1 m0m0 m1m1 p0p0 p1p1 m0m0 m1m1

3. CPSC 668Set 12: Causality3 Logical Clocks Motivation In Example A, processors cannot tell which message was sent first. Probably not important. In Example B, processors can tell which message was sent first. Might be important. Let's try to determine relative ordering of some (not all) events.

4. CPSC 668Set 12: Causality4 Happens Before Partial Order Given an execution, computation event a happens before computation event b, denoted a  b, if 1.a and b occur at same processor and a precedes b, or 2.a results in sending m and b includes receipt of m, or 3.there exists computation event c such that a  c and c  b (transitive closure)

5. CPSC 668Set 12: Causality5 Happens Before Partial Order Happens before means that information can flow from a to b, i.e., that a might cause b. p0p0 p1p1 m0m0 m1m1 ad bc b  c a  b a  c c  d a  d b  d

6. CPSC 668Set 12: Causality6 Concurrent Events If a does not happen before b, and b does not happen before a, then a and b are concurrent, denoted a || b.

7. CPSC 668Set 12: Causality7 Happens Before Example Rule 1: a  b, c  d  e  f, g  h i Rule 2: a  d, g  e, f  i Rule 3: a  e, c  i, … h || e, …

8. CPSC 668Set 12: Causality8 Logical Clocks Logical clocks are values assigned to events to provide some information about the order in which events happen. Goal is to assign an integer L(e) to each computation event e in an execution such that if a  b, then L(a) < L(b).

9. CPSC 668Set 12: Causality9 Logical Timestamps Algorithm Each p i keeps a counter (logical timestamp) L i, initially 0 Every message p i sends is timestamped with current value of L i L i is incremented at each step to be greater than –its current value –the timestamps on all messages received at this step If a is an event at p i, then assign L(a) to be the value of L i at the end of a.

10. CPSC 668Set 12: Causality10 Logical Timestamps Example 1 234 12 5 2 1 a  b : L(a) = 1 < 2 = L(b) f  i : L(f) = 4 < 5 = L(i) a  e : L(a) = 1 < 3 = L(e) etc.

11. CPSC 668Set 12: Causality11 Getting a Total Order If a total order is required, break ties using ids. In the example, L(a) = (1,0), L(c) = (1,1), etc. Timestamps are ordered lexicographically. In the example, L(a) < L(c).

12. CPSC 668Set 12: Causality12 Drawback of Logical Clocks a  b implies L(a) < L(b), but L(a) < L(b) does not necessarily imply a  b. In previous example, L(g) = 1 and L(b) = 2, but g does not happen before b. Reason is that "happens before" is a partial order, but logical clock values are integers, which are totally ordered.

13. CPSC 668Set 12: Causality13 Vector Clocks Generalize logical clocks to provide non-causality information as well as causality information. Implement with values drawn from a partially ordered set instead of a totally ordered set. Assign a value V(e) to each computation event e in an execution such that a  b if and only if V(a) < V(b).

14. CPSC 668Set 12: Causality14 Vector Timestamps Algorithm Each p i keeps an n-vector V i, initially all 0's Entry j in V i is p i 's estimate of how many steps p j has taken Every msg p i sends is timestamped with current value of V i At every step, increment V i [i] by 1 When receiving a message with vector timestamp T, update V i 's components j ≠ i so that V i [j] = max(T[j],V i [j]) If a is an event at p i, then assign V(a) to be value of V i at end of a.

15. CPSC 668Set 12: Causality15 Manipulating Vector Timestamps Let V and W be two n-vectors of integers. Equality: V = W iff V[i] = W[i] for all i. Example: (3,2,4) = (3,2,4) Less than or equal: V ≤ W iff V[i] ≤ W[i] for all i. Example: (2,2,3) ≤ (3,2,4) and (3,2,4) ≤ (3,2,4) Less than: V < W iff V ≤ W but V ≠ W. Example: (2,2,3) < (3,2,4) Incomparable: V || W iff !(V ≤ W) and !(W ≤ V). Example: (3,2,4) || (4,1,4)

16. CPSC 668Set 12: Causality16 Manipulating Vector Timestamps The partial order on n-vectors just defined is not the same as lexicographic ordering. Lexicographic ordering is a total order on vectors. Consider (3,2,4) vs. (4,1,4) in the two approaches.

17. CPSC 668Set 12: Causality17 Vector Timestamps Example (1,0,0) (1,2,0)(1,3,1)(1,4,1) (0,0,1)(0,0,2) (1,4,3) (2,0,0) (0,1,0) V(g) = (0,0,1) and V(b) = (2,0,0), which are incomparable. Compare with logical clocks L(g) = 1 and L(b) = 2.

18. CPSC 668Set 12: Causality18 Correctness of Vector Timestamps Theorem (6.5 & 6.6): Vector timestamps implement vector clocks. Proof: First, show a  b implies V(a) < V(b). Case 1: a and b both occur at p i, a first. Since V i increases at each step, V(a) < V(b).

19. CPSC 668Set 12: Causality19 Correctness of Vector Timestamps Case 2: a occurs at p i and causes m to be sent, while b occurs at p j and includes the receipt of m. –During b, p j updates its vector timestamp in such a way that V(a) ≤ V(b). –p i 's estimate of number of steps taken by p j is never an over-estimate. Since m is not received before it is sent, p i 's estimate of the number of steps taken by p j when a occurs is less than the number of steps taken by p j when b occurs. So V(a)[j] < V(b)[j]. –Thus V(a) < V(b).

20. CPSC 668Set 12: Causality20 Correctness of Vector Timestamps Case 3: There exists c such that a  c and c  b. By induction (from Cases 1 and 2) and transitivity of <, V(a) < V(b). Next show V(a) < V(b) implies a  b. Equivalent to showing !(a  b) implies !(V(a) < V(b))

21. CPSC 668Set 12: Causality21 Correctness of Vector Timestamps Suppose a occurs at p i, b occurs at p j, and a does not happen before b. Let V(a)[i] = k. Since a does not happen before b, there is no chain of messages from p i to p j originating at p i 's k-th step or later and ending at p j before b. Thus V(b)[i] < k. Thus !(V(a) < V(b)).

22. CPSC 668Set 12: Causality22 Size of Vector Timestamps Vector timestamps are big: –n components in each one –values in the components grow without bound Is there a more efficient way to implement vector clocks? Answer is NO, at least under some conditions.

23. CPSC 668Set 12: Causality23 Vector Clock Size Lower Bound Theorem (6.9): Any implementation of vector clocks using vectors of real numbers requires vectors of length n (number of processors). Proof: For any value of n, consider this execution:

24. CPSC 668Set 12: Causality24 Example Bad Execution For n = 4:

25. CPSC 668Set 12: Causality25 Vector Clock Size Lower Bound Claim 1: a i+1 || b i for all i (with wraparound) Proof: Since each proc. does all sends before any receives, there is no transitivity. Also p i+1 does not send to p i. Claim 2: a i+1  b j for all j ≠ i. Proof: If j = i+1, obvious. If j ≠ i+1, then p i+1 sends to p j :

26. CPSC 668Set 12: Causality26 Vector Clock Size Lower Bound Suppose in contradiction, there is a way to implement vector clocks with k- vectors of reals, where k < n. By Claim 1, a i+1 || b i => V(a i+1 ) and V(b i ) are incomparable => V(a i+1 ) is larger than V(b i ) in some coordinate h(i) => h : {0,…,n-1}  {0,…,k}

27. CPSC 668Set 12: Causality27 Vector Clock Size Lower Bound Since k < n, the function h is not 1-1. So there exist distinct i and j such that h(i) = h(j). Let r be this common value of h. V(a 0 ) V(a 1 ) … V(a i+1 ) … V(a j+1 ) … V(a n-1 ) V(b 0 ) … V(b i ) … V(b j ) … V(b n-2 ) V(b n-1 ) > in h(0) comp > in h(i) comp > in h(j) comp > in h(n-2) comp > in h(n-1) comp two of these components are the same, say h(i) = h(j) = r

28. CPSC 668Set 12: Causality28 Vector Clock Size Lower Bound V(a i+1 ) V(a j+1 ) V(b i ) V(b j ) > in component r ≤ in all components, since a i+1  b j > in component r, contradicts a j+1  b i

29. CPSC 668Set 12: Causality29 Vector Clock Size Lower Bound So V(a i+1 ) is larger than V(b i ) in coordinate r and V(a j+1 ) is larger than V(b j ) in coordinate r also. V(a j+1 )[r] > V(b j )[r] by def. of r ≥ V(a i+1 )[r] by Claim 2 (a i+1  b j ) & correct. ≥ V(b i )[r] by def. of r Thus V(a j+1 ) !< V(b i ), contradicting Claim 2 (a j+1  b i ) and assumed correctness of V.

30. CPSC 668Set 12: Causality30 Application of Causality: Consistent Cuts Consider an asynchronous message passing system with –FIFO message delivery per channel –at most one msg received per computation step Number the computation steps of each processor 1,2,3,… A cut of an execution is K = (k 0,…,k n-1 ), where k i indicates number of computation steps taken by p i

31. CPSC 668Set 12: Causality31 Consistent Cuts In a consistent cut K = (k 0,…,k n-1 ), if step s of p j happens before step k i of p i, then s ≤ k j. (1,3) and (2,4) are consistent. (3,6) is inconsistent: step 4 by p 0 happens before step 6 of p 1, but 4 is greater than 3. some cuts

32. CPSC 668Set 12: Causality32 Finding a Recent Consistent Cut Problem Version 1: Processors all given a cut K and must find a maximal consistent cut that is ≤ K. Application: Logging-based crash recovery. –Procs periodically write their state to stable storage –When a proc recovers from a crash, it tries to recover to latest logged state, but needs to coordinate with other procs

33. CPSC 668Set 12: Causality33 Vector Clocks Solution Implement vector clocks using vector timestamps appended to application msgs. Store the vector clock of each computation step in a local array store[1,…] When p i is given input cut K: for x := K[i] downto 1 do if store[x] ≤ K then return x return x (entry for p i of global answer)

34. CPSC 668Set 12: Causality34 What About Channel State? Processor states are not sufficient to capture entire system state. Messages in transit must be calculated. Solution here requires –additional storage (number of messages) –additional computation at recovery time (involving replaying original execution to capture messages sent but not received)

35. CPSC 668Set 12: Causality35 Another Take on Recent Consistent State Problem Version 2: A subset of procs initiate (at arbitrary times) trying to find a consistent cut that includes the state of at least one of the initiators when it started. Called a distributed snapshot. Snapshot info can be collected at one proc. and then analyzed. Application: termination detection

36. CPSC 668Set 12: Causality36 Marker Algorithm Instead of adding extra information on each application message, insert control messages ("markers") into the channels. Code for p i : initially answer = -1 and num = 0 when application msg arrives: num++; do application action when marker arrives or when initiating snapshot: if answer = -1 then answer := num // p i 's part of final answer send marker to all neighbors

37. CPSC 668Set 12: Causality37 What About Channel States? p i records sequence of msgs received from p j between the time p i records its answer and the time p i gets the marker from p j These are the msgs in transit from p j to p i in the cut returned by the algorithm.