1. CS542 Topics in Distributed Systems Diganta Goswami

2. Algorithms to Find Global States Why? –(Distributed) garbage collection [think multiple processes sharing and referencing objects] –(Distributed) deadlock detection, termination [think database transactions] –Global states most useful for detecting stable predicates : once true always stays true (unless you do something about it) »e.g., once a deadlock, always stays a deadlock What? –Global state=states of all processes + states of all communication channels –Capture the instantaneous state of each process –And the instantaneous state of each communication channel, i.e., messages in transit on the channels How? –We’ll see this lecture!

3. Obvious First Solution… Synchronize clocks of all processes Ask all processes to record their states at known time t Problems? –Time synchronization possible only approximately (but distributed banking applications cannot take approximations) –Does not record the state of messages in the channels Again: synchronization not required – causality is enough!

4. Two Processes and Their Initial States

5. Execution of the Processes p 1 p 2 (empty) (empty) c 2 c 1 1. Global state S 0 2. Global state S 1 3. Global state S 2 4. Global state S 3 p 1 p 2 (Order 10, $100) (empty) c 2 c 1 p 1 p 2 (Order 10, $100) (five widgets) c 2 c 1 p 1 p 2 (Order 10, $100) (empty) c 2 c 1 Send 5 freebie widgets!

6. Cuts  Cut = time frontier, one at each process  f  cut C iff f is to the left of the frontier C P1 P2 P3 e10e10 e11e11 e12e12 e13e13 e20e20 e21e21 e22e22 e30e30 e31e31 e32e32 Inconsistent cut Consistent cut

7. Consistent Cuts  f  cut C iff f is to the left of the frontier C  A cut C is consistent if and only if  e  C (if f  e then f  C)  A global state S is consistent if and only if it corresponds to a consistent cut  A consistent cut == a global snapshot P1 P2 P3 e10e10 e11e11 e12e12 e13e13 e20e20 e21e21 e22e22 e30e30 e31e31 e32e32 Inconsistent cut Consistent cut Lamport’s “happens-before”

8. The “Snapshot” Algorithm  Problem: Record a set of process and channel states such that the combination is a global snapshot/consistent cut.  System Model:  There is a uni-directional communication channel between each ordered process pair (Pj  Pi and Pi  Pj)  Communication channels are FIFO-ordered  No failure, all messages arrive intact, exactly once  Any process may initiate the snapshot (by sending a special message called “Marker”)  Snapshot does not require application to stop sending messages, does not interfere with normal execution  Each process is able to record its state and the state of its incoming channels (no central collection)

9. The “Snapshot” Algorithm (2) 1. Marker sending rule for initiator process P 0  After P 0 has recorded its own state for each outgoing channel C, send a marker message on C 2. Marker receiving rule for a process P k on receipt of a marker over channel C  if P k has not yet received a marker -record P k ’s own state -record the state of C as “empty” -for each outgoing channel C, send a marker on C -turn on recording of messages over other incoming channels -else -record the state of C as all the messages received over C since P k saved its own state; stop recording state of C

10. Chandy and Lamport’s ‘Snapshot’ Algorithm Marker receiving rule for process p i On p i ’s receipt of a marker message over channel c: if (p i has not yet recorded its state) it records its process state now; records the state of c as the empty set; turns on recording of messages arriving over other incoming channels; else p i records the state of c as the set of messages it has received over c since it saved its state. end if Marker sending rule for process p i After p i has recorded its state, for each outgoing channel c: p i sends one marker message over c (before it sends any other message over c).

11. Snapshot Example P1 P2 P3 e10e10 e20e20 e23e23 e30e30 e13e13 a b M e 1 1,2 M 1- P1 initiates snapshot: records its state (S1); sends Markers to P2 & P3; turns on recording for channels C21 and C31 e 2 1,2,3 M M 2- P2 receives Marker over C12, records its state (S2), sets state(C12) = {} sends Marker to P1 & P3; turns on recording for channel C32 e14e14 3- P1 receives Marker over C21, sets state(C21) = {a} e 3 2,3,4 M M 4- P3 receives Marker over C13, records its state (S3), sets state(C13) = {} sends Marker to P1 & P2; turns on recording for channel C23 e24e24 5- P2 receives Marker over C32, sets state(C32) = {b} e31e31 6- P3 receives Marker over C23, sets state(C23) = {} e13e13 7- P1 receives Marker over C31, sets state(C31) = {}

12. Provable Assertion: Chandy-Lamport algo. determines a consistent cut Let e i and e j be events occurring at p i and p j, respectively such that e i  e j The snapshot algorithm ensures that if e j is in the cut then e i is also in the cut. if e j , then it must be true that e i . By contradiction, suppose  e i Consider the path of app messages (through other processes) that go from e i  e j Due to FIFO ordering, markers on each link in above path precede regular app messages Thus, since  e i, it must be true that p j received a marker before e j Thus e j is not in the cut => contradiction

13. Formally Speaking…. Process Histories  For a process P i, where events e i 0, e i 1, … occur: history(P i ) = h i = prefix history(P i k ) = h i k = S i k : P i ’s state immediately after k th event  For a set of processes P 1, …,P i, …. : global history: H =  i (h i ) global state: S =  i (S i k i )  channels a cut C  H = h 1 c1  h 2 c2  …  h n cn the frontier of C = {e i ci, i = 1,2, … n}

14. Global States useful for detecting Global Predicates  A cut is consistent if and only if it does not violate causality  A Run is a total ordering of events in H that is consistent with each h i ’s ordering  A Linearization is a run consistent with happens- before (  ) relation in H (history of all events).  Linearizations pass through consistent global states.  A global state S k is reachable from global state S i, if there is a linearization, L, that passes through S i and then through S k.  The distributed system evolves as a series of transitions between global states S 0, S 1, ….

15. Reachability between states in the snapshot algorithm S init S final S snap actual execution e 0,e 1,... recording begins ends pre-snap: e ' 0,e ' 1,...e ' R-1 post-snap: e ' R,e ' R+1,... '

16. Distributed debugging  Examine the problem of recording a system’s global state so that we may make useful statements about whether a transitory state – as opposed to a stable state – occurred in an actual execution  This is what we require, in general, when debugging a distributed system  Is |x i – x j | <=  where x i is a variable in process P i

17. Distributed debugging  Chandy and Lamport’s algorithm collects state in a distributed fashion  The processes in the system can send the state they gather to a monitor process for collection  Algorithm [Marzullo and Neiger, ‘91] – The observed processes send their states to a process called a monitor, which assembles globally consistent states from what it receives  The monitor lie outside the system, observing its execution

18. Distributed debugging  Goal is to determine cases when a given global state predicate  was definitely True at some point in the execution we observed, and cases when it was possibly True  Possibly – because we may extract a consistent global state S from an executing system and find that  (S) is True.  No single observation of a consistent global state allows us to conclude whether a non-stable predicate ever evaluated to True in the actual execution

19. Distributed debugging  Possibly  : There is a consistent global state S through which a linearization of H passes such that  (S) is True  Definitely  : For all linearization L of H, there is a consistent global state S through which L passes such that  (S) is True

20. Distributed debugging  We now describe  How the process states are collected  How the monitor extracts consistent global states  How the monitor evaluates possibly  and definitely  in both asynchronous and synchronous systems

21. Distributed debugging  The observed processes p i (I = 1, 2, … N) send their initial state to monitor process initially, and thereafter from time to time, in state messages  No need to send state except initially and when it changes  Global state predicate may depend only on certain parts of the process’ states – hence need only send relevant state  Need only send state at times when the predicate may become True or cease to be True  The monitor process records the state messages from process pi in a separate queue Q i, for each i= 1, 2, … N

22. Distributed debugging  In order that the monitor can distinguish consistent global states from inconsistent global states, the observed processes enclose their vector clock values with their state messages  Each queue Q i is kept ordered in sending order (can be established by examining the i-th component of the vector clock)

23. Distributed debugging  Let S = (s 1, s 2, …, S N ) be a global statedrawn from the state messages that the monitor has received. Let V(s i ) be the vector clock of the state s i received from p i  S is a consistent global state iff  V(s i )[i] >= V(s j )[i] for i, j = 1, 2, …, N  That is, the no. of p i ’s events known at p j when it sent s j is no more than the no. of events that have occurred at p i when it sent s i.  Hence, if one process’s state depends upon another, then the global state also encompasses the state upon which it depends

24. Distributed debugging  The monitor process may establish whether a given global state is consistent, using the vector timestamps sent by the observed processes  It can construct a lattice of consistent global states corresponding to the execution of the processes – captures the relation of reachability between consistent global states  The nodes denote global states, and the edges denote possible transitions between these states

25. Vector timestamps and variable values m 1 m 2 p 1 p 2 Physical time Cut C 1 (1,0)(2,0)(4,3) (2,1)(2,2)(2,3) (3,0) x 1 = 1x 1 = 100x 1 = 105 x 2 = 100x 2 = 95x 2 = 90 x 1 Cut C 2

26. The lattice of global states for the execution of previous Fig Sij = global state after i events at process 1 and j events at process 2 S 00 S 10 S 20 S 21 S 30 S 31 S 32 S 22 S 23 S 33 S 43 Level 0 1 2 3 4 5 6 7

27. Distributed debugging  A linearization traverses the lattice from any global state to any global state reachable from it on the next level – that is, in each step some process experiences one event. For ex. S 22 is reachable from S 20, but S 22 is not reachable from S 30.  The lattice shows all linearizations corresponding to a history  A monitor process can now evaluate possibly  and definitely 

28. Distributed debugging  To evaluate possibly , the monitor process starts at the initial state and steps through all consistent states reachable from that point, evaluating  at each stage. It stops when  evaluates to True  To evaluate definitely , the monitor process must attempt to find a set of states through which all linearizations must pass, and at each of which  evaluates to True  Note that, the state S’ is reachable from S iff  V(s j )[j] >= V(s’ i )[j] for j = 1, 2, …, N, j ≠ i

29. Algorithms to evaluate possibly  and definitely 

30. Global State Predicates  A global-state-predicate is a function from the set of global states to {true, false},e.g., deadlock, termination  A global state S0 satisfies liveness property P iff: liveness(P(S 0 ))   L  linearizations from S0 L passes through an S L & P(S L ) = true  Ex: P(S) = the computation will terminate  A global state S0 satisfies this safety property P if: safety(P(S 0 ))   S reachable from S 0, P(S) = false  Ex: P(S) = S has a deadlock  Global states often useful for detecting stable global- state-predicate: it is one that once it becomes true, it remains true in subsequent global states, e.g., an object O is orphaned, or deadlock  A stable predicate may be a safety or liveness predicate

31. Liveness versus Safety Can be confusing, but terms are very important: Liveness=guarantee that something good will happen, eventually –“Guarantee of termination” is a liveness property –Guarantee that “at least one of the atheletes in the 100m final will win gold” is liveness –A criminal will eventually be jailed –Completeness in failure detectors Safety=guarantee that something bad will never happen –Deadlock avoidance algorithms provide safety –A peace treaty between two nations provides safety –An innocent person will never be jailed –Accuracy in failure detectors Can be difficult to satisfy both liveness and safety!