1. Consistent Global States of Distributed Systems: Fundamental Concepts and Mechanisms CS 249 Project Fall 2005 Wing Wong

2. Consist Global States2 Outline Introduction Asynchronous distributed systems, distributed computations, consistency Two different strategies to construct global states  Monitor passively observes the system (reactive- architecture)  Monitor actively interrogates the system (snapshot protocol) Properties of global predicates Sample applications: deadlock detection and debugging

3. Consist Global States3 Introduction global state = union of local states of individual processes many problems in distributed computing require:  construction of a global state and  evaluation of whether the state satisfies some predicate Φ difficulties:  uncertainties in message delays  relative speeds of computations global state obtained can be obsolete, incomplete, or inconsistent

4. Consist Global States4 Distributed Systems collection of sequential processes p 1, p 2, …, p n unidirectional communication channels between pairs of processes reliable channels messages may be delivered out of order network strongly connected (not necessarily completely)

5. Consist Global States5 Asynchronous Distributed Systems no bounds on relative process speeds no bounds on message delays no synchronized local clocks communication is the only possible mechanism for synchronization

6. Consist Global States6 Distributed Computations distributed program executed by a collection of processes each process executes a sequence of events communication through events send(m) and receive(m), m as message identifier

7. Consist Global States7 Distributed Computations h i = e i 1 e i 2 …  local history of process p i  canonical enumeration  total order imposed by sequential execution h i k = e i 1 e i 2 … e i k  initial prefix of h i containing first k events H = h 1 U … U h n  global history containing all events  does not specify relative timing between events

8. Consist Global States8 Distributed Computations to order events, define binary relation “→” to capture “cause-and-effect”: e → e’ if and only if e “causally precedes” e’ concurrent events: neither e → e’ nor e’ → e, write e || e’ distributed computation = partially ordered set defined by (H, →)

9. Consist Global States9 Distributed Computations e 2 1 → e 3 6 ; e 2 2 || e 3 6

10. Consist Global States10 Global States, Cuts and Runs σ i k  local state of process p i after event e i k Σ = (σ 1,…,σ n )  global state of distributed computation  n-tuple of local states cut C = h 1 c 1 U … U h n c n or (c 1, …, c n )  subset of global history H

11. Consist Global States11 Global States, Cuts and Runs (σ 1 c 1,…,σ n c n )  global state correspond to cut C (e 1 c 1,…,e n c n )  frontier of cut C  set of last events run  a total ordering R including all events in global history  consistent with each local history

12. Consist Global States12 Global States, Cuts and Runs cut C = (5,2,4); cut C’ = (3,2,6) a consistent run R = e 3 1 e 1 1 e 3 2 e 2 1 e 3 3 e 3 4 e 2 2 e 1 2 e 3 5 e 1 3 e 1 4 e 1 5 e 3 6 e 2 3 e 1 6

13. Consist Global States13 Consistency cut C is consistent if for all events e and e’  closed under the causal precedence relation consistent global state corresponds to a consistent cut run R is consistent if for all events, e → e’ implies e appears before e’ in R

14. Consist Global States14 Consistency run R = e 1 e 2 … results in a sequence of global states Σ 0 Σ 1 Σ 2 Σ i is obtained from Σ i-1 by some process executing event e i, or Σ i-1 leads to Σ i denote the transitive closure of the leads- to relation by ~> R Σ’ is reachable from Σ in run R iff Σ ~> R Σ’

15. Consist Global States15 Lattice of Global States lattice = set of all consistent global states, along with leads-to relation Σ k 1 …k n = shorthand for global state (σ 1 k 1,…,σ n k n ) k 1 + … + k n = level of lattice

16. Consist Global States16 Lattice of Global States path = sequence of global states of increasing level (downwards) each path corresponds to a consistent run a possible path: Σ 00 Σ 01 Σ 11 Σ 21 Σ 31 Σ 32 Σ 42 Σ 43 Σ 44 Σ 54 Σ 64 Σ 65

17. Consist Global States17 Observing Distributed Computations (reactive-architecture) processes notify monitor process p 0 whenever they execute an event monitor constructs observation as the sequence of events corresponding to the notification messages problem:  observation may be inconsistent due to variability in notification message delays

18. Consist Global States18 Observing Distributed Computations

19. Consist Global States19 Observing Distributed Computations any permutation of run R is a possible observation we need:  delivery rule at monitor process to restore message order we have First-In-First-Out (FIFO) delivery using sequence number for all source-destination pair p i, p j :  send i (m) → send i (m’) => deliver j (m) → deliver j (m’)

20. Consist Global States20 Delivery Rule 1 assume:  global real-time clock  message delays bound by δ process includes timestamp (real-time clock value) when notifying p 0 of local event e DR1: At time t, deliver all received messages with timestamps up to t – δ in increasing timestamp order

21. Consist Global States21 Delivery Rule 1 let RC(e) denotes value of global clock when e is executed real-time clock satisfies Clock Condition:  e → e’ => RC(e) < RC(e’) but logical clocks also satisfies clock condition…

22. Consist Global States22 Logical Clocks event orderings based on increasing clock values LC(e i ) denotes value of logical clock when e i is executed by p i each sent message m contains timestamp TS(m) update rules by p i at occurrence of e i :

23. Consist Global States23 Logical Clocks

24. Consist Global States24 Delivery Rule 2 replace real-time clock by logical clock need gap-detection property:  given events e, e’ where LC(e) < LC(e’), determine if some event e’’ exists such that LC(e) < LC(e’’) < LC(e’)  message is “stable” at p if no future messages with timestamps smaller than TS(m) can be received by p

25. Consist Global States25 Delivery Rule 2 with FIFO, when p 0 receives m from p i with timestamp TS(m), can be certain no other message m’ from p i with TS(m’) ≤ TS(m) message m at p 0 guaranteed stable when p 0 has received at least one message from all other processes with timestamps > TS(m) DR2: Deliver all received messages that are stable at p 0 in increasing timestamp order

26. Consist Global States26 Strong Clock Condition DR1, DR2 assume RC(e) e → e’ recall RC and LC guarantee clock condition: e → e’ => RC(e) < RC(e’) DR1, DR2 can unnecessarily delay delivery want timing mechanism TC that gives Strong Clock Condition:  e → e’ ≡ TC(e) < TC(e’)

27. Consist Global States27 Timing Mechanism 1 - Causal Histories causal history as “clock” value  set of all events that causally precede event e:  smallest consistent cut that includes e  projection of θ(e) on process p i : θ i (e) = θ(e) ∩ h i

28. Consist Global States28 Timing Mechanism 1 - Causal Histories

29. Consist Global States29 Timing Mechanism 1 - Causal Histories To maintain causal histories:  θ initially empty  if e i is an internal or send event θ(e i ) = {e i } U θ(previous local event of p i )  if e i = receive of message m by p i from p j θ(e i ) = {e i } U θ(previous local event of p i ) U θ(corresponding send event at p j )

30. Consist Global States30 Timing Mechanism 1 - Causal Histories new event e 1 5 new event e 2 3 new send event: new receive event:

31. Consist Global States31 Timing Mechanism 1 - Causal Histories can interpret clock comparison as set inclusion:  e → e’ ≡ θ(e)  θ(e’) (why not set membership, [e → e’ ≡ e  θ(e’)]?) unfortunately, causal histories grow too rapidly

32. Consist Global States32 Timing Mechanism 2 - Vector Clocks note:  projection θ i (e) = h i k for some unique k  e i r  θ i (e) for all r < k  can use single number k to represent θ i (e)  θ(e) = θ 1 (e) U … U θ n (e) represent entire causal history by n-dimensional vector clock VC(e), where for all 1 ≤ i ≤ n  VC(e)[i] = k, if and only if θ i (e) = h i k

33. Consist Global States33 Timing Mechanism 2 - Vector Clocks

34. Consist Global States34 Timing Mechanism 2 - Vector Clocks To maintain vector clock:  each process p i initializes VC to contain all zeros  update rules by p i at occurrence of e i : VC(e i )[i] ≡ number of events p i has executed up to and including e i VC(e i )[j] ≡ number of events of p j that causally precede event e i of p i

35. Consist Global States35 Timing Mechanism 2 - Vector Clocks new send event: new receive event: causal historiesvector clocks

36. Consist Global States36 Vector Clock Comparison Define “less than” relation:  V < V’ ≡ (V ≠ V’)  (  1 ≤ k ≤ n: V[k] ≤ V’[k])

37. Consist Global States37 Properties of Vector Clocks 1. Strong Clock Condition: e → e’ ≡ VC(e) < VC(e’) 2. Simple Strong Clock Condition: given event e i of p i and event e j of p j, i ≠ j e i → e j ≡ VC(e i )[i] ≤ VC(e j )[i]

38. Consist Global States38 Properties of Vector Clocks 3. Test for Concurrency: given event e i of p i and event e j of p j e i || e j ≡ (VC(e i )[i] > VC(e j )[i])  (VC(e j )[j] > VC(e i )[j]) 4. Pairwise Inconsistent: given event e i of p i and e j of p j, i ≠ j if e i, e j cannot belong to the frontier of the same consistent cut (VC(e i )[i] < VC(e j )[i])  (VC(e j )[j] < VC(e i )[j]) (concurrent)

39. Consist Global States39 Properties of Vector Clocks 5. Consistent Cut: frontier contains no pairwise inconsistent events VC(e i c i )[i]  VC(e j c j )[i],  1 ≤ i, j ≤ n 6. Counting # of events causally precede e i : #(e i ) = (Σ j=1.. n VC(e i )[j]) – 1 # events = 4+1+3-1 = 7

40. Consist Global States40 Properties of Vector Clocks 7. Weak Gap-Detection: given event e i of p i and e j of p j, if VC(e i )[k] < VC(e j )[k] for some k ≠ j, there exists event e k such that  (e k → e i )  (e k → e j )

41. Consist Global States41 Causal Delivery and Vector Clocks assume processes increment local component of VC only for events notified to monitor p 0 p 0 maintains set M for messages received but not yet delivered suppose we have:  message m from p j  m’ = last message delivered from process p k, k ≠ j

42. Consist Global States42 Causal Delivery and Vector Clocks To deliver m, p 0 must verify: 1. no earlier message from p j is undelivered (i.e. TS(m)[j] – 1 messages have been delivered from p j ) 2. no undelivered message m’’ from p k s.t. send k (m’) →send k (m’’)→send j (m),  k ≠ j (i.e. whether TS(m’)[k]  TS(m)[k] for all k)

43. Consist Global States43 Causal Delivery and Vector Clocks p 0 maintains array D[1…n] where D[i] = TS(m i )[i], m i being last message delivered from p i e.g. on right, delivery of m is delayed until m’’ is received and delivered

44. Consist Global States44 Delivery Rule 3 Causal Delivery:  for all messages m, m’, sending processes p i, p j and destination process p k send i (m) → send j (m’) => deliver k (m) → deliver k (m’) DR3 (Causal Delivery): Deliver message m from process p j as soon as D[j] = TS(m)[j] – 1, and D[k]  TS(m)[k],  k ≠ j p 0 set D[j] to TS(m)[j] after delivery of m

45. Consist Global States45 Causal Delivery and Hidden Channels should apply to closed systems incorrect conclusion with hidden channels (communication channel external to the system)

46. Consist Global States46 Active Monitoring - Distributed Snapshots monitor p 0 requests states of other processes and combine into global state assume channels implement FIFO delivery channel state χ i,j for channel p i to p j : messages sent by p i not yet received by p j

47. Consist Global States47 Distributed Snapshots notations: IN i = set of processes having direct channels to p i OUT i = set of processes to which p i has a channel for each execution of the snapshot protocol, process p i record its local state σ i and the states of its incoming channels (χ j,i for all p j  IN i )

48. Consist Global States48 Distributed Snapshots Snapshot Protocol (Chandy-Lamport) 1. p 0 starts the protocol by sending itself a “take snapshot” message 2. when receiving the “take snapshot” message for the first time from process p f  p i records local state σ i and relays the “take snapshot” message along all outgoing channels  channel state χ f,i is set to empty  p i starts recording messages on other incoming channels

49. Consist Global States49 Distributed Snapshots Snapshot Protocol (Chandy-Lamport) 3. when receiving the “take snapshot” message beyond the first time from process p s :  p i stops recording messages along channel from p s  channel state χ s,i are messages that have been recorded

50. Consist Global States50 Distributed Snapshots dash arrows indicate “take snapshot” messages constructed global state: Σ 23 ; χ 1,2 empty; χ 2,1 = {m} p 1 done p 2 done

51. Consist Global States51 Properties of Snapshots Let Σ s = global state constructed Σ a = global state when protocol initiated Σ f = global state when protocol terminated Σ s is guaranteed to be consistent actual run that the system followed may not pass through Σ s but  a run R such that Σ a ~> R Σ s ~> R Σ f

52. Consist Global States52 Properties of Snapshots Σ a = Σ 21 Σ f = Σ 55 r does not pass through Σ s ( = Σ 23 )

53. Consist Global States53 Properties of Snapshots but Σ 21 ~> Σ 23 ~> Σ 55

54. Consist Global States54 Properties of Global Predicates Now we have two methods for global predicate evaluation:  monitor passively observing runs  monitor actively constructing snapshots utility of either approach depends (in part) on properties of the predicate

55. Consist Global States55 Stable Predicates communication delays => Σ s can only reflect some past state of the system stable predicate: once become true, remain true e.g. deadlock, termination, loss of all tokens, unreachable storage if Φ is stable, then (Φ is true in Σ s ) => (Φ is true in Σ f ) and (Φ is false in Σ s ) => (Φ is false in Σ a )

56. Consist Global States56 Stable Predicates deadlock detection through snapshots (p.29, 30)

57. Consist Global States57 Stable Predicates deadlock detection using reactive protocol (p.31, 32)

58. Consist Global States58 Nonstable Predicates e.g. debugging, checking if queue lengths exceed some thresholds Two problems: 1. condition may not persist long enough for it to be true when the predicate is evaluated 2. if a predicate Φ is found true, do not know whether Φ ever held during the actual run

59. Consist Global States59 Nonstable Predicates e.g. monitoring condition (x = y)  7 states where (x = y) holds  but no longer hold after state Σ 54 e.g. (y – x) = 2  condition hold only in Σ 31 and Σ 41  monitor might detect (y - x) = 2 even if actual run never goes through Σ 31 or Σ 41

60. Consist Global States60 Nonstable Predicates very little value to detect nonstable predicate

61. Consist Global States61 Nonstable Predicates With observations, can extend predicates: Possibly(Φ): There exist a consistent observation O of the computation such that Φ holds in a global state of O Definitely(Φ): For every consistent observation O of the computation, there exists a global state of O in which Φ holds e.g. Possibly((y – x) = 2), Definitely(x = y)

62. Consist Global States62 Nonstable Predicates use of extended predicate in debugging: if Φ = some erroneous state, then Possibly(Φ) indicates a bug, even if it is not observed during an actual run if predicate Φ is stable, then Possibly(Φ) ≡ Definitely(Φ)

63. Consist Global States63 Detecting Possibly and Definitely Φ detection based on the lattice of consistent global states If any global state in the lattice satisfies Φ, then Possibly(Φ) holds Definitely(Φ) requires all possible runs to pass through a global state that satisfies Φ

64. Consist Global States64 Detecting Possibly and Definitely Φ Possibly((y – x) = 2) Definitely(y = x) (why?)

65. Consist Global States65 Detecting Possibly and Definitely Φ set of global state current with progressively increasing levels any member of current satisfies Φ => Possibly(Φ) true

66. Consist Global States66 Detecting Possibly and Definitely Φ iteratively construct set of global states of level l without passing through a state that satisfies Φ set empty => Definitely(Φ) true set contains the final state =>  Definitely(Φ) true

67. Consist Global States67 Conclusions many distributed system problems require recognizing certain global conditions two approaches to constructing global states:  reactive-architecture based  snapshot based timing mechanism that captures causal precedence relation applying to distributed deadlock detection and debugging solutions can be adapted to deal with nonstable predicates, multiple observations and failures