1. Distributed Systems Concepts and Design Chapter 11: Time and Global States Steve Wallis, Raymond Ho, Bruce Hammer

2. 2 2 11.1:Introduction  Time and Global States Time is a quantity we need to measure accurately Synchronization of Clocks Development Of Algorithms Relative Order Of Events Notion of Physical Time – this results from the need to know the order in which a pair of events or whether they occur simultaneously

3. Steve Wallis, Raymond Ho, Bruce Hammer3 Bruce Hammer, Steve Wallis, Raymond Ho3 11.2: Clocks, events and Process States  Model of interaction between the processes within a distributed system History of process

4. Steve Wallis, Raymond Ho, Bruce Hammer4 Bruce Hammer, Steve Wallis, Raymond Ho4 11.2: Clocks, events and Process States  Clocks Clocks are electronic devices Operating System reads the node’s hardware clock, scales it and adds an offset to produce a software clock In general, clock is not completely accurate

5. Steve Wallis, Raymond Ho, Bruce Hammer5 Bruce Hammer, Steve Wallis, Raymond Ho5 11.2: Clocks, events and Process States  Clock skew and clock drift Computer clocks tend not to be in perfect agreement Clock skew - Instantaneous difference between the readings of any two clocks is called their skew

6. Steve Wallis, Raymond Ho, Bruce Hammer6 Bruce Hammer, Steve Wallis, Raymond Ho6 11.2: Clocks, events and Process States Clock drift – computer crystal-based clocks count time at different rates, and so diverge

7. Steve Wallis, Raymond Ho, Bruce Hammer7 Bruce Hammer, Steve Wallis, Raymond Ho7 11.2: Clocks, events and Process States  Coordinated Universal Time Synchronize computer clocks to external source of highly accurate time Such as atomic clock with drift rate is about one part in 10 13 Output from these atomic clocks known as International Atomic Time Seconds and years and other time units are rooted in astronomical time

8. Steve Wallis, Raymond Ho, Bruce Hammer8 Bruce Hammer, Steve Wallis, Raymond Ho8 11.2: Clocks, events and Process States Coordinated Universal Time (UTC)  International standard for timekeeping  Based on atomic time  UTC signals are synchronized and broadcast regularly from land-based radio stations and satellites to many parts of the world

9. Steve Wallis, Raymond Ho, Bruce Hammer9 Bruce Hammer, Steve Wallis, Raymond Ho9 11.3: Synchronizing physical clocks External synchronization - For process accountancy purposes, necessary to synchronize the processes’ clocks with an authoritative, external source of time

10. Steve Wallis, Raymond Ho, Bruce Hammer10 Bruce Hammer, Steve Wallis, Raymond Ho10 11.3: Synchronizing physical clocks Internal synchronization – measure of the interval between two events occurring at different computers by appealing to their local clocks

11. Steve Wallis, Raymond Ho, Bruce Hammer11 Bruce Hammer, Steve Wallis, Raymond Ho11 11.3: Synchronizing physical clocks Clocks not necessary synchronized internally and externally Common to define a hardware clock H to be correct if it drift rate falls with a known bound p>0, mean the error in measuring the interval between real times t and t’ (t’ >t) is bound Monotonicity – is the condition that a clock C only ever advances

12. Steve Wallis, Raymond Ho, Bruce Hammer12 Bruce Hammer, Steve Wallis, Raymond Ho12 11.3: Synchronizing physical clocks Faulty – A clock that does not keep to whatever correctness condition apply

13. Steve Wallis, Raymond Ho, Bruce Hammer13 Bruce Hammer, Steve Wallis, Raymond Ho13 11.3: Synchronizing physical clocks  Synchronization in a synchronous system Synchronous System  Has minimum transmission time min  Also has maximum transmission time max  So transmission time be u = (max - min)  Optimum bound on clock skew when synchronizing N clocks is u(1 – 1/N)

14. Steve Wallis, Raymond Ho, Bruce Hammer14 Bruce Hammer, Steve Wallis, Raymond Ho14 11.3: Synchronizing physical clocks  Asynchronous System Found in most distributed systems Message delays are not bounded in their effect No upper bound max on message transmission delays T trans = min + x, where x>0, x is not known in a particular case, although a distribution of values may be measurable for a particular installation

15. Steve Wallis, Raymond Ho, Bruce Hammer15 Bruce Hammer, Steve Wallis, Raymond Ho15 11.3: Synchronizing physical clocks  Cristian’s method for synchronizing clocks Probabilistic: Algorithm achieves synchronization only if he observed round-trip times between client and server are sufficiently short compared with the required accuracy

16. Steve Wallis, Raymond Ho, Bruce Hammer16 Bruce Hammer, Steve Wallis, Raymond Ho16 11.3: Synchronizing physical clocks  Discussion of Cristian’s algorithm Cristian method suffers from the problem associated with all services implemented by a single server but the server might fail and thus render synchronization impossible temporarily Cristian suggested that time should be provided by a group of synchronized time servers and use only the first reply obtained

17. Steve Wallis, Raymond Ho, Bruce Hammer17 Bruce Hammer, Steve Wallis, Raymond Ho17 11.3: Synchronizing physical clocks  The Berkeley algorithm Algorithm from Gusella and Zatti for internal synchronization Developed for collections of computers running Berkeley UNIX A coordinator computer to act as the master This computer periodically polls the other computers whose clocks are to be synchronized, called slaves

18. Steve Wallis, Raymond Ho, Bruce Hammer18 Bruce Hammer, Steve Wallis, Raymond Ho18 11.3: Synchronizing physical clocks Master estimates the local clock times by observing the round-trip times sent from each slave computer Master sends the amount by which each individual slaves clock requires adjustment The algorithm eliminates readings from faulty clocks Should the master fail, then another can be elected to take over and function as it predecessor

19. Steve Wallis, Raymond Ho, Bruce Hammer19 Bruce Hammer, Steve Wallis, Raymond Ho19 11.3: Synchronizing physical clocks  The Network Time Protocol (NTP) NTP [Mills 1995] defines an architecture for a time service and a protocol to distribute time information over the Internet Provide a service enabling clients across the Internet to be synchronized accurately to UTC. NTP employs statistical techniques for the filtering of timing data and discriminates between the quality of timing data from servers

20. Steve Wallis, Raymond Ho, Bruce Hammer20 Bruce Hammer, Steve Wallis, Raymond Ho20 11.3: Synchronizing physical clocks To provide a reliable service that can survive lengthy losses of connectivity To enable clients to resynchronize sufficiently frequently to offset the rates of drift found in most computers To provide protection against interference with the time service, whether malicious or accidetal

21. Steve Wallis, Raymond Ho, Bruce Hammer21 Bruce Hammer, Steve Wallis, Raymond Ho21 11.3: Synchronizing physical clocks NTP service is provided by a network of servers located across the Internet Primary servers are connected directly to a time source such as a radio clock receiving UTC Secondary servers are synchronized wit primary servers The servers are connected in a logical hierarchy called a synchronization subnet

22. Steve Wallis, Raymond Ho, Bruce Hammer22 Bruce Hammer, Steve Wallis, Raymond Ho22 11.3: Synchronizing physical clocks

23. Steve Wallis, Raymond Ho, Bruce Hammer23 Bruce Hammer, Steve Wallis, Raymond Ho23 11.3: Synchronizing physical clocks The subnet levels are called strata Primary servers occupy stratum 1 at the root Secondary servers are stratum 2, and so on The lowest-level (leaf) serves execute in users’ workstations The synchronization subnet can reconfigure as servers become unreachable or failures occur

24. Steve Wallis, Raymond Ho, Bruce Hammer24 Bruce Hammer, Steve Wallis, Raymond Ho24 11.3: Synchronizing physical clocks NTP servers synchronize with one another in one of three modes:  Multicast mode – intended for use on a high-speed LAN, can achieve only relatively low accuracies  Procedure-call mode – one server accepts requests from other computers and reply with its timestamp, it is suitable for higher accuracies than multicast  Symmetric mode – intended for use by the servers that supply time information in LANs and by the higher levels (lower strata) of synchronization subnet to achieve highest accuracies

25. Steve Wallis, Raymond Ho, Bruce Hammer25 Bruce Hammer, Steve Wallis, Raymond Ho25 11.3: Synchronizing physical clocks In all modes, messages are delivers unreliably, using the standard UDP Internet transport protocol In procedure-call and symmetric modes, processes exchange pairs of messages Each message bears timestamps of recent message events

26. Steve Wallis, Raymond Ho, Bruce Hammer26 Bruce Hammer, Steve Wallis, Raymond Ho26 11.3: Synchronizing physical clocks

27. Steve Wallis, Raymond Ho, Bruce Hammer27 Bruce Hammer, Steve Wallis, Raymond Ho27 11.3: Synchronizing physical clocks NTP calculates an offset for each pair of messages sent between two servers Which is an estimate of the actual offset between the two clocks, a delay, which is the total transmission time for the two messages If the true offset of the clock at B relative to that at A is o, and if the actual transmission times for m and m’ are t ad t’ respectively then we have:

28. Steve Wallis, Raymond Ho, Bruce Hammer28 Bruce Hammer, Steve Wallis, Raymond Ho28 11.3: Synchronizing physical clocks

29. Steve Wallis, Raymond Ho, Bruce Hammer29 Bruce Hammer, Steve Wallis, Raymond Ho29 11.3: Synchronizing physical clocks In general, NTP server engages in message exchanges with several of its peers with data filtering NTP applies a peer-selection algorithm to examines the values obtained from exchanges with each of several peers. The output may cause a server to change the peer that it primarily uses for synchronization

30. Steve Wallis, Raymond Ho, Bruce Hammer30 Bruce Hammer, Steve Wallis, Raymond Ho30 11.3: Synchronizing physical clocks NTP employs a phase lock loop model, which modifies the local clock’s update frequency in accordance with observations of its drift rate

31. Steve Wallis, Raymond Ho, Bruce Hammer31 Steve Wallis, Raymond Ho, Bruce Hammer31 11.4: Logical Time and Logical Clocks  Ordering Of Events For a single process, events are ordered uniquely by times on the local clock Clocks cannot be synchronized perfectly across a distributed system, so we cannot use general time to find out the order of a pair of events occurring within it. In the ordering of events, there are 2 obvious points If 2 events occurred at the same process p i (i=1,2,..,N) then they occurred in the order p i observes them. This is denoted by  i

32. Steve Wallis, Raymond Ho, Bruce Hammer32 Steve Wallis, Raymond Ho, Bruce Hammer32 11.4: Logical Time and Logical Clocks  Ordering Of Events Whenever a message is sent between processes, the sending event always occurs before the receiving event.  Happened Before ■ This partial ordering was named the ‘happened before’ relation by Lamport ■ Also known as casual ordering or potential casual ordering ■ The ‘happened before’ relation is denoted by 

33. Steve Wallis, Raymond Ho, Bruce Hammer33 Steve Wallis, Raymond Ho, Bruce Hammer33 11.4: Logical Time and Logical Clocks  Happened Before (contd) ■ Definitions of HB ■ For any message m, send(m)  receive(m), where send(m) is the sending event and receive(m) the receiving event ■ If e, e` and e`` are 3 events where e  e`, and e`  e`` then e  e`` ■ HB does not imply actual causality, If event e 1 happens before event e 2 it does not imply e 1 caused e 2 ■ If 2 events a and b are not related i.e they occur in different processes and there is no chain of messages between them, so neither a  b or b  a is true, then they are concurrent and denoted as a||b

34. Steve Wallis, Raymond Ho, Bruce Hammer34 Steve Wallis, Raymond Ho, Bruce Hammer34 11.4: Logical Time and Logical Clocks  Happened Before (contd)

35. Steve Wallis, Raymond Ho, Bruce Hammer35 Steve Wallis, Raymond Ho, Bruce Hammer35 11.4: Logical Time and Logical Clocks  Logical Clocks Happened Before ordering can be captured numerically by a mechanism called a logical clock Invented by Lamport A software counter with no relationship to any physical clock Each process p i keeps its own Logical Clock L i This is used to apply the Lamport Timestamp to the event

36. Steve Wallis, Raymond Ho, Bruce Hammer36 Steve Wallis, Raymond Ho, Bruce Hammer36 11.4: Logical Time and Logical Clocks  Logical Clocks (contd) The Lamport Timestamp is denoted by L i (e) To capture the happened before relation , processes update their Logical Clocks and transmit the values as follows: LC1: L i is incremented by 1 before each event is issued at process p i : L i = L i + 1 LC2: when process p i sends message m, it appends to the message the value t = L i On receiving message (m,t) process p j computes

37. Steve Wallis, Raymond Ho, Bruce Hammer37 Steve Wallis, Raymond Ho, Bruce Hammer37 11.4: Logical Time and Logical Clocks  Logical Clocks (contd) L j = max (L j, t) and applies LC1 before time stamping the event receive(m). For the happened before relation it then follows for 2 events a, b that a  b => L(a) < L(b) Note : converse if not true : if L(b) > L(a) then it does not follow a  b (they may be concurrent e.g a || b)

38. Steve Wallis, Raymond Ho, Bruce Hammer38 Steve Wallis, Raymond Ho, Bruce Hammer38 11.4: Logical Time and Logical Clocks  Logical Clocks ■ Lamport Timestamps

39. Steve Wallis, Raymond Ho, Bruce Hammer39 Steve Wallis, Raymond Ho, Bruce Hammer39 11.4: Logical Time and Logical Clocks  Totally Ordered Logical Clocks Some pairs of distinct events, generated by different processes, have numerically equivalent Lamport Timestamps A total order of events – one for which all pairs of distinct events are ordered – can be created by taking into account the identifiers of the processes at which events occur. If e is the event at process p i, with local Timestamp T i, then global logical timestamp is denoted as (T i, i)

40. Steve Wallis, Raymond Ho, Bruce Hammer40 Steve Wallis, Raymond Ho, Bruce Hammer40 11.4: Logical Time and Logical Clocks  Vector Clocks ■ Vector Clocks were developed by Mattern and Fidge to overcome the shortcoming of Lamport clocks ■ That is if L(e`) > L(e) we cannot conclude that e  e` ■ A Vector Clocks for a system of N process is an array of N integers ■ Each process p i keeps its own Vector Clock V i, and like Lamport timestamps processes piggy back Vector timestamps to messages they send to other processes.

41. Steve Wallis, Raymond Ho, Bruce Hammer41 Steve Wallis, Raymond Ho, Bruce Hammer41 11.4: Logical Time and Logical Clocks  Vector Clocks (contd) ■ Vector Clocks are updated as follows : ■ VC1 V i [j]=0 for i, j=1,2,…,N ■ VC2: V i is incremented by 1 before each event is issued at process p i : V i [i] = V i [i] +1 ■ VC3: process p i includes the value t= V i in every message it sends ■ When a process p i receives a Vector timestamp in a message it sets V i [j]=max (V i [j],t[j]) for j=1,2,…,N

42. Steve Wallis, Raymond Ho, Bruce Hammer42 Steve Wallis, Raymond Ho, Bruce Hammer42 11.4: Logical Time and Logical Clocks  Vector Clocks

43. Steve Wallis, Raymond Ho, Bruce Hammer43 Steve Wallis, Raymond Ho, Bruce Hammer43 11.4: Logical Time and Logical Clocks  Vector Clocks (contd) ■ Comparing Vector Timestamps :  V = V′ if V[j]=V′ [j] for j=1,2,…,N  V ≤ V′ if V[j] ≤ V′[j] for j=1,2,…,N  V < V′ if V ≤ V′ ⋀ V ≠ V′ ■ As in Logical Clocks, it follows that :  For 2 events e, e′ that e  e′ => V(e) < V(e′ ) ■ But now the converse IS true  If V(e) < V(e′ ) then e  e′

44. Steve Wallis, Raymond Ho, Bruce Hammer44 11.5: Global States  Using Vector Timestamps Reconstruct the State of a system thru logical sequencing of events across multiple processes Find out the state of a process Whether a particular property of the system or a process is true or has been true

45. Steve Wallis, Raymond Ho, Bruce Hammer45 11.5: Global States Examples  Distributed Garbage Collection – Are there any references to an object anywhere in the distributed system? If no, reuse memory  Deadlock Detection Are two processes just waiting for each other to send?  Distributed Termination Detection Has a process terminated or is it merely waiting for a response? Similar to Deadlock

46. Steve Wallis, Raymond Ho, Bruce Hammer46 Figure 11.9 Cuts m 1 m 2 p 1 p 2 Physical time e 1 0 Consistent cut Inconsistent cut e 1 1 e 1 2 e 1 3 e 2 0 e 2 1 e 2 2

47. Steve Wallis, Raymond Ho, Bruce Hammer47 11.5: Global States  Slicing a distributed system to determine the system’s state or a process’s state  The vector times, as noted above, sequence the system’s messages and processes  Cut 1 is inconsistent because the receipt of the message by p 2 is included but the sending by p 1 is not

48. Steve Wallis, Raymond Ho, Bruce Hammer48 11.5: Global States  Linearization or Consistent Run An ordering of all the events in a cut or global history that is consistent with happened-before Determine what state is reachable from sequence of events  Global State Predicate Function that maps from the set of global states of processing in the system to true or false  Termination, deadlock, garbage

49. Steve Wallis, Raymond Ho, Bruce Hammer49 11.5: Global States  Chandy and Lamport’s ‘snapshot’ algorithm Goal to record a snapshot of channels and processes at any point in time Uses a designated collector process Each process records its state and its messages Uses marker messages – initiates the snapshot  Marker receiving rule – obligates a process that has not recorded its state to do so  Marker sending rule – obligates a process to send a marker after they have recorded their state and before they send messages

50. Steve Wallis, Raymond Ho, Bruce Hammer50 11.5: Global States (contd) Reachability between states in the snapshot algorithm  S init – the global state before the first process recorded its state  S snap – the constantly-changing global state  S final – the global state when the snapshot terminates  Upper linearization is Sys; Lower linearization is Sys’  Establishes the reachability relationship S init S final S snap actual execution e 0,e 1,... recording begins ends pre-snap: e ' 0,e ' 1,...e ' R-1 post-snap: e ' R,e ' R+1,...

51. Steve Wallis, Raymond Ho, Bruce Hammer51 11.6: Distributed Debugging  Can we determine whether the system was stable or transitory in an execution?  Practical example – were all the pipes in the factory open at one time?  Uses the snapshot algorithm  Uses a central monitor, which lies outside the system, (Marzullo and Neiger)

52. Steve Wallis, Raymond Ho, Bruce Hammer52 11.6: Distributed Debugging   a global state predicate  Possibly  – there is a consistent global state S through which a linearization of H (history) passes such that  (S  is true  Definitely  all linearizations of H passes such that  (S) is true  Evaluating possibly  entails a search through all consistent global states derived from the observed execution

53. Steve Wallis, Raymond Ho, Bruce Hammer53 11.6: Distributed Debugging  Processes may react to what condition is being tested for and respond accordingly  The monitor must assemble consistent global states against which it evaluates   The monitor uses the vector clock values of each message to determine the consistent global state.

54. Steve Wallis, Raymond Ho, Bruce Hammer54 11.6: Distributed Debugging  For possibly , the monitor process starts at the initial state and steps through all consistent states reachable from that point, evaluating  at each stage. When  is true, it stops  For definitely  The monitor process must attempt to find a set of states through which all linearizations must pass, and at each of which  is true

55. Steve Wallis, Raymond Ho, Bruce Hammer55 Figure 11.16 Algorithms to evaluate possibly  and definitely 

56. Steve Wallis, Raymond Ho, Bruce Hammer56 11.6: Distributed Debugging  Cost of distributed debugging Development of code to respond to the snapshot request, with different variables Processing of the request and messaging cost to the monitor Cost of comparing all the states of each of the observed processes Space of storing all the values for comparison

57. Steve Wallis, Raymond Ho, Bruce Hammer57 11.6: Distributed Debugging  Debugging in a synchronous system An asynchronous system requires the vector timestamps to determine a consistent global state A synchronous system can send vector timestamps and physical timestamps to the monitor.  The monitor can then determine more accurately whether events occurred simultaneously, crossed paths, et al.

58. Steve Wallis, Raymond Ho, Bruce Hammer58 Steve Wallis, Raymond Ho, Bruce Hammer58 11.7: Summary  Items Covered The importance of accurate timekeeping for distributed systems Algorithms for synchronizing clocks The ‘happened before’ relation Logical Clocks and Vector Clocks Concepts of events, local and global histories, local and global states, consistent states Algorithms of Chandy/Lamport and Marzullo/Neiger

59. Steve Wallis, Raymond Ho, Bruce Hammer59 Steve Wallis, Raymond Ho, Bruce Hammer59 11.7: Summary ‘Time Is An Illusion – Lunchtime Especially So’ (Ref : Adams D – Hitch Hikers Guide To the Galaxy 1977)