Global States

Topics Usefulness of global state Difficulties in determining snapshot Determining global state Examples

Readings Van Steen and Tanenbaum: 5.3 Coulouris: 10.5 “Distributed Snapshots: Determining Global States of Distributed Systems”, K. Mani Chandy and Leslie Lamport, ACM Transactions on Computer Systems, vol 3, no 1, February 1985.

Motivating Example Assume that we have processes interacting in a client-server mode. Client makes request to server Waits for response While waiting for response, client simply blocks; does not answer to requests from other nodes. B This is a deadlock A C

Motivating Example Assume that you have a centralized server. It queries each node Each node responds with a list of requests that are pending (requests for which a response has not been sent). Centralized server can then build a “waits-for” graph. Cycle in graph implies deadlock This is an example where being able to determine the global state is useful.

Other Motivating Examples Distributed Debugging Distributed Garbage Collection An object is considered to be garbage if there are no longer any references to it anywhere in the distributed system. p 2 1 message garbage object object reference

Global State Occasionally it is useful to know the global state of a distributed system. The global state of a distributed system consists of the local state of each process, together with the messages that are currently in transit, that is, that have been sent but not delivered. Exactly what a local state of a process is, depends on our interest. In a distributed database system, the local state may consist of only those records that form part of the database and excludes temporary records used in computation.

Global State in a Single Process Application Assume that we have the following code: x = y = z =0; x = z+1; y = x; z = y The state can be represented as the values of x,y,z. The sequence of states for the execution of the above program is the following: 000, 100, 110, 111 Deterministic Computation At any point in computation there is at most one event that can happen next. There is only one sequence of states for the same input.

Global State for a Multiple Process Application Say in process P1 you have a code segment as follows: 1.1 x = 5; 1.2 y = 10*x; 1.3 send(y,P2); 1.4 z = 8; Say in process P2 you have a code segment as follows: 2.1 a=8; 2.2 b=10*a; 2.3 rcv(c,P1); 2.4 b = b+c;

Global State for a Multiple Process Application Non-Deterministic Computation At any point in computation there can be more than one event that can happen next There is more than one possible sequence of states for the same input. Assume that a state is represented as (x,y,z,a,b,c). Observation: In this example, regardless of the sequence, eventually all sequences reach these global states (5,50,0,8,80,0), (5,50,0,8,80,50)

Global State for Multiple Process Application 0,0,0,0,0,0 0,0,0,8,0,0 5,0,0,0,0,0 5,0,0,8,0,0 5,0,0,8,0,0 5,0,0,8,80,0 5,50,0,8,0,0 5,50,0,8,80,0 5,50,0,8,80,50 5,50,8,8,80,50 5,50,0,8,80,130 5,50,8,8,80,130 5,50,8,8,80,130

A Non-Deterministic Computation Each of the paths represent a possible sequence of events that could have occurred during execution. This assumes that the same inputs were applied. Different paths occur for different reasons that include machines that have different loads at different runs, the network load at different runs, etc; The different paths correspond to a different sequence of global states that the distributed program goes through.

Viewing Global States A process in a distributed system has only a limited view of the system, which consists of its local state and the messages it has sent or received. Given this, is it possible for processes in the system to determine a global system state? Instantaneous recording not possible No global clock: Distributed recording of local states cannot be synchronized based on time. Random network delays; What messages are in transit?

Viewing Global States The main difficulty is that global state computation requires computation and communication, and by the time such effort is completed, the global state has changed. A snapshot of the system is a single configuration of the system. It is not necessarily a global state that has actually occurred, but one that could have occurred during the computation.

A Naive Snapshot Algorithm Processes record their state at any arbitrary point A designated process collects these states Simple, but not correct.

Example of Naive Snapshot Algorithm in Use P records its state P Q X = 5

Example of Naive Snapshot Algorithm in Use Let m be a message to update a variable Y in Q by incrementing it by the value of X. P Q m

Example of Naive Snapshot Algorithm in Use Q records its state after receiving m but before applying the operation. P Q m

Example of Naive Snapshot Algorithm in Use Process P records its state before it actually sends m Process Q records its state after it has received m. This is bad. The result is a global state that has recorded the receive event but no send event.

Distributed Snapshot Represents a state in which the distributed system might have been in. A distributed snapshot should reflect a consistent state. If we have recorded that a process P has received a message from a process Q, then we should have also recorded that process Q had actually send that message. Otherwise, a snapshot will contain the recording of messages that have been received but never sent. The reverse condition (Q has sent a message that P has not received) is allowed.

Distributed Snapshot The notion of a global state can be graphically represented by what is called a cut (example on the next slide). A cut represents the last event that has been recorded for each process.

Consistent State A consistent cut An inconsistent cut

Assumptions for Algorithm The algorithm to determine global states records process states locally and the states are collected by a designated server. No failures in channels and processes – exactly once delivery. Unidirectional channel and FIFO ordered message delivery. Always a path between two processes. Global snapshot initiation at any process at any time. No process activity halt during snapshot.

Features of Algorithm Does not promise us to give us exactly what is there i.e., an actual state However does give us a consistent state

Algorithm P starts by recording its own local state P subsequently sends a marker along each of its outgoing channels When Q receives a marker through channel C, its action depends on whether it had already recorded its local state: Not yet recorded: It records its local state and sends the marker along each of its outgoing channels Already recorded: The marker on C indicates that the channel’s state should be recorded: all messages received before this marker and since the last time Q records its own state. Q is finished when it has received a marker along each of its incoming channels.

Algorithm Organization of a process and channels for a distributed snapshot

Algorithm Process Q receives a marker for the first time and records its local state Q records all incoming messages Q receives a marker for its incoming channel and finishes recording the state of the incoming channel

Algorithm How do we know when we are done? If a process Q receives the marker requesting a snapshot for the first time, it considers the process that sent the marker as its predecessor When Q completes its part of the snapshot, it sends its predecessor a DONE message. By recursion, when the initiator of the distributed snapshot has received a DONE message from all its successors, it knows that the snapshot has been completely taken.

Example Assume two process p1 and p2 connected by two unidirectional channels, c1 and c2. The two processes trade in widgets. Process p1 sends orders for widgets over c2 to p2, enclosing payment at the rate of $10 per widget. Process p2 sends widgets along channel c1 to p1. The processes have the initial states shown in the figure on the next slide. It is assumed that p2 has already received an order for 5 widgets, which it will shortly dispatch to p1.

Example This is the initial state when the algorithm was initiated; Before initiation, the process p2 had received an order for 5 widgets; Thus account has $50

Example Upon request for global snapshot, process p1 records its state in the actual global state, S0, when p1’s state is <$1000,0>. 1. Actual Global State S0 <$1000, 0> p c (empty) p <$50, 2000> 1 2 2 c (empty) 1

Example Process p1 then emits a marker message over its outgoing channel c2 before it sends the next application-level message: (Order 10, $100) over channel c2. The system enters actual global state S1. 2. Actual Global state S1 <$900, 0> p c (Order 10, $100), M p <$50, 2000> 1 2 2 c (empty) 1

Example Before p2 receives the marker, it emits an application message (5 widgets) over c1 in response to p1’s previous order, yielding a new actual global state S2. 3. Actual Global State S2 p 1 2 (Order 10, $100), M <$900, 0> <$50, 1995> (five widgets) c

Example Now process p1 receives p2’s message (5 widgets) and p2 receives the marker. p2 records its state as <$50,1995> and that of channel c2 as the empty sequence. p2 sends a marker message over c1. When process p1 receives p2’s marker message, it records the state of c1 which is the single message (5 widgets) that it received after it first recorded its state.

Example The final recorded state is p1: <$1000,0>; p2:<$50,1995>;c1:<(five widgets)>; c2:<> 4. Actual Global state S3 <$900, 5> p c (Order 10, $100) p <$50, 1995> 1 2 2 c (empty) 1

Example The sequence of actual global states was the following: S0: p1: <$1000,0>; p2:<$50,2000>;c1:<>; c2:<> S1: p1: <$900,0>; p2:<$50,2000>; c1:<>; c2:<(order 10, $100)>; S2: p1: <$900,0>; p2:<$50,1995>; c1:<five widgets>; c2:<(order 10, $100)>; S3: p1: <$900,5>; p2:<$50,1995>; c1:<empty>;c2:<(order 10, $100)>;

Example A possible sequence of global states is the following: S0: p1: <$1000,0>; p2:<$50,2000>;c1:<>; c2:<> S1’: p1: <$1000,0>; p2:<$50,1995>;c1:<(five widgets)>; c2:<> Recorded State: p1: <$1000,0>; p2:<$50,1995>;c1:<(five widgets)>; c2:<>

Example S0 S1 S1’ … S2 S3

Example The recorded state is not the same as one of the actual states. The recorded state is one that was possible in one of the possible sequences of global states. Why is this useful? What can we do with this? To illustrate we will look at another example.

Example What if p1 initiates the snapshot algorithm after it issues a request for the 10 widgets and before it receives the 5 widgets? This would imply that p2 gets the order for the 10 widgets before the marker. This causes p2 to update its account and number of widgets to $150 and 1985 respectively. p2 will record this state when it receives the marker. p2 sends the 10 widgets. The state of c1 is <(5 widgets),(10 widgets)> and c2 is marked as empty.

Example 2 Say in process P1 you have a code segment as follows: 1.2 recv(b, P2); 1.3 c = a+ b; 1.4 send(c,P2); Say in process P2 you have a code segment as follows: 2.1 x = 5; 2.2 recv(y,P1); 2.3 z = x+y; 2.4 send(z, P1); State includes values of a,b,c,x,y,z and for each process records what that process is waiting (if there is anything).

Example 2 S0 c p p c a=5 b=0 c=0 x=0 y=0 z=0 1 2 c 1 a=5 b=0 c=0 x=0 y=0 z=0 This is the state when the algorithm was initiated; 1.1 has been executed but 2.1 has not; marker is sent out after 1.1 has been executed but before trying to execute 1.2 Channels are empty

Example 2 S1 c p p c a=5 b=0 c=0 p2 is waiting for message from p1 x=5 y=0 z=0 2.1 has been executed before it receives the marker from p1 Upon receiving the marker it records that c2 is empty , that x=5,y=0,z=0 and that it is waiting for a message from p1 It sends out a marker on c1

Example 2 S2 c p p c a=5 b=0 c=0 p2 is waiting for message from p1 x=5 y=0 z=0 p1 receives a marker on c1 It records the state of its incoming channel as empty. The final state recorded is this:a=5,b=0,c=0, x=5,y=0,z=0; p2 is waiting for a message from p1 and channels are empty.

Example 2 So far the deadlock has not been detected. Let’s say that after P1 starts waiting for a message from P2 that it initiates the global snapshot algorithm again. P1 records its state as a=5,b=0,c=0 and that it is waiting for a message from P2 P1 sends the marker to P2 . P2 records its state as x=5,y=0,z=0, incoming channel is empty and it is waiting for a message from P1 P2 sends a marker on c1; When P1 receives the marker it records the state of channel c1 as empty

Example 2 At this point the global state recorded includes the following: a=5,b=0,c=0 x=5,y=0,z=0 Both channels are empty P1 is waiting for a message from P2 and P2 is waiting for a message from P1 This recorded state has enough information to determine that there is a deadlock. If the algorithm is initiated again the global state still shows a deadlock.

Example 3 Send A Recv C A M A Send B Recv M, Record State, Channel (2,1)empty p1 p0 Record State Send M M Recv A B C Recv M, Record State, Channel (0,1)A B C M p2 Send C Recv B Recv M, Record State, Channel (0,1)empty, Send M

Stable Properties The global state computed using the algorithm described can be used for detecting stable properties, e.g., Distributed deadlock detection Distributed garbage collection A stable property is a property that once it holds true in a state Si then it holds true in state Sj where j > i

Stable Properties Let’s look at deadlock detection. Assume each process has a list waits-for which records the list of processes that the process is waiting for a message from. The global state includes values of the waits-for list from each process. Let’s assume two processes p1 and p2. Let p1.waits-for = {p2} and p2.waits-for = {p1}. This represents a deadlock. If there is a deadlock in state Si then there is a deadlock in Sj (where j > i).

Stable Properties S0 S1 S1’ S2 S2’ S3 If deadlock has happened in S3 then it happens in subsequent states.

Summary No shared memory in distributed systems Very difficult to determine exactly what a system’s current state is. Determining the global state of a distributed system can be done by synchronizing all processes so that each collects its own local state along with the messages that are currently in transit.