1 Principles of Reliable Distributed Systems Lecture 10: Atomic Shared Memory Objects and Shared Memory Emulations Spring 2007 Prof. Idit Keidar

2 Material Attiya and Welch, Distributed Computing –Ch. 9 & 10 Nancy Lynch, Distributed Algorithms –Ch. 13 & 17 Linearizability slides adapted from Maurice Herlihy

3 Shared Memory Model All communication through shared memory! –No message-passing. Shared memory registers/objects. Accessed by processes with ids 1,2,… Note: we have two types of entities: objects and processes.

4 Motivation Multiprocessor architectures with shared memory Multi-threaded programs Distributed shared memory (DSM) Abstraction for message passing systems –We will see how to emulate shared memory in message passing systems. –We will see how to use shared memory for consensus and state machine replication.

5 Linearizability Semantics for Concurrent Objects

6 FIFO Queue: Enqueue Method q.enq ( ) Process

7 FIFO Queue: Dequeue Method q.deq()/ Process

8 Sequential Objects Each object has a state –Usually given by a set of fields –Queue example: sequence of items Each object has a set of methods –Only way to manipulate state –Queue example: enq and deq methods

9 Methods Take Time time Method call invocation 12:00 q.enq(... ) response 12:01 void

10 Split Method Calls into Two Events Invocation –method name & args –q.enq(x) Response –result or exception –q.enq(x) returns void –q.deq() returns x –q.deq() throws empty

11 A Single Process (Thread) Sequence of events First event is an invocation Alternates matching invocations and responses This is called a well-formed interaction

12 Concurrent Methods Take Overlapping Time time Method call

13 Concurrent Objects What does it mean for a concurrent object to be correct? What is a concurrent FIFO queue? –FIFO means strict temporal order –Concurrent means ambiguous temporal order Help!

14 Sequential Specifications Precondition, say for q.deq( … ) –Queue is non-empty Postcondition: –Returns & removes first item in queue You got a problem with that?

15 Concurrent Specifications Naïve approach –Object has n methods –Must specify O(n 2 ) possible interactions –Maybe more If the quque is empty and then enq begins and deq begins after enq(x) begins but before enq(x) ends then … Linearizability: same as it ever was

16 Linearizability Each method should – –“Take effect” Effect defined by the sequential specification –Instantaneously Take 0 time –Between its invocation and response events.

17 Linearization A linearization of a concurrent execution  is –A sequential execution Each invocation is immediately followed by its response Satisfies the object’s sequential specification –Looks like  Responses to all invocations are the same as in  –Preserves real-time order Each invocation-response pair occurs between the corresponding invocation and response in 

18 Linearizability and Atomicity A concurrent execution that has a linearization is linearizable. An object that has only linearizable executions is atomic.

19 Why Linearizability? “Religion”, not science Scientific justification: –Facilitates reasoning –Nice mathematic properties Common-sense justification –Preserves real-time order –Matches my intuition (sorry about yours)

20 Example time q.enq(x) q.enq(y)q.deq(x) q.deq(y) time

21 Example time q.enq(x) q.enq(y) q.deq(y)

22 Example time q.enq(x) q.deq(x) time

23 Example time q.enq(x) q.enq(y) q.deq(y) q.deq(x) time

24 Read/Write Variable Example time read(1)write(0) write(1) time read(0)

25 Read/Write Variable Example time read(1)write(0) write(1) write(2) time read(1)

26 Read/Write Variable Example time read(1)write(0) write(1) write(2) time read(2)

27 Concurrency How much concurrency does linearizability allow? When must a method invocation block? Focus on total methods –defined in every state –why?

28 Concurrency Question: when does linearizability require a method invocation to block? Answer: never Linearizability is non-blocking

29 Non-Blocking Theorem If method invocation A q.invoc() is pending in linearizable history H, then there exists a response A q:resp() such that H + A q:resp() is legal

30 Note on Non-Blocking A given implementation of linearizability may be blocking The property itself does not mandate it –for every pending invocation, there is always a possible return value that does not violate linearizability –the implementation may not always know it…

31 Atomic Objects An object is atomic if all of its concurrent executions are linearizable What if we want an atomic operation on multiple objects?

32 Serializability A transaction is a finite sequence of method calls A history is serializable if –transactions appear to execute serially Strictly serializable if –order is compatible with real-time Used in databases

33 Serializability is Blocking x.read(0) y.read(0)x.write(1) y.write(1)

34 Comparison Serializability appropriate for –fault-tolerance –multi-step transactions Linearizability appropriate for –single objects –multiprocessor synchronization

35 Critical Sections Easy way to implement linearizability –take sequential object –make each method a critical section Like synchronized methods in Java™ Problems? –Blocking –No concurrency

36 Linearizability Summary Linearizability –Operation takes effect instantaneously between invocation and response Uses sequential specification –No O(n 2 ) interactions Non-Blocking –Never required to pause method call Granularity matters

37 Atomic Register Emulation in a Message-Passing System [ Attiya, Bar-Noy, Dolev ]

38 Distributed Shared Memory (DSM) Can we provide the illusion of atomic shared-memory registers in a message- passing system? In an asynchronous system? Where processes can fail?

39 Liveness Requirement Wait-freedom (wait-free termination): every operation by a correct process p completes in a finite number of p’s steps Regardless of steps taken by other processes –In particular, the other processes may fail or take any number of steps between p’s steps –But p must be given a chance to take as many steps as it needs. (Fairness).

40 Register Holds a value Can be read Can be written Interface: –int read(); /* returns a value */ –void write(int v); /* returns ack */

41 Take I: Failure-Free Case Each process keeps a local copy of the register Let’s try state machine replication –Step1: Implement atomic broadcast (how?) Recall: atomic broadcast service interface: –broadcast(m) –deliver(m)

42 Emulation with Atomic Broadcast (Failure-Free) Upon client request ( read / write ), –Broadcast the request Upon deliver write request –Write to local copy of register –If from local client, return ack to client Upon deliver read request –If from local client, return local register value to client Homework questions: –Show that the emulated register is atomic –Is broadcasting reads required for atomicity?

43 What If Processes Can Crash? Does the same solution work?

44 ABD: Fault-Tolerant Emulation [ Attiya, Bar-Noy, Dolev ] Assumes up to f<n/2 processes can fail Main ideas: –Store value at majority of processes before write completes –read from majority –read intersects write, hence sees latest value

45 Take II: 1-Reader 1-Writer (SRSW) Single-reader – there is only one process that can read from the register Single-writer – there is only one process that can write to the register The reader and writer are just 2 processes; –The other n-2 processes are there to help

46 Trivial Solution? Writer simply sends message to reader –When does it return ack ? –What about failures? We want a wait-free solution: –if the reader (writer) fails, the writer (reader) should be able to continue writing (reading)

47 SRSW Algorithm: Variables At each process: –x, a copy of the register –t, initially 0, unique tag associated with latest write

48 SRSW Algorithm Emulating Write To perform write(x,v) –choose tag > t –set x ← v; t ← tag –send (“write”, v, t) to all Upon receive (“write”, v, tag) –if (tag > t) then set x ← v; t ← tag fi –send (“ack”, v, tag) to writer When writer receives (“ack”, v, t) from majority (counting an ack from itslef too) –return ack to client

49 SRSW Algorithm Emulating Read To perform read(x,v) –send (“read”) to all Upon receive (“read”) –send (“read-ack”, x, t) to reader When reader receives (“read-ack”, v, tag) from majority (including local values of x and t) –choose value v associated with largest tag –store these values in x,t –return x

50 Does This Work? Only possible overlap is between read and write –why? When a read does not overlap any write – –it reads at least one copy that was written by the latest write (why?) –this copy has the highest tag (why?) What is the linearization order when there is overlap? What if 2 read s overlap the same write ?

51 Example time read(1)read(?) write(1) time

52 Wait-Freedom Only waiting is for majority of responses There is a correct majority All correct processes respond to all requests –Respond even if the tag is smaller

53 Take III: n-Reader 1-Writer (MRSW) n-reader – all the processes can read Does the previous solution work? What if 2 read s by different processes overlap the same write ?

54 Example time read(1) read(?) write(1) time

55 MRSW Algorithm Extending the Read When reader receives (“read-ack”, v, tag) from majority –choose value v associated with largest tag –store these values in x,t –send (“propagate”, x, t) to all (except writer) Upon receive (“propagate”, v, tag) from process i –if (tag > t) then set x ← v; t ← tag fi –send (“prop-ack”, x, t) to process i When reader receives (“prop-ack”, v, tag) from majority (including itself) –return x

56 The Complete Read S1 S2 Sn S1 S2 Sn S1 (“read”)(“read-ack”,v, t) Phase 1Phase 2 : Multi-reader only read() return (“propagate”, v, t) (“prop-ack”)

57 Take IV: n-Reader n-Writer (MRMW) n-writer – all the processes can write to the register Does the previous solution work?

58 Playing Tag What if two writers use the same tag for writing different values? Need to ensure unique tags –That’s easy: break ties, e.g., by process id What if a later write uses a smaller tag than an earlier one? –Must be prevented (why?)

59 MRMW Algorithm Extending the Write To perform write(x,v) –send (“query”) to all Upon receive (“query”) from i –send (“query-ack”, t) to i When writer receives (“query-ack”, tag) from majority (counting its own tag) –choose unique tag > all received tags –continue as in 1-writer algorithm What if another writer chooses a higher tag before write completes?

60 The Complete Write S1 S2 Sn S1 S2 Sn S1 (“query”)(“query-ack”, t) Phase 1: Multi-writer onlyPhase 2 write(v) ack (“write”, v, t)(“ack”)

61 How Long Does it Take? The write emulation –Single-writer: 2 rounds (steps) –Multi-writer: 4 rounds (steps) The read emulation –Single-reader: 2 rounds (steps) –Multi-reader: 4 rounds (steps)

62 What if A Majority Can Fail? You guessed it! Homework question

63 Can We Emulate Every Atomic Object the Same Way?

64 Difference from Consensus Works even if the system is completely asynchronous In Paxos, there is no progress when there are multiple leaders Here, there is always progress – multiple writers can write concurrently –One will prevail (Which?)