1. Shared Memory – Consistency of Shared Variables The ideal picture of shared memory: CPU0CPU1CPU2CPU3 Shared Memory Read/ Write The actual architecture of shared memory systems: R/W of Misses + Cache Invalidate CPU0CPU1CPU2CPU3 Shared Memory Read/ Write Local Cache Local Cache Local Cache Local Cache Symmetric Multi-Processor (SMP): CPU0CPU1CPU2CPU3 Local Memory Module Local Memory Module Local Memory Module Local Memory Module Network Distributed Shared Memory (DSM):

2. The Million $$s Question: How/When Does One Process Read Other Process’s Writes? CPUi Write value x to local copy of shared variable V W V,x Assumption: Initial value of shared variables is always 0. CPUj R V,0? R V,x? Read V from local copy Why is this a question? Because temporal order relations like “before/after” do not necessarily hold in a distributed system.

3. Non-Atomic write s/ read s (also called load s /store s) A read by Pi is considered performed with respect to Pk at a point in time when the issuing of a write to the same address by Pk cannot affect the value returned by the read. A write by Pi is considered performed with respect to Pk at a point in time when an issued read to the same address by Pk returns the value defined by this write (or a subsequent write to the same location). An access is performed when it is performed with respect to all processors. A read is globally performed if it is performed and if the write that is the source of the returned value has been performed. In what follows, we will think of atomic read/write but these definitions can be used to generalize.

4. Why Memory Model? a=0,b=0 Print(b)Print(a) a=1b=1 Printed:0,0? Printed:1,0? Printed:1,1? Answers the question: “Which writes by a process are seen by which reads of the other processes?”

5. Memory Consistency Models Pi: R V; W V,7; R V; R V Pj: R V; W V,13; R V; R V Example program: A consistency/memory model is an “agreement” between the execution environment (H/W, OS, middleware) and the processes. Runtime guarantees to the application certain properties on the way values written to shared variables become visible to reads. This determines the memory model, what’s valid, what’s not. Example execution: Pi: R V,0; W V,7; R V,7; R V,13 Pj: R V,0; W V,13; R V,13; R V,7 Order of writes to V as seen to Pi: (1) W V,7; (2) W V,13 Order of writes to V as seen to Pj: (1) W V,13; (2) W V,7

6. Memory Model: Coherence Coherence is the memory model in which (the runtime guarantees to the program that) writes performed by the processes for every specific variable are viewed by all processes in the same full order. Example program:All valid executions under Coherence: Pi: W V,7 R V Pj: W V,13 R V Note: the view of a process consists of the values it “sees” in its reads, and the writes it performs. Thus, if a R V in P which is later than a W V,x in P sees a value different than x, then a later R V cannot see x. Pi: W V,7 R V,7 Pj: W V,13 R V,13 R V,7 Pi: W V,7 R V,7 Pj: W V,13 R V,7 Pi: W V,7 R V,7 R V,13 Pj: W V,13 R V,13 Pi: W V,7 R V,13 Pj: W V,13 R V,13 Pi: W V,7 R V,7 Pj: W V,13 R V,13

7. Formal definition of Coherence Program Order: The order in which instructions appear in each process. This is a partial order on all the instructions in the program. A serialization: A full order on all the instructions (reads/writes) of all the processes, which is consistent with the program order. A legal serialization: A serialization in which each read X returns the value written by the latest write X in the full order. Let P be a program; let P X be the “sub-program” of P which contains all the read X/write X operations on X only. Coherence: P is said to be coherent if for every variable X there exists a legal serialization of P X. (Note: a process cannot distinguish one such serialization from another for a given execution)

8. Examples Process 1 write x,1 write x,2 Process 2 read x,2 read x,1 Coherent. Serializations: x: write x,1, read x,1 y: write y,1, read y,1 Not Coherent. Cycle of dependencies. Cannot be serialized. Not Coherent. Cannot be serialized. Process 2 read y,1 write x,1 Process 1 read x,1 write y,1 Process 1 read x,1 write x,2 Process 2 read x,2 write x,1 Process 2 read y,1 write x,1

9. Sequential Consistency [Lamport 1979] Sequential Consistency is the memory model in which all reads/writes performed by the processes are viewed by all processes in the same full order. Coherent. Not Sequentially consistent. Coherent. Not Sequentially consistent. Process 1 write x,1 write y,1 Process 2 read y,1 read x,0 Process 1 read x,1 write y,1 Process 2 read y,1 write x,1

10. Strict (Strong) Memory Models a=0,b=0 Print(b)Print(a) a=1b=1 Printed:0,0 or 0,1 or 1,0 Printed:1,1 Sequential Consistency: Given an execution, there exists an order of reads/writes which is consistent with all program orders. Coherence: For any variable x, there exists an order of read x/write x consistent with all p.o.s.

11. Formal definition of Sequential Consistency Let P be a program. Sequential Consistency: P is said to be sequentially consistent if there exists a legal serialization of all reads/writes in P. Observation: Every program which is sequentially consistent is also coherent. Conclusion: Sequential Consistency has stronger requirements and we thus say that it is stronger than Coherence. In general: A consistency model A is said to be (strictly) stronger than B if all executions which are valid under A are also valid under B.

12. The problem of strong consistency models The runtime system should ensure the existence of legal serialization, and the same consistent view for all processes. This requires lots of expensive coordination  degrades performance! P1: Print(U) Write V,1 P2: Print(V) Write U,1 SC: Hardware cannot reorder locally in each thread for this will result in a possible printing 1,1. HW may reorder anyway and postpone writes, but then why reorder in the first place?

13. Coherence Forbids Reordering p.x = 0 p.x=1 a=p.x b=q.x assert(a  b) Once thread sees an update – cannot “forget” it has seen it.  Cannot reorder two reads of the same memory location. q.x is aliased to p.x. Reordering may make assignment to B early (seeing 0) and that to A late (seeing 1). The right thread see order of writes different from left thread.

14. Coherence makes read s prevent common compiler optimizations p and q might point to same object p.x = 0 p.x=1 a=p.x b=q.x assert(p==q  a  b  c) Cannot put c=ac=p.x reads can make a process see writes by another process. The read “kills” later reuse of local values.

15. Causal Consistency If event B is caused or influenced by an earlier event A, Causal Consistency requires that all the processes first see A, then see B. Causally related events: Events A and B are causally related if A causes or influences B. Concurrent events: Events A and B are concurrent/independent if they are not causally related. Formally: Write After Read: When a Read operation is followed in program order by a Write operation, then the two events are potentially causally related. Read After Write: A Read(x) operation is causally related to the Write(x) operation that provided the data the Read got. Transitive closure: when there is a C so A is [potentially] causally related to C and C is [potentially] causally related to B then A is causally related to B. Casual Consistency: An execution obeys causal consistency if Writes that are potentially casually related are seen by all processes in the same order. Concurrent writes may be seen in a different order on different machines.

16. Example – Causally Consistent P1:P1:W 1 (x),1W 1 (x),3 P2:P2:R 2 (x),1W 2 (x),2 P3:P3:R 3 (x),1R 3 (x),3R 3 (x),2 P4:P4:R 4 (x),1R 4 (x),2R 4 (x),3 W2(x),2 and W1(x),3 are concurrent events (so it is not required that all processes see them in the same order). The only writes that are potentially causally related are W1(x),1 and W2(x),2. The above sequence of events is Causally Consistent but not Sequentially Consistent (the readings of P3 – R3(x),3 and R3(x),2 are not allowed in a sequentially consistent memory model).

17. Example – Causally Inconsistent P1:P1:W 1 (x),1 P2:P2:R 2 (x),1W 2 (x),2 P3:P3:R 3 (x),2R 3 (x),1 P4:P4:R 4 (x),1R 4 (x),2 W2(x),2 is potentially causally related to W1(x),1 (because the writing of 2 may result from the value read by R2(x),1). Since the two writes are potentially causally related all processes must see them in the same order, however, P3 and P4 see them in a different order.

18. Example – Causally Consistent P1:P1:W 1 (x),1 P2:P2:W 2 (x),2 P3:P3:R 3 (x),2R 3 (x),1 P4:P4:R 4 (x),1R 4 (x),2 W2(x),2 and W1(x),1 are concurrent events (the read R2(x),1 was removed). Since the two writes are concurrent they may be seen in a different order by different processes.