1. Chapter 10 Distributed Shared Memory
Introduction
Design and implementation issues
Sequential consistency and IVY
Release consistency and Munin
Summary

2. 10.1 Introduction to DSM Why DSM (Distributed Shared Memory)?
- Message passing is complex (dropped msgs, etc)
- Hard to pass complex data structures in RPCs
- Shared-memory multiprocessors easier to program(such as synchronization)
- Processes communicating via DSM may execute with
non-overlapping lifetimes
- But multicomputers easier to build and cheaper
What is Distributed shared memory?
- Processors share a virtual address space(VAS)
- Perhaps there is no global memory
- Perhaps they have private memories
- Private memories cache pages from VAS
- May need consistency between memory caches

3. Main approaches to DSM? - Hardware-based: rely on specialized hardware
- Page-based: as a region of virtual memory occupying the same
address range in the address space of every participating process
- Library-based: library calls are responsible for accessing DSM and
maintaining consistency
Example: Mether system program
#include “world.h” #include “world.h”
struct shared { struct shared {
int a,b; int a,b;
} }
Program Writer: Program Reader:
main() main()
{ {
struct shared *p; struct shared *p;
methersetup(); methersetup();
p = (struct shared*) METHERBASE; p = (struct shared*) METHERBASE;
p->a = p->b = 0; while(TRUE){
while(TRUE){ printf(“a=%d,b=%d\n”, p->a, p-b);
p->a = p->a + 1: sleep(1);
p->b = p->b - 1; }
} } }

4. 10.2 Design and implementation issues
Structure
- Byte-oriented
- Shared objects
- Immutable data
Synchronization model
- A distributed synchronization service needs to help applications to
apply constraints concerning the values in DSM.
Example: (two processes share a & b)
constraints: a=b
initially: a=b=0
a ++;
b ++;
Consistency model
- It’s different from the above synchronization model.

5. Consistency model - Example: two processes accessing shared variables
Process Process 2
br:=b; a:=a+1;
ar:=a; b:=b+1
if (ar>=br) then
printf(“OK”);
Could ar =0, br=1 occur? If occur, is it right?
- A contract between software and memory
- If software obeys certain rules, memory provides certain guarantees
- The fewer guarantees, the better the performance
- some consistency models
strict consistency, sequential consistency, causal consistency, weak
consistency, release consistency

6. Sequential consistency
Strict consistency
- Any read to location x returns most recent value stored in x
- We expect this on uniprocessors
- Assumes absolute global time
- Is this even possible in a distributed system?
- What if two writes from processors are a ns apart?
- Use locking and critical sections instead
Sequential consistency
- Slightly weaker - from Lamport 1979
- Any interleaving of operations is okay, but they must satisfy the following
conditions:
all the reads and writes issued by the individual processes concerned are
satisfied in program order;
the memory operations belonging to different processes occur in some
serial order
- Programmer-friendly, but slow

7. - Example: An example serialization under sequential consistency
Process Process 2
br:=b; a:=a+1;
ar:=a; b:=b+1
if (ar>=br) then
printf(“OK”);
Could ar =0, br=1 occur under sequential consistency model?
Time
read
write
Causal consistency
- Only concerned with events that are potentially causally related
- Causally-related writes must be seen by all processes in same order
- Concurrent writes can be seen in different orders
- Example: In a newsgroup, there are 3 messages:
Msg 1: Does anyone know the time of our next meeting?
Msg 2: 3 pm
Msg 3: A good recipe for mushrooms stuffed with salmon is ...
- Msg 1 & 2 are causally related, Msg 3 is not
Msg 2 wouldn’t make sense unless you’d received Msg 1 already
Msg 3 makes sense anytime

8. Weak consistency Example: valid ordering P1: W(x) 1 W(x) 2 S
- Not all applications even to see writes
- Example: writes in critical section
- Who cares what order others see them, they shouldn’t look!
- Need synchronization variable
- Synch. Var. Accesses are sequentially consistent
- Synchronize memory only at synchronization points
- Accessing a synch. Var. “flushes the pipeline”
- No data access allowed until all previous synch. Accesses have been performed
- To get consistent value, do a synch before read
Example:
valid ordering
P1: W(x) 1 W(x) 2 S
P2: R(x) 1 R(x) 2 S
P3: R(x) 2 R(x) 1 S
invalid ordering
P2: S R(x) 1

9. Release consistency Example: valid ordering
- When synch done before, is this end of writes or beginning of reads?
- Had to finish local writes and gather remote ones
- Release consistency fixes this
- Two kinds of synch. Accesses
- Acquire means you’re about to enter critical section
- Release means you’ve exited critical section
- Can also use barrier synchronization
Example:
valid ordering
P1: Acq(L) W(x) 1 W(x)2 Rel(L)
P2: Acq(L) R(x) 2 Rel(L)
P3: R(x) 1
- Eager versus lazy release consistency
Eager: process doing the release pushes modified data to all processors
with cached copies
But do they all need it?
Lazy: pull most recent values upon acquire
With a critical section in a loop, this saves a lot!

10. Update options DSM using write-update printf(“after”); P1 P2 P3
- How to propagate updates? => Write-update vs write-invalidate
- Write-update
multiple-reader-multiple-writer sharing
memory consistency model mainly depends on multicast ordering property
DSM using write-update
a:=7
b:= 7;
if(b=8) then
printf(“after”);
If(a=7)then
b:=b+1
... P1
time
time P2
If(b=a)then
printf(“before”); P3
time

11. Granularity Thrashing Data items laid out over pages A B Page n
-Write-invalidate
multiple-reader-single-writer sharing
this scheme can achieve sequential consistency
Granularity
- Page-based implementation
- large or small?
- False sharing
Data items laid out over pages
A B
Page n
Thrashing
- occurs when several processes compete for the same data item, or for
falsely shared data items.

12. 10.3 Sequential consistency and Ivy
The system model
Process accessing paged DSM segment
Page
faults
Kernel
Pages transferred over network
- sequential consistent, page-based DSM
- paging is transparent to the processes
- DSM run time restricts page access permissions: none, read-only or
read-write.

13. Write-invalidation - Can’t use write-update, why?
Write protected Page fault write permission
Trace mode trace exception handler
- Write invalidation
Page protection: read-only permission & read/write permission
owner & copyset
State transitions under write-invalidation
Multiple reader
Single writer R
W (invalidation) R
W (invalidation)
Note: R = read fault occurs; W= write fault occurs

14. - Write fault handling procedure is as follows:
The page is transferred to Pw’s(Pw is a process which attempts to
write a page p) kernel, if it does not already have an up-to-date read-
only copy.
All other copies are invalidated: the page permissions are set to no-
access at all members of copyset(p).
copyset(p) := {Pw}.
owner(p) := Pw.
The kernel maps the page with read-write permissions into Pw’s
address space, and Pw is restarted.

15. - Read fault handling procedure is as follows:
The page is copied from owner(p) to Pr’s(Pr is a process which
attempts to read a page p) kernel.
If owner(p) is a single writer, then its access permission for p is set
to read-only access and it remains as p’s owner..
copyset(p) := copyset(p) {Pr}.
Pr’s kernel maps the page with read-only permissions into Pr’s
address space, and Pr continues.
- Two problems remain to be addressed:
How to locate owner(p) for a given page p?
Where to store copyset(p)?

16. - The approaches to these problems Li Kai described include:
centralized manager algorithm
fixed distributed page management
multicast -based distributed management
dynamic distributed management
Centralized manager algorithm
Faulting process (i.e. client) Current owner
3. Page
2. Requestor, page no., access
1. Page no., access(R/W)
Page Owner
no.
Manager

17. Fixed distributed page management
- pages are divided statically among multiple managers, but processes
maybe don’t access the pages equally.
Using multicast to locate the owner
- If two or more clients request the same page at more or less the same
time, we must ensure that each client must obtain the page eventually! 3
Owner(p) O 1 2
Client C2
Client C1 4

18. A dynamic distributed manager algorithm
- allows page ownership to be transferred between kernels
- to divide the overheads of locating pages among the computers that
access them
- probable owner of p, or probOwner(p) is just a hint
- the owner of a page is located by following chains of hints
- hints are updated and requests are forwarded as follows:
when a kernel transfers ownership of page p to another kernel, it
updates probOwner(p) to be the recipient.
when a kernel(not the owner) handles an invalidation request for a
page p, it updates probOwner(p) to be the requester.
when a kernel that has requested read access to a page p receives it,
it updates probOwner(p) to be the provider.
when a kernel receives a request(whether it’s for read access or
write access ) for a page p that it does not own, it forwards the
request to probOwner(p), and resets probOwner(p) to be the
requester.

19. B A C D E Owner B A C D E Owner B A C D E Owner
(a) probOwner pointers just before process A takes a page fault for a page owned by E B A C D E
Owner
(b) Write fault:probOwner pointers after A’s write request is forwarded. B A C D E
Owner
(c) Read fault:probOwner pointers after A’s read request is forwarded.

20. periodically broadcasting the current owner’s location to all kernels
- optimization 1
periodically broadcasting the current owner’s location to all kernels
to reduce the average length of pointer chains.
Simulation results:
Page faults per broadcast average length of pointer chains
1024
3.64
2.34
256
- optimization 2
a client can obtain a copy from any kernel with a valid copy
owner 1 2 3
Some invalidations can occur in parallel! 4 5 6 7 8

21. 10.4 Release consistency and Munin
- Sequential consistency is costly to implement
1、using multicasts; 2、locating the owner of a page.
- Release consistency is weaker than sequential consistency and
cheaper to implement, but has reasonable semantics.
E.g., DASH, Munin
Re-examine release consistency
- its idea is to reduce DSM overhead by exploiting the fact that
programmers use synchronization objects
- ordinary accesses(to DSM) vs synchronization accesses
- its main guarantee is as follows:
All ordinary memory access issued prior to a release have taken
effect at all other processes before the release completes -
including those accesses issued prior to the preceding acquire
- by using appropriate synchronization accesses, it can give equivalent
results to executions under sequential consistency model

22. - Munin’s synchronization objects
>> acquireLock, releaseLock and waitAtBarrier
E.g.,
Process 1: Process 2:
acquireLock(); /*enter critical section*/ acquireLock(); /*enter critical section*/
a:= a+1; printf(“the values of a & b are:”,a,b);
b:= b+1; releaseLock(); /*leave critical section*/
releaseLock(); /*leave critical section*/
Advantages of release consistency:
- to avoid some blocking of processes
- to delay some communication until a release occurs

23. Munin - Shared memory programs on shared distributed memory
multiprocessors, in this case a workstation cluster
- Release consistent memory and eager approach
Munin sends update or invalidation information as soon as a lock is
released
- Multiple consistency protocols, parameterized according to some
options, such as:
whether to use a write-update or write-invalidate protocol
whether the data item has a fixed owner
whether or not to delay updates or invalidations
- Annotate shared variables with expected access patterns, examples:
read-only - replication on demand
producer-consumer - update rather than invalidate
write-shared - avoiding false-sharing and only the differences
between the two versions are sent in an update

24. That’s it, thanks for your attention!
In next class we’ll continue discussing:
Chapter 11 Time and Coordination
Synchronizing physical clocks
Logical time and logical clocks
Distributed coordination
That’s it, thanks for your attention!