1. Introduction to Software Distributed Shared Memory Systems Chang-Yi Lin 2004 / 02 / 26

2. Outlines  What is a software DSM system?  Message-passing vs. Shared-memory  How does it work?  Memory Consistency Models  Cache Coherence  Implementation Levels  Granularity

3. What is a software DSM system?  A distributed-memory system (often called a multicomputer) consist of multiple independent processing nodes with local memory modules, connected by a general interconnection network.  Global shared memory.

6. What is a software DSM system?  A DSM system logically implements the shared-memory model on a physically distributed-memory system.  The DSM system hides the remote communication mechanism from the application writer, preserving the programming ease and portability typical of shared-memory systems.

7. Message-passing vs. Shared-memory  Two different programming type.  Shared-memory programming is easier.

8. Message-passing vs. Shared-memory  Message-passing Point-to-point communication

9. Message-passing vs. Shared-memory  Message-passing buffers and data types blocking and nonblocking

10. Message-passing vs. Shared-memory

11.  Pthread Thread 1 Lock A = A++; unlock Thread 2 Lock A = A++; unlock Thread 3 Lock A = A++; unlock

13. How does it work?

16. Memory Consistency Models  What is Memory consistency? 在單機上 P1: w(x)1 R(x)1 ----------------------------------  time 在兩台機器上 P1: w(x)1 P2: R(x)? ----------------------------------------  time

17. Memory Consistency Models  A consistency model is essentially a contract between the software and the memory.  If the software agrees to obey certain rules, the memory promises to work correctly.  If the software violates these rules, correctness of memory operation is no longer guaranteed.

18. Memory Consistency Models  Strict consistency  Sequential consistency (SC)  Release consistency (RC)  Scope consistency (ScC)

19. Memory Consistency Models  Strict consistency Definition: any read to memory location x returns the value stored by the most recent write operation to x. Impossible to DSM P1: w(x)1 P2: R(x)1 ----------------------------------------  time

20. Memory Consistency Models  Sequential consistency (SC) Definition: the result of any execution is the same as if the operations of all processors were executed in some sequential order, and the operations of each individual processor appear in this sequence in the order specified by its program. Any valid interleaving is acceptable behavior, but all processes must see the same sequence of memory reference.

21. Memory Consistency Models  Sequential consistency (SC) P1 (A) a=1; (B) Print(b,c); P2 (C) b=1; (D) Print(a,c); P3 (E) c=1; (F) Print(a,b); (A) (B) (C) (D) (E) (F) (A) (C) (D) (B) (E) (F) (C) (E) (F) (D) (A) (B) (A) (C) (E) (B) (D) (F)

22. Memory Consistency Models  Release Consistency (RC) Two types of access  Ordinary access: read and write  Synchronization access: acquire lock, release lock and barrier Rules:  Before an ordinary access to a shared variable is performed, all previous acquires done by the process must have completed successfully.  Before a release is allowed to be performed, all previous reads and writes done by the process must have completed.

23. Memory Consistency Models  Release Consistency (RC) Rule1: … Acq(L1) … Rel(L1) … Acq(L5) … Acq(L3) … R(x) Rule2: Acq(L2) w(x) R(y) R(z) Rel(L2) Example: P1: Acq(L) w(x)1 w(x)2 Rel(L) P2: Acq(L) R(x)2 Rel(L) P3: R(x)?

24. Memory Consistency Models  Scope consistency (ScC)

25. Memory Consistency Models  Relaxing consistency permits temporary inconsistencies (delayed updates) Lazy release consistency (LRC) (TreadMarks, CVM) Scope consistency (ScC) (JIAJIA, JUMP)

26. Cache Coherence  Write invalidate Suffer from false sharing  Write update Too expansive when many replicas Work best in application with tight sharing

27. Implementation Levels  Modifying OS kernel IVY (SC): modifying the memory management unit (MMU) of OS to map between the shared virtual memory address space and the local memory.  Language level Linda, Orca  User-level runtime library Trademarks, CVM, JIAJIA, JUMP, Brazos  Combination of multiple implementation levels, even hardware support Munin, Midway, NCP2

28. Granularity  The choice of the block size depends on the cost of communcation  1 byte message v.s. 1024 byte message Locality of reference in the application  Most DSM systems use a page-based granularity with 1K byte to 8K byte.  Larger page size, better locality of reference

29. 系統開發者實作層次顆粒度一致性模型一致性協定 IVYYale 函式庫＋作業系統頁 (1KB) SCWI MuninRice 函式庫＋作業系統可變 Eager RCWU/WI TreadMarksRice 函式庫頁 (4KB) LRC MW ， WI CVMMaryland 函式庫頁 LRC-MW ， LRC- SW ， SC WU MidwayCMU 函式庫＋編譯器可變 EC ， PC ， RC WU NCP2UFRJ, Brail 函式庫＋硬體支援頁 (4KB)EC ， RC WU/WI QuarksUtah 函式庫 region 、頁 RC ， SCWU/WI ， MW softFLASHStandford 作業系統頁 (16KB)RC ， DIRC FLASH-like Cashmere-2LRochester 函式庫頁 (8KB) HLRCWU BrazosRice 函式庫頁 ScC Early update ， WU ShastaDEC WRL 編譯器可變 SCWI MermaidToronto 函式庫＋作業系統頁 (1KB ， 8KB) SCWI Dsoftware DSM6KIBM Research 作業系統頁 (4KB) SCWI MirageUCLA 作業系統 512BytesSCWI JIAJIA 中國科學院函式庫頁 (4KB) ScCWI Simple-COMASICS(Sweden) and SUN 作業系統頁 SCWI Blizzard-SWisconsin 函式庫快取行 SCWI ShrimpPrinceton 作業系統＋硬體支援頁 AURC ， SC WU/WI LindaYale 語言可變 SCImpl.dependent OrcaVrije Univ., Netherlands 語言可變同步相關 WU