1. CSCS: A Concise Implementation of User-Level Distributed Shared Memory Zhi Zhai Feng Shen Computer Science and Engineering University of Notre Dame Dec. 11, 2009 Final Presentation

2. DSM Overview DSM Characteristics: Physically: distributed memory Logically: a single shared address space Figure 1 DSM architecture

3. Related Work Models and Main Features: IVY (Yale) - Divided Space: Shared & Private space Mirage (UCLA) - Time Interval d : Avoid page thrashing TreadMarks (Rice) - Lazy Release Consistency : Improve efficiency SAM (Stanford)

4. System Design Figure 2 Server/Client mode

5. System Design Server –Holder of metadata only –Thread-based Connection –Event-based Service

6. System Design Figure 3 Server Process/Threads

7. System Design Client –Physical memory owner –UI/Work/Page Fetch Thread –Fixed-home Protocol –Not Aware of Peer Clients

8. System Design Figure 4 Client process/thread

9. System Design Figure 5 Sample Operation

10. Implementation Message Passing: TCP socket Figure 6 Message Passing

11. Implementation Server/Client Page Table –Server holds most up-to-date meta data –Server managers whole virtual memory space –Server records id & addresses of all nodes –Client owns the most up-to-date local memory segment –Client caches referenced pages from peer nodes

12. Client IDIP Address 0129.74.155.107 (e.g.) 1129.74.155.122 ….… Page #Frame #Access BitsPage Owner 057PROT_READ1 167PROT_READ|PROT_WRITE1 257PROT_READ3 ………… Figure 7 Connection Table Figure 8 Server Page Table

13. Implementation Page #Frame #Access BitsPage OwnerRef Count 030PROT_READ10 131PROT_READ10 232PROT_READ14 360PROT_READ|PROT_WRITE11 4200PROT_READ50 …………… Figure 9 Client Page Table

14. Implementation Page fault handler –Client  Server Check the access right Fetch the page owner id/address Update global access bits –Client  Client Connect to the page owner Cache the referenced page Update local access bits

15. Implementation Page fault handler –Page fault type Read remote page Write on a page –Assumption Reading happens more often than writing Writing needs most-to-date copy more than reading

16. Implementation Assume reading remote page dsm call: dsm_do_no_page () Truly a remote reading fault? NO: double page fault dsm call: dsm_do_wrt_page () YES: continue Figure 10 Page fault handler wordflow

17. Implementation Memory Consistency Model –Assumption Revisit Reading happens more often than writing Writing needs most-to-date copy more than reading –Multi-Reader/Single Writer Snap-shot for reading Every writing triggers page fault –Locks on pages being referenced Semaphore-like reference counts: If ref_count > 0  Waiting/Re-random

18. DSM Evaluation Figure 11 Parallel Computation on ASP Problem

19. DSM Evaluation Figure 12 Execution time comparison

20. DSM Evaluation Figure 13 Message Transmission Comparison

21. DSM Evaluation Figure 14 Network Traffic Comparison

22. Future Work Enhance system robustness Evaluate scalability boundary Provide better programmability