Implementation of Page Management in Mome, a User-Level DSM Yvon Jégou Paris Project, IRISA/INRIA FRANCE

Overview Mome memory model Consistency management Multiple writers Node page management

Run-time systems for parallel languages Object-sharing between parallel programs Parallel code coupling Persistent data repository Checkpointing Background

Mome memory model Posix memory model Posix mmap Mome MomeMMap single node multiple nodes Process 1 Process 2 Mome space

Mome memory model Process 1 Process 2 Mome space

F77 common block sharing double precision tab(n,m) common /arrays/tab.. call MomeMMap(tab(1,1), sz(tab), WRITE, FIXED, seg1, 0).. tab(i, j)= … …= tab(k, l)…

Mome page management Directory-based –Directory records status of page in each node –No page home (asynchronous) directory migration or redistribution

Mome page consistency call MomeConsistency(tab(1,1),sz(tab),STRONG) Each process selects the consistency model at page level –Strong consistency model –Release consistency model (under development) –Weak consistency model Weak consistency: pages made consistent only on explicit request –Example: integrate all writes before last barrier

Weak consistency management Based on a scalar (Lamport) distributed clock –Page manager: valid-before date –Each node: Valid-before date Consistency-constraint date Consistency request: updates the consistency- constraint date –Example: date of last barrier call MomeSyncMem(tab(1,j), n, lastbarrierdate())

Weak consistency management Valid-before < Consistency-constraint –Accesses to page are invalidated –Access: page fault Page fault: request sent to manager –Manager: new Valid-before date –Manager: new version of page Clock: shifted on –Synchronization requests –Program request

Consistency management All requests from node to manager contain (date, contract) –Contract: consistency model in use during request As long as the page is valid on the node, the manager guarantees the last contract –Strong: no other writers –Release: no other writer before last synchronization –Weak consistency: no restriction Manager invalidates the page if the contract cannot be graranteed anymore

Multiple consistency modes Consistency at (page/node) level –Heterogeneous application on same DSM –Code coupling (application/coupling library) Code optimization: optimized code in weak consistency No global synchronization for consistency management

Multiple writers in Mome Mome favors the single writer case (most frequent case) –No twin on the first writer Twinning requested on second writer Merging modifications: when some consistency request is not satisfied –Modified pages sent to one node with twin –Result: exclusive OR of modified pages and twin –It is possible to send diffs

Multiple mappings The same page can be mapped multiple time on the same node –By the same process (synchronous) –By different processes of the same node (asynchronous): persistent data repository –By a distant process (no shared memory) Hierarchical DSM (work in progress) Experimented on Mach micro-kernel

Internal node page management On each node: Mome memory is mapped on the DSM file (temp file) Application pages made accessible through aliasing on Mome memory (temp file mmap ) Mome Memory DSM file

Internal node page management Mome Memory DSM file Application space

Internal node page management Page fault: page made available in Mome memory Mome Memory DSM file Application space

Internal node page management Page fault:application page is aliased with page in Mome memory Mome Memory DSM file Application space

Internal node page management Application process can read/write the page Mome Memory DSM file Application space mmap

DSM memory management Each page: reference counter –Mapping references –Communication layer references –Copy-on-write (refcount 1) Free page list DSM memory management thread –Free unused pages –Distant page-out (swapping)

Conclusion Same DSM kernel for –Parallel run-time library (HPF, OpenMP…) OpenMP: all shared / explicit sharing –Library for code coupling –Persistent data repository Close to a well-known memory model (Posix) Coupling OpenMP programs

DSM-based run-time systems for parallel languages Run-time library for F77+explicit parallel loops HPF (SPMD execution model) On-going work on OpenMP – all shared – explicitly shared

Object sharing Two parallel applications running on the same DSM Independent adress-spaces D S M Application 1 Application 2

Dynamic coupling between clusters Application 1Application 2

Dynamic coupling Application 1Application 2

Dynamic coupling Application 1Application 2 Communication threads

Persistent data repository DSM-based data repository

Persistent data repository DSM-based data repository Application 1

Persistent data repository DSM-based data repository Application 2 Checkpointing of DSM repository